SGLang-Neuron is publicly available: serve models and generate RL rollouts on AWS Trainium with the same SGLang interface, including prefill-decode disaggregation
An SGLang plugin from RadixArk and Annapurna Labs that brings Trainium to an inference framework developers already know.
// Key points
- It is an out-of-tree plugin selected with SGLANG_PLATFORM=neuron; SGLang core still handles the API server, request lifecycle and RadixAttention prefix caching.
- The plugin runs on native PyTorch with NKI kernels compiled through torch.compile, and implements the Trainium-specific models, attention, MoE and graph runner.
- It supports prefill-decode disaggregation, with the prefill server sending the KV cache to the decode server via NIXL over EFA.
- It ships as an AWS Deep Learning Container; the walkthrough serves gpt-oss-120b on a Trainium3 instance.
Builder's takeThe biggest cost of switching chips is switching the inference stack; SGLang users can now try Trainium without changing how they call it. If you self-host open models, run one current model with the same load on Trainium and on GPUs and compare cost per million tokens. Check the list of validated models first and keep anything off that list out of production for now.
// Background · from #Open models
Full timeline →- Nov 13 IBM releases the LLMON markup language with Granite 4.1 3B and 8B preview models: reserved special tokens separate instructions from data, blocking 47% more injection attacks on OPI than the best frontier model tested
- Nov 13 DeepSeek V4-Pro, V4-Flash and V4.1-Flash arrive on SageMaker JumpStart: MIT-licensed MoE models with a 1M-token context
- Nov 12 Microsoft Research discloses Lazarus: reprogramming fewer than 10 dormant neurons, with no training, backdoors an agent built on the model