Open source #Open models Nov 13, 2026 · Originally published Nov 12

SGLang-Neuron is publicly available: serve models and generate RL rollouts on AWS Trainium with the same SGLang interface, including prefill-decode disaggregation

An SGLang plugin from RadixArk and Annapurna Labs that brings Trainium to an inference framework developers already know.

Image: AWS Machine Learning Blog
Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

  • It is an out-of-tree plugin selected with SGLANG_PLATFORM=neuron; SGLang core still handles the API server, request lifecycle and RadixAttention prefix caching.
  • The plugin runs on native PyTorch with NKI kernels compiled through torch.compile, and implements the Trainium-specific models, attention, MoE and graph runner.
  • It supports prefill-decode disaggregation, with the prefill server sending the KV cache to the decode server via NIXL over EFA.
  • It ships as an AWS Deep Learning Container; the walkthrough serves gpt-oss-120b on a Trainium3 instance.

Builder's takeThe biggest cost of switching chips is switching the inference stack; SGLang users can now try Trainium without changing how they call it. If you self-host open models, run one current model with the same load on Trainium and on GPUs and compare cost per million tokens. Check the list of validated models first and keep anything off that list out of production for now.

// Background · from #Open models

Full timeline →
  1. Nov 13 IBM releases the LLMON markup language with Granite 4.1 3B and 8B preview models: reserved special tokens separate instructions from data, blocking 47% more injection attacks on OPI than the best frontier model tested
  2. Nov 13 DeepSeek V4-Pro, V4-Flash and V4.1-Flash arrive on SageMaker JumpStart: MIT-licensed MoE models with a 1M-token context
  3. Nov 12 Microsoft Research discloses Lazarus: reprogramming fewer than 10 dormant neurons, with no training, backdoors an agent built on the model

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →