Tools Nov 8, 2026 · Originally published Nov 6

SageMaker Inference adds Multi-Instance GPU partitioning: half an H100 beats a full A100-40GB by 53 to 89 percent in throughput and is 18 to 34 percent cheaper per million output tokens

One endpoint can now host more models than it has GPUs, with hardware isolation between them, which suits per-tenant and fine-tuned variants.

Image: AWS Machine Learning Blog
Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

  • NVIDIA MIG splits one physical GPU into isolated partitions, each with a fixed share of compute and dedicated memory enforced in hardware, up to seven depending on the GPU
  • It works only on endpoints that use inference components; single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned
  • In AWS tests, a 40 GB partition of an H100 against a full A100-40GB delivered 53 to 89 percent more output tokens per second at each scenario's best-cost load, and 5.6 to 47.5 percent lower time to first token
  • Above a crossover that comes earlier as prompts lengthen, it was 18 to 34 percent cheaper per million output tokens, at on-demand prices

Builder's takeMany teams give every small model its own GPU and pay for idle silicon. For AI Interview's role-specific fine-tuned models I would move them onto inference components first, swap a full older GPU for half a newer one for a week, and compare latency and the bill before migrating everything.

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →