SageMaker Inference adds Multi-Instance GPU partitioning: half an H100 beats a full A100-40GB by 53 to 89 percent in throughput and is 18 to 34 percent cheaper per million output tokens
One endpoint can now host more models than it has GPUs, with hardware isolation between them, which suits per-tenant and fine-tuned variants.
// Key points
- NVIDIA MIG splits one physical GPU into isolated partitions, each with a fixed share of compute and dedicated memory enforced in hardware, up to seven depending on the GPU
- It works only on endpoints that use inference components; single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned
- In AWS tests, a 40 GB partition of an H100 against a full A100-40GB delivered 53 to 89 percent more output tokens per second at each scenario's best-cost load, and 5.6 to 47.5 percent lower time to first token
- Above a crossover that comes earlier as prompts lengthen, it was 18 to 34 percent cheaper per million output tokens, at on-demand prices
Builder's takeMany teams give every small model its own GPU and pay for idle silicon. For AI Interview's role-specific fine-tuned models I would move them onto inference components first, swap a full older GPU for half a newer one for a week, and compare latency and the bill before migrating everything.