SageMaker AI real-time endpoints support GPU partitioning: with NVIDIA MIG, one GPU splits into up to 7 isolated partitions for different models
Small models no longer need a whole GPU each, which suits teams hosting several fine-tuned variants, embedding models and rerankers at once.
// Key points
- Built on NVIDIA Multi-Instance GPU (MIG), each GPU splits into up to 7 partitions, each with its own memory and compute.
- You configure it when creating an endpoint that hosts inference components, and your application still calls each model by name.
- AWS cites fine-tuned model variants, embedding models, rerankers and classifiers; supported instances are P4d, P5, G7 and G7e.
- It launches in 7 Regions: N. Virginia, Ohio, Oregon, Singapore, Tokyo, Frankfurt and Ireland.
Builder's takeAI Cloud Drive's retrieval path is exactly this case, with a small embedding model and a small reranker each holding a whole card: put both on two partitions of one GPU, load-test latency, then check the bill. Note that partitions are hard-isolated with fixed memory each, so model size and concurrency have to be recalculated against the partition size.