AWS shows decision models served with llama.cpp on SageMaker: a quantized 4B model scores 77.9% on 231 JevBench tasks at 0.352 s p50 on GPU
A managed path for small models that score a fixed set of options instead of generating text, with CPU and GPU numbers to choose between.
// Key points
- The AWS Deep Learning Container for llama.cpp serves OpenAI-compatible chat completions; a SemIf extension built as a child image returns probabilities over declared options without decoding an answer token.
- Qwen3.5-4B Q4_K_M on ml.c6i.8xlarge (CPU): 77.5% accuracy, p50 1.715 s, p95 19.291 s; on ml.g6e.xlarge (GPU): 77.9%, p50 0.352 s, p95 0.599 s.
- The GPU endpoint was 4.9 times faster at p50 and 32.2 times faster at p95; 2B and 0.8B candidates reached only 66.7% and 58.4%, so the sample keeps 4B as its default.
- AWS advises keeping authorization and deterministic business rules outside the model and validating probabilities before treating them as confidence.
Builder's takeAfter Liquid’s d1, Cloudflare’s Clef and Amazon’s Strands deciders, this is the deployment manual. For fixed-choice work like routing interview answers into categories in AI Interview or classifying uploads in the AI cloud drive, one small GPU with p95 under 0.6 s is cheap enough to try now. Keep CPU for offline batches: a 19-second p95 rules it out for anything interactive.
// Background · from #Open models
Full timeline →- Oct 13 Anthropic’s Frontier Red Team says small models may be months from self-replicating on IoT devices; a 9B model set up inference on 8 of 9 hardware types
- Oct 12 Qwen releases Qwen-Image-2.1-Turbo weights: a 7B image model that generates in 8 steps, under a research license
- Oct 12 Tencent releases Youtu-Parsing-Omni, a 5B omni-modal parsing model scoring 96.96 overall on OmniDocBench v1.6