Tools #Open models Nov 22, 2026 · Originally published Nov 20

SageMaker JumpStart adds a model leaderboard: more than 70 LLMs compared on quality under one evaluation setup, plus measured time to first token, throughput and instance cost

The hard part of choosing an open model is that published scores use different setups. What this leaderboard measures consistently, and how to read it for the configuration you will deploy.

Primary source AWS · aws.amazon.com Read the original ↗

// Key points

  • AWS evaluates every model on three reasoning benchmarks, BIG-Bench Hard, GPQA Diamond and MMLU-Pro, under the same setup, and measures time to first token, throughput and SageMaker instance cost for each JumpStart deployment configuration.
  • At launch it covers more than 70 LLMs from providers including Meta, Google, Qwen, IBM and Arcee.
  • You can filter by provider, license and modality, then pick a use-case profile and an optimization goal (balanced, cost, latency or throughput) to see metrics for the configuration you plan to deploy.
  • It is available at no additional charge in all AWS commercial Regions where SageMaker Studio is available.

Builder's takePutting quality scores next to latency and hourly cost for a specific configuration is far more useful than a bare leaderboard. All three benchmarks are generic reasoning tests and say nothing about your workload, so I'd use it as a first cut down to three or four candidates, then run my own task set; for latency-sensitive cases like AI Interview, sort by the latency goal first.

// Background · from #Open models

Full timeline →
  1. Nov 22 AWS ships a distributed KV caching agent skill: evaluate and set up a shared ElastiCache KV tier from your coding agent, cutting self-hosted inference cost by up to 40% and time to first token by up to 80%
  2. Nov 22 NVIDIA brings Dynamo into RL training: KV-aware routing and delta weight updates cut H Company's average training-step time by up to 20% and GPU-hours by up to 5%
  3. Nov 22 NVIDIA open-sources frontend-crates, the Rust tool-call parsers behind Dynamo, with a conformance corpus shared across model families so AI agents can write parsers for new models

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →