Tools #Open models#Coding agents Nov 22, 2026 · Originally published Nov 20

AWS ships a distributed KV caching agent skill: evaluate and set up a shared ElastiCache KV tier from your coding agent, cutting self-hosted inference cost by up to 40% and time to first token by up to 80%

When you self-host, a returning conversation often recomputes prefill for hundreds of thousands of tokens. The idea behind a shared KV tier, and how the skill does the math before touching anything.

Image: AWS Machine Learning Blog
Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

  • A shared KV tier next to the GPU fleet extends GPU-local caches so any server can reuse KV state for days across sessions, routing shifts and server failures; AWS says this cuts cost by up to 40% and time to first token by up to 80%.
  • The skill lives in the Agent Toolkit for AWS and works in coding agents that support Agent Skills, such as Kiro, Claude Code and OpenAI Codex: it runs a fit check, estimates savings from your own traffic with KVTrace, and stops if they do not cover the cache's cost.
  • It adds the LMCache connector to existing vLLM or SGLang engines, including vLLM under llm-d and Ray Serve, and aligns KV ID mechanism and data type across servers; before production it empties local tiers and checks that two cache-loaded runs give the same output.
  • Prerequisites include GPU instances with EFA RDMA write such as p5, LMCache 0.5.8rc1+, vLLM 0.11.1+ (or SGLang 0.5.13+) and ElastiCache for Valkey 9.1+; creating, changing or deleting resources requires explicit approval.

Builder's takeThe more useful lesson is how the skill is written: read-only checks first, savings estimated from your own traffic, stop if the numbers do not work, human approval for high-impact steps. It is a good template for turning ops knowledge into a skill. For self-hosted services with many long conversations I'd run KVTrace on hit rates before deciding, and heed the trap it calls out: with a layout mismatch the hit counters look healthy while answers are wrong, so check output consistency before going live.

// Background · from #Open models

Full timeline →
  1. Nov 22 SageMaker JumpStart adds a model leaderboard: more than 70 LLMs compared on quality under one evaluation setup, plus measured time to first token, throughput and instance cost
  2. Nov 22 NVIDIA brings Dynamo into RL training: KV-aware routing and delta weight updates cut H Company's average training-step time by up to 20% and GPU-hours by up to 5%
  3. Nov 22 NVIDIA open-sources frontend-crates, the Rust tool-call parsers behind Dynamo, with a conformance corpus shared across model families so AI agents can write parsers for new models

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →