AWS ships a distributed KV caching agent skill: evaluate and set up a shared ElastiCache KV tier from your coding agent, cutting self-hosted inference cost by up to 40% and time to first token by up to 80%
When you self-host, a returning conversation often recomputes prefill for hundreds of thousands of tokens. The idea behind a shared KV tier, and how the skill does the math before touching anything.
// Key points
- A shared KV tier next to the GPU fleet extends GPU-local caches so any server can reuse KV state for days across sessions, routing shifts and server failures; AWS says this cuts cost by up to 40% and time to first token by up to 80%.
- The skill lives in the Agent Toolkit for AWS and works in coding agents that support Agent Skills, such as Kiro, Claude Code and OpenAI Codex: it runs a fit check, estimates savings from your own traffic with KVTrace, and stops if they do not cover the cache's cost.
- It adds the LMCache connector to existing vLLM or SGLang engines, including vLLM under llm-d and Ray Serve, and aligns KV ID mechanism and data type across servers; before production it empties local tiers and checks that two cache-loaded runs give the same output.
- Prerequisites include GPU instances with EFA RDMA write such as p5, LMCache 0.5.8rc1+, vLLM 0.11.1+ (or SGLang 0.5.13+) and ElastiCache for Valkey 9.1+; creating, changing or deleting resources requires explicit approval.
Builder's takeThe more useful lesson is how the skill is written: read-only checks first, savings estimated from your own traffic, stop if the numbers do not work, human approval for high-impact steps. It is a good template for turning ops knowledge into a skill. For self-hosted services with many long conversations I'd run KVTrace on hit rates before deciding, and heed the trap it calls out: with a layout mismatch the hit counters look healthy while answers are wrong, so check output consistency before going live.
// Background · from #Open models
Full timeline →- Nov 22 SageMaker JumpStart adds a model leaderboard: more than 70 LLMs compared on quality under one evaluation setup, plus measured time to first token, throughput and instance cost
- Nov 22 NVIDIA brings Dynamo into RL training: KV-aware routing and delta weight updates cut H Company's average training-step time by up to 20% and GPU-hours by up to 5%
- Nov 22 NVIDIA open-sources frontend-crates, the Rust tool-call parsers behind Dynamo, with a conformance corpus shared across model families so AI agents can write parsers for new models