Open source #AI agents #Open models Oct 20, 2026 · Originally published Oct 19

AWS contributes bi-directional KV transfer to vLLM, cutting time to first token in multi-turn agent sessions by up to 3.3x

In disaggregated prefill/decode serving, multi-turn sessions keep recomputing history; this feature sends KV generated on the decode side back for the prefill side to reuse.

Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

The problem: multi-turn agent sessions resend the full history each turn, and with one-way prefill-to-decode transfer the prefill node cannot reuse KV the decode node generated for earlier replies, so it recomputes it and TTFT rises as the conversation grows.
AWS built and validated it on Trainium and contributed it to vLLM as an accelerator-agnostic capability; it needs vLLM v0.21.0 or later with NixlConnector and bidirectional_kv_xfer enabled on both prefill and decode servers, plus a KV-cache-aware router.
With one prefill and one decode node on P5en, Qwen3-32B TTFT at 24k fell from 778 ms to 460 ms; GLM-5-FP8 at 20k fell from 2,334 ms to 702 ms, which AWS describes as up to 3.3x lower at long prompts.
The added KV transfer costs about 30 to 130 ms, far less than the recompute it removes.

Builder's takeIf you self-host inference for agents, this is one of the most practical latency fixes this year: agent sessions are long-history and multi-turn by nature, and recomputing history is the main reason first tokens are slow. It assumes you already split prefill and decode; if you haven't, don't re-architect for it, pin sessions to one node and turn on prefix caching first, which is cheaper.

// Background · from #AI agents

Full timeline →