Open-weight models on Amazon Bedrock now support explicit prompt caching alongside implicit caching, through the Chat Completions API
Long system prompts, reference documents and tool definitions are resent every time, and caching saves that input. How to choose between the two modes, how they bill and how to confirm hits, for Q&A or agent apps running open models on Bedrock.
// Key points
- Implicit caching reuses eligible prefixes automatically on a best-effort basis; explicit caching marks reusable prefixes with breakpoints, several up to the model's maximum, to cache layers that change at different rates. Minimum prefix length, breakpoint limits and TTL are on each model card, and the TTL resets with each hit.
- Cache reads are billed below the standard input rate; depending on the model, cache writes can cost more than standard input. When content after a shared prefix changes on every request, use explicit caching so you do not pay to write content that is never read back.
- In the example, an OpenCode coding agent session on the same API uses implicit caching with no configuration, and each later request reads everything up to the end of the previous one from cache. Cross-Region inference can cause extra cache writes at times of high demand.
Builder's takeAI Cloud Drive's document Q&A is the classic fixed long prefix with a different question each time, which suits explicit breakpoints; agent loops are fine with implicit caching. Before launch, check the cached_tokens field in responses, as the post suggests, rather than guessing from the monthly bill.
// Background · from #Open models
Full timeline →- Nov 17 Hugging Face moves Inference Providers onto OpenRouter's router: open-weight models served by 80+ providers are callable with one Hugging Face token, with no markup
- Nov 17 NVIDIA's provisional MLPerf Endpoints v1.0 results: Vera Rubin NVL72 delivers 5x to 10x the throughput per kilowatt of GB300 NVL72 on agentic inference
- Nov 16 tiny-voca is released under Apache 2.0: a 4.5M-parameter English TTS model that runs about 70x faster than real time on four CPU threads