Today's thread is the infrastructure agents run on: cache lifetimes stretch to 12 hours, voice is packaged from transcription to speech in one pipeline, and enterprises get their own gate both at package install and before inference. For product teams, the work is re-running cache math and locking down agent installs, not chasing new models.
The Responses API can now use 12-hour prompt caching, keeping shared instructions, tools and context cached longer so requests hours apart still get lower cached-input pricing.
One request can set both 12-hour and 30-minute breakpoints; upgrading existing 30-minute coverage to 12 hours is charged as a new 12-hour write, with no credit for the earlier one.
Builder's takeThis picks up from yesterday's Cloudflare item: cut cost through cache structure first. I'll pull out the system prompts PandaClaws reuses all day but with gaps longer than half an hour, put them before a 12-hour breakpoint, and keep session content on 30-minute breakpoints; work out from your own call spacing how many reuses a day a prefix needs before the 2.5x write pays off.
The transcription model Voxtral Small Transcribe 3 covers 50+ languages with diarization, context biasing and end-of-turn detection; Mistral says it outperforms Whisper, Scribe v2 and Gemini 3.5 Transcribe.
The Mini transcriber has 4B parameters and costs $0.0035 per minute; Voxtral Realtime Transcribe 3 runs at under 500 ms latency in 35+ languages.
The TTS model Voxtral Mini TTS 3.0 supports 35+ languages and 50+ dialects at $13 per million characters; Pro Voice reproduces a speaker's style from about 30 minutes of audio.
A new voice agent SDK combines listening, reasoning, tool calling and speaking in one real-time streaming pipeline.
Builder's takeFor voice-first products like AI Interview, what I care about most is end-of-turn detection and self-deployment: a bad interruption call wrecks the interview, and candidate recordings should ideally stay in my own environment. I'll first compare Chinese error rate and end-to-end latency against my current transcription on a batch of real interview recordings before deciding to switch.
The firewall inspects both package metadata requests and downloads, in public preview for all GitHub Enterprise Cloud customers, including data residency.
Admins can set a cooldown period, check the GitHub Advisory Database for malware or vulnerability alerts, and run supply-chain checks like attestation downgrades; erroneous blocks go through an exception request.
Coding agents and CI hosts must be configured to route requests through the firewall instead of going straight to upstream registries.
Free during preview, metered like Actions at general availability; only npm packages are supported for now.
Builder's takeCoding agents install dependencies on their own, inside environments that hold secrets, and that's the most overlooked gap. Even without GitHub Enterprise, I'd do two things now: set a cooldown so npm doesn't pull versions published a few hours ago, and never inject production credentials while an agent is installing packages.
Early benchmarks: with the same number of type-check workers, tsrs is up to 8x faster than TypeScript 7.0 with about 70% less memory; TypeScript 7.1 beta is up to 2.5x faster than 7.0, partly from optimizations upstreamed from tsrs.
It is incubated in the Oxc toolchain and an experimental release is on npm as tsrs; differences from upstream remain, such as diagnostic order.
Cloudflare says frontier AI models can now do large-scale ports almost independently in days, one reason to start now.
Maintenance follows Vinext: agents review upstream changes each morning and open tracking issues, and the upstream test suite builds a compatibility matrix nightly.
Builder's takeWhat I'd learn from most is the maintenance method: agents track upstream daily and a compatibility matrix runs nightly, which is what keeps an AI port from rotting. tsrs itself is experimental; I'll run it alongside tsc in CI on one large repo to compare time and diagnostics, without replacing the real type check.
As of October 20, Inference hooks are out of beta for Claude Enterprise organizations.
Once Claude points at the organization's AI security server, each governed prompt across claude.ai, Cowork and Claude Code is held for the server's allow or deny verdict before inference proceeds.
The docs include a configuration guide and instructions for building an integration endpoint.
Builder's takeRead with the October 15 Compliance API item, Anthropic is moving enterprise control from after-the-fact audit to blocking before inference. If you build AI security gateways for enterprises, this is a ready integration point; if you wire one up yourself, decide first whether a timeout or outage means allow or deny, since that default matters more than the rules.
Structured outputs work for all Anthropic Claude models on both the bedrock-runtime and bedrock-mantle endpoints, where the model is available.
Two mechanisms: JSON Schema output format controls the response structure, and strict tool use validates tool parameters against schemas.
It works with batch inference, with cross-Region inference on bedrock-runtime, and can stream structured responses as typed events.
Responses contain only the fields the schema defines, which cuts output tokens and can lower cost for high-volume workloads.
Builder's takeTeams on Bedrock can finally delete the retry-on-parse-failure glue. File metadata extraction in AI Cloud Drive is exactly this case; I'll tighten the schema to required fields only, then compare output tokens and failure rate. Constrained decoding guarantees format, not correctness, so keep sampling the results.
MoE models decode 5–6.5x faster than in 2.14: on an M3 Ultra with torch.compile, Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s and LFM2.5-8B-A1B from 23.6 to 123.6 tokens/s.
FP8 comes to MPS, storing float8_e4m3fn in half the space of BF16; softmax over the vocabulary is about 8x faster.
Wan2.2's video VAE decode now peaks at 8 GiB of memory instead of 270 GiB.
MPS now requires macOS 15 or later, and PyTorch 2.15 requires Python 3.11 or later.
Builder's takeA 35B MoE at 61 tokens/s is enough for local development and offline batch work. I'll move some internal evaluation scripts to a Mac with an MoE model for a first pass and skip spinning up GPU machines for small experiments; before upgrading, check you're on macOS 15 and Python 3.11 or later.
TorchTPU is a native PyTorch backend for TPU, with TPU-tuned Pallas kernels, distributed training support and a torch.compile integration.
In Transformers, passing device_map="tpu" loads a model directly onto the TPU; eager mode is the default, with torch.compile for more speed.
Diffusers treats TPU as a first-class device, and models too big for one chip can be sharded across chips at load time with a TensorParallelConfig.
With native support, Optimum TPU is no longer required to run Transformers and Diffusers.
Builder's takeRead with yesterday's TorchNeuron, PyTorch has taken both Trainium and TPU this week. For product teams, inference pricing is no longer GPU quotes alone; my advice is still to move one running inference service over unchanged and compare unit cost, rather than rewrite code for new hardware.
With GPT-4.1, an older non-reasoning model, on τ²-bench's stock ReAct scaffold, retail Pass^4 (all four independent attempts correct) rose from 57.5% to 67.5%, above every retail entry in the October 12 leaderboard snapshot.
Retail mean accuracy rose from 78.1% to 82.5%; airline from 57.5% to 65.0%.
How many rules to keep depends on the domain: retail did best with all 292; airline's Pass^4 peaked at 45% with only 14 high-support rules, against 40% with all 101.
The same model extracts the guidelines and runs the agent, with no weight updates or human annotation.
Builder's takeThis matches the approach in my checklist on fallback and human confirmation in agent systems: turning failures into explicit rules beats switching to a pricier model. I'll keep a failure log for every agent in production, distill it into rules for the prompt on a schedule, and watch every-time metrics like Pass^4, not just the average.
It contains 300 realistic two-candidate decision scenarios across 29 domains including hiring, healthcare, lending, education, housing and criminal justice.
Identity is varied from none to a demographically coded name, implied identity and an explicit label; adversarial prefixes appeal to authority, consensus, historical disadvantage and more, each with a matched version for the other candidate.
Across models including Olmo, Llama, Qwen, Gemma, Claude and the GPT family, and across training stages, the same persuasive arguments make models more or less likely to change decisions depending on which group they favor.
The benchmark is available to download and is being built into Ai2's open evaluation workbench Olmo-eval.
Builder's takeThis bears directly on AI Interview: in resume screening, a single line like "leadership recommends this candidate" from either side can sway the model. I'll bring FairStress's hiring setup into my own eval set, swapping names and adding adversarial prefixes to the same resume to see whether the verdict holds, and have the model output only scores against the job requirements rather than free-form reversals.
Researched and drafted with AI assistance; editorial standards and views set by Darius.