Airbnb shares how it picks models for coding agents: eight new models in about five weeks, with defaults and effort ceilings set by cost per task
How a large company actually runs internal coding agents: score and cost per task on its own tasks before rollout, then check against production data after. For anyone managing model choice and spend for a team.
// Key points
- Thousands of engineers and engineering-adjacent people use AirChat, Airbnb's internal tool for coding agents, every week; requests go through an LLM gateway running on Amazon Bedrock.
- Eight new models came out in about five weeks; by mid-October about two thirds of coding agent requests went to those new models.
- On one model, moving from low to high effort raised the score by 0.8 points while cost per task was 2.5 times as high; the gateway enforces an effort ceiling per model.
- A newer model scored higher on the same tasks and grader at about a sixth of the cost per task, and users were moved to it with a configuration change.
Builder's takeRead with yesterday's GitHub items, the conclusion is the same: models change too fast, and the only reliable yardstick is your own tasks plus cost per task. I would copy two things: run every new model on your own tasks at each effort level on release day, and set effort ceilings in the gateway with low or medium as default. Paying 2.5x for 0.8 points is not worth it for most everyday changes.
// Background · from #Coding agents
Full timeline →- Nov 19 Google releases Gemini 4 Neon at $3 input and $15 output per million tokens, with 32% lower cost per task than Carbon at its default high level
- Nov 19 RadixArk open-sources DaisyRose-27B: reinforcement learning on Qwen3.8-27B with 1,200 long-horizon agent tasks lifts Terminal-Bench 4.1 from 5.6% to 12.0%
- Nov 19 GitHub Copilot team customizations are generally available: team maintainers can turn on approved agent plugins by default for direct members