CWE-bench v2 opens 100 vulnerability-fixing tasks across 91 CWEs: Gemini 4 Argon leads at 67.8% pass@1, Claude Opus 5.5 at 67.3%, DeepSeek V4.1 Flash reaches 57.0% at $0.04 a rollout
A reproducible test of whether coding agents can find and fix real security weaknesses without breaking the code, with costs alongside the scores.
// Key points
- An agent gets a checkout of a real open-source repository and a short instruction, and must find and fix a security weakness without breaking existing functionality; scoring is fully programmatic, and a held-out set is kept private
- Of 11 models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, ahead of Claude Opus 5.5 at 67.3% and 78.0%; even the leader fails 19% of tasks on every attempt
- On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: 57.0% pass@1 at $0.04 per rollout, about ten points more for Opus at $1.83, and Gemini slightly ahead at $3.92
- 95 tasks are solved by at least one model; the public set, grader and harness are open, and tasks are tagged as RL environments for training defensive agents
Builder's takeThe cost column is the useful one: a cheap open model gets most of the way, which suggests a two-tier security pass in CI, with the cheap model sweeping every PR and a frontier model on what it flags. Whatever the model, keep the benchmark's own bar for agent fixes: existing tests must still pass, and a person signs off on security patches.
// Background · from #Coding agents
Full timeline →- Nov 4 Dependabot version updates now support Agent Package Manager, opening pull requests for agent skills, prompts and instructions declared in apm.yml
- Nov 4 GitHub Copilot comes to JetBrains Air over the Agent Client Protocol, with cloud tasks billed to the Copilot plan instead of JetBrains AI credits
- Nov 3 GitHub Actions will block pull_request_target by default in public repositories from December 2, with brownouts on November 9 and 16