🔍 Read the full analysis: CWE-bench V2 Brings More Coverage And Clarity To AI Evaluation on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks drawn from open-source repositories and spanning 91 CWE categories. The release reports a top pass@1 score of 67.8% among 11 models, while the benchmark authors say every evaluated model failed some tasks on every attempt.
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks for evaluating coding agents on real open-source repositories. The release includes the tasks, a deterministic grader and an evaluation harness; reported results for 11 models put Gemini 4 Argon at 67.8% pass@1, though the authors say no evaluated model solved every task on every attempt.
Each task gives an agent a checkout of a real repository and a short instruction to find and repair a security weakness without breaking existing functionality. The 100 tasks cover 91 CWE categories. The benchmark is packaged in self-contained Harbor environments, and a registry file is provided for running the tasks from Hugging Face. Users must accept the dataset’s access conditions before running it.
In the release’s reported evaluation of 11 models, Gemini 4 Argon scored 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 followed at 67.3% and 78.0%, respectively. Pass@1 and pass@4 report results under different numbers of attempts; they are not guarantees of performance on other software. The authors say at least one model solved 95 tasks, but that does not mean one model solved all 95.
The release also compares cost and performance. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout; Claude Opus 5.5 scored about ten percentage points higher at $1.83, while Gemini 4 Argon scored slightly higher at $3.92. These are figures reported by the benchmark authors, and costs may vary with model pricing and evaluation setup. The announcement says each task was reviewed and that checks were designed to test behavior rather than a particular code structure.
What Security-Fix Scores Show
CWE-bench v2 offers researchers and developers a shared set of practical tasks for comparing coding agents on security repairs. Publishing the tasks, grader and harness gives others a way to inspect the benchmark and repeat evaluations on the public set, rather than relying only on a vendor’s description of model performance. The coverage across 91 weakness categories also makes the test broader than one centered on a single vulnerability pattern.
The results are not evidence that a model is ready to handle security work unsupervised. The leading reported pass@1 score remains below 70%, and the authors say the leading model failed 19% of the set on every attempt. A benchmark measures performance on its particular tasks and checks; it cannot establish how an agent will behave across all repositories, threat models or production conditions. The results instead offer a point of comparison and show that substantial gaps remain within this test.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Waterproof, Dustproof, USB‑C, Up to 12 hr Battery, Black
As an affiliate, we earn on qualifying purchases.
How Version 2 Is Evaluated
Version 2 retains the earlier task format and expands the public set to 100 tasks. According to the release, reviewers checked whether task instructions were realistic, whether the expected behavior could be inferred, and whether verifiers tested correct outcomes rather than one prescribed implementation. They also checked that starter code failed, a reference fix passed, existing functionality remained intact, and checks rejected memorized solutions such as known upstream patches. The authors say LLM judge panels helped shape programmatic checks for difficulty and fairness.
The public set is only part of the evaluation program. Collinear AI says it maintains a separate held-out task set to track progress on tasks not seen by evaluated models; teams must contact the company for access. Public results are tracked in two Hugging Face channels: a dataset-page leaderboard for open-weight models and a leaderboard Space for broader comparisons, including API-only models. Contributors can submit results through pull requests.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparisons
The supplied announcement does not provide enough methodological detail to independently assess every comparison, including exact model configurations and the conditions used for cost estimates. Public scores may change as new submissions are added. The held-out tasks are not available for direct inspection, so outside researchers cannot reproduce evaluations on that set from the public materials.
It also remains unclear how well results on these curated tasks predict performance in production security reviews. The authors describe task review and verifier checks, but a strong benchmark score alone does not show how an agent handles the full range of software, threats and deployment conditions. Independent repetitions of the public evaluation and comparisons with real security work would help clarify how much the scores generalize.
More Runs and Community Review
Researchers and developers can run models on the public tasks using Harbor after accepting the dataset terms and setting up Hugging Face authentication. Collinear AI invites feedback and task contributions, and leaderboard results can be submitted through pull requests. Further runs may change the public rankings or expose problems in the task set and its checks.
Teams interested in the held-out evaluation can contact Collinear AI, according to the release. The next useful signals will include independent reproductions, clearer reporting of model configurations and costs, and evidence about whether benchmark results correspond with outcomes in security reviews. The source does not specify a date for any further release or evaluation milestone.
Key Questions
What is CWE-bench v2?
It is a public benchmark from Collinear AI containing 100 security-fix tasks based on real open-source repositories. The tasks span 91 CWE categories and are distributed with a grader and evaluation harness.
Which model scored highest in the reported results?
Gemini 4 Argon had the highest reported pass@1 score, at 67.8%, and a pass@4 score of 81.0%. These are results reported in the release, not independent confirmation of performance beyond the benchmark.
Does the benchmark show that an AI agent can safely fix software vulnerabilities?
No. It measures performance on a defined set of tasks. The reported scores leave some tasks unsolved, and the release does not establish how well results predict performance across production repositories or real security reviews.
Can researchers reproduce the results?
The tasks, grader and harness for the public set are available, subject to the dataset’s access conditions. A separate held-out set is not public; interested teams must contact Collinear AI to evaluate models on it.
Primary source: Hugging Face · via ThorstenMeyerAI.com
