🔍 Read the full analysis: FrontierCode: What Makes Maintainers Consider An AI Pull Request? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a benchmark built with open-source maintainers to assess whether coding-agent patches meet repository-specific review criteria. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; results indicate substantial limits remain, while higher scores can cost much more.
Artificial Analysis tested 23 coding models on FrontierCode, a benchmark that asks whether a software patch meets repository-specific review standards, not merely whether it passes tests. Claude Opus 5.5 ranked first with a 52.2% score, but passed all blocking checks in 57.7% of attempts, indicating that even the top result falls short of reliable, hands-off pull requests under this test.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, using the benchmark’s 100-task Main set. Claude Opus 5.5 (max with fallback) scored 52.2%. GPT-6.1 Sol (max) placed second at 49.1%, followed closely by Claude Fable 5.1 (max with fallback) at 49.0%. The reported pass rate measures how often a model cleared every blocking check: Opus did so in 57.7% of attempts, while GPT-6.1 Sol did so in 54.7%.
FrontierCode’s scoring combines hidden tests, command checks, reverse tests, adaptive tests, scope limits and an LLM code review based on maintainers’ criteria. A failure on any blocking check gives an attempt a score of zero; passing those checks earns a score between 70% and 100%, depending on quality checks. The benchmark also penalizes oversized diffs, changes outside relevant files and unhelpful tests. Artificial Analysis ran models through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the review judge.
The results also show a marked cost spread. Artificial Analysis lists GPT-6 Luna (max) at 41.9% and $0.09 per task, MiMo-V2.6-Pro at 45.4% and $0.18, GPT-6.1 Sol at 49.1% and $0.81, and Opus at 52.2% and $10.15. Those figures are benchmark-specific estimates, not a guarantee of costs in other coding workflows. Nine of the 100 tasks were not solved by any model on any attempt, according to the source report.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer-Style Checks Matter
Many coding evaluations emphasize whether generated code passes tests. FrontierCode adds checks intended to reflect the work involved in reviewing a real repository change: whether the patch stays within scope, whether its tests distinguish correct behavior from the original bug, and whether it meets stated review criteria. That makes its results relevant to teams considering where coding agents can assist—and where human review remains necessary.
The results do not establish how often AI-generated pull requests are accepted in day-to-day projects. They do show that, on this benchmark, the top model failed at least one blocking criterion in a substantial share of attempts. A high score is not the same as dependable autonomous merging. The cost figures also highlight a practical trade-off: moving from GPT-6.1 Sol to Opus raises the benchmark score by 3.1 percentage points, while the listed cost per task rises from $0.81 to $10.15.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Patches
FrontierCode was developed by Cognition and open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against looking up solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source report says task authors spent more than 40 hours on each task. The full benchmark has 150 tasks, while Artificial Analysis evaluated the 100-task Main set.
The tasks are private, according to the benchmark description, to reduce the chance that models encountered their solutions during training. A trajectory scanner can disqualify an attempt if it flags solution lookup. The design aims to measure more than functional correctness, but the resulting scores still reflect a particular set of tasks, checks, agent settings and review judge—not every repository or development process.
Artificial Analysis’ rankings also differ from those on its Intelligence Index. For example, Claude Sonnet 5.5 ranked second among the listed models on that index but fifth on FrontierCode; DeepSeek V4.1 Flash placed seventh on FrontierCode despite a lower Intelligence Index score. The comparison suggests that broad capability rankings do not directly predict performance on this benchmark’s patch-quality criteria.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results cover 23 models and 100 tasks, not the full 150-task benchmark or every real-world coding situation. The source report does not provide enough detail here to determine how results vary across individual repositories, task types or repeated runs beyond the stated three attempts per task. It is also unclear how closely the benchmark’s automated review judge reflects decisions made by human maintainers on live pull requests.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is incomplete. The reported costs and scores are specific to Artificial Analysis’ setup, and should not be treated as direct forecasts for other tools, prompts or production environments.
As an affiliate, we earn on qualifying purchases.
More Models and Runs to Watch
The next useful update will be additional results for models not yet tested, followed by further runs that can show whether the ranking holds across a broader set of systems. Readers can also watch for fuller reporting on task-level outcomes, repository variation and how benchmark scores compare with maintainers’ decisions on actual pull requests.
For now, the published results support a limited conclusion: FrontierCode rewards more than passing tests, and no listed model cleared every blocking check on every attempt. Teams evaluating coding agents should treat the benchmark as one measure of patch quality, rather than as proof that a system can safely merge changes without human review.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It scores coding-agent patches against repository-specific checks, including hidden and adaptive tests, scope limits, command checks and an LLM review based on maintainers’ criteria.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis results with a 52.2% FrontierCode score. It passed all blocking checks in 57.7% of attempts.
Does the top score mean the model can merge code autonomously?
No. The score is a benchmark result, not evidence that autonomous merging is dependable in live repositories. The leading model did not clear every blocking check in 42.3% of attempts.
Which model was the least expensive among the cost-frontier examples?
Artificial Analysis listed GPT-6 Luna (max) at $0.09 per task with a 41.9% score. These are estimates for the benchmark setup and may not match costs in other workflows.
Were all major models included?
No. As of October 13, 2026, the source report said there were no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
