TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
An MIT Technology Review report published Nov. 2 argues that companies should assess AI coding tools by the cost and quality of software that gets merged, not token consumption alone. It recommends tracking cost per active developer and cost per merged pull request while testing whether hybrid or owned computing capacity lowers fully loaded costs.
An MIT Technology Review report published Nov. 2 argues that companies using AI coding agents should measure spending against software changes that pass review and are merged, rather than treating token consumption as the main measure of value. The report says token-based bills can rise as agents plan, write code, run tests and retry, but do not show whether that work produced usable software.
The report proposes tracking cost per merged pull request: the total cost of AI-assisted development divided by changes that pass review and enter a codebase. It recommends tracking rework and defects alongside that figure, since a lower cost per change would not represent an improvement if quality also fell. It also recommends measuring cost per active developer, rather than spreading spending across every licensed seat.
To reduce costs without imposing broad usage limits, the report recommends routing work to different models. It suggests reserving frontier models for difficult planning or novel reasoning, while assigning suitable repeatable tasks—such as code generation, test creation and refactoring—to open-weight models run by the organization. A gateway can route requests according to complexity, sensitivity, cost and latency.
The report also contrasts rented inference, where each call adds to a bill, with owned computing capacity, whose economics can improve when usage is sustained. It cautions that ownership brings costs for power, operations, maintenance and depreciation. Its proposed test is whether fully loaded cost per merged pull request and cost per active developer improve as usage grows—not whether infrastructure is owned in itself.
Linking AI Spend to Merged Code
The proposed measure addresses a gap between what AI coding tools consume and what engineering teams deliver. Token counts can identify usage, but cannot by themselves distinguish productive work from repeated retries or runaway loops. Conversely, a cap may curb waste but also constrain workflows that produce valuable code. Measuring merged work, with quality indicators, gives teams a way to examine both sides.
That question matters as agentic coding expands. The report says one coding task can use 10 to 50 times more tokens than a basic chat prompt because an agent may plan, generate code, test and revise. The report cites cost pressures including Uber reportedly exhausting its 2026 AI budget by April after expanding use of Claude Code, GitHub Copilot’s move from flat-rate pricing to metered credits, and Gartner’s prediction that AI coding token costs will surpass average developer salaries by 2028. These examples are attributed in the report; they do not establish that the same spending pattern applies to every company.
For technology leaders, the suggested measurements make infrastructure choices more testable. A team can compare model routing or owned capacity with its current setup, while checking that quality does not deteriorate. That is more informative than assuming either that more usage is automatically valuable or that less usage is automatically efficient.
Products Worth Considering
Apple Watch Series 12 GPS Smartwatch, 46 mm
Jackery Solar Generator 1000 v2 with 200W Solar Panel
Samsung 32-inch Odyssey OLED G8 (G80SH) 4K QD-OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
From Token Bills to Delivery Metrics
AI coding agents differ from one-shot chat prompts because they can carry out multiple steps within a single task: planning, code generation, test execution and corrections. Each step can trigger additional model calls. Under per-token pricing, a workflow that is more useful because it performs more work may also incur higher variable charges.
The report’s alternative is not to ignore consumption, but to connect it to an outcome. It recommends recording cost per active developer and cost per merged pull request before changing infrastructure, then running a complete development cycle on owned capacity and comparing results. The report says open-weight models can approach frontier performance on some software-engineering benchmarks, while stressing that this does not establish equivalent performance across repositories or workflows. Teams should test candidates against their own code, tests and acceptance criteria.
HPE’s experience is presented as a vendor example rather than independent evidence of typical results. The report says HPE routes routine workloads from more than 150 internal applications to open-weight models on premises and reports token-cost reductions of 30% to 50%. It also says the top 5% of users generated most consumption, and the heaviest users consumed roughly 450 times as much as the lightest on the same license. These figures describe the cited example and should not be treated as universal benchmarks.
“Cost per token measures consumption, not results.”
— MIT Technology Review report
Costs and Quality Need Local Tests
The report does not establish that owned capacity will save money for every company. Usage levels and predictability matter: it says APIs may suit low or uneven demand, while ownership can make sense when sustained usage keeps equipment busy. The cited HPE reductions are vendor-reported, and the report does not provide independent verification or enough detail here to compare the result across organizations.
It is also unclear how well any particular open-weight model will perform on a company’s own codebase. Performance on selected benchmarks does not guarantee comparable results across different repositories, security requirements or development workflows. Teams need to assess quality, defects and rework as well as cost before routing production tasks to a model.
The report’s projected cost trends and examples signal possible budget pressures, but do not set out a common accounting method for comparing all providers, internal infrastructure and labor. A meaningful cost-per-merged-change calculation depends on what a company includes in its total development cost and how it accounts for unsuccessful work.
Establish a Baseline Before Routing
The report recommends that companies first establish baselines for cost per active developer and cost per merged pull request. They can then test a complete development cycle on owned capacity, including operating and maintenance expenses, and compare costs alongside defects and rework. The test should show whether costs improve as usage rises without reducing software quality.
For companies with mixed demand, the report describes a hybrid approach: use external frontier-model APIs for demanding or intermittent tasks, and owned capacity for repeatable, sustained workloads. A governed gateway can apply rules about cost, task type and sensitive data. The next decision is not simply whether to impose a cap or buy infrastructure, but which arrangement produces reliable, reviewed software at an acceptable fully loaded cost.
Key Questions
What metric does the report recommend instead of token counts?
It recommends cost per merged pull request, calculated by dividing AI-assisted development costs by changes that pass review and are merged. The report says to track defects and rework alongside it.
Does the report say companies should stop tracking token use?
No. Its argument is that token consumption alone does not measure results. Token use can still help identify spending and usage patterns, but the report says it should be considered alongside delivery cost and software quality.
Are open-weight models equivalent to frontier models?
The report says leading open-weight models can approach frontier performance on some software-engineering benchmarks. It does not establish equivalence for every repository or workflow and recommends testing models against a company’s own code and acceptance criteria.
Will owning AI computing capacity always reduce costs?
No. The report says ownership can become more efficient with sustained use, but companies must account for power, operations, maintenance and depreciation. Low or unpredictable demand may favor rented APIs instead.
What should companies measure before changing infrastructure?
The report recommends baselining cost per active developer and cost per merged pull request, then comparing results after a complete development cycle on the proposed setup. The comparison should include quality indicators such as defects and rework.
Source: technologyreview.com
