🔍 Read the full analysis: How OpenAI’s Software-Based Agent Training Raises Questions For Ironclad on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI says it trained GPT-6 Astra in hosted copies of contract-management platform Ironclad, using synthetic tasks based on public contracts. Astra met an average 55% of rubric criteria across 11 workflows; the reported time estimates were simulated, and the results do not establish customer productivity gains or readiness for unsupervised work.
OpenAI said on October 6 that it trained its frontier model GPT-6 Astra in hosted copies of contract-management software company Ironclad’s product, testing it on 11 legal, commercial and procurement workflows. The results show progress on complex software tasks, but Astra met an average of 55% of evaluation criteria, and OpenAI says its time figures are simulated estimates—not measured customer savings.
Ironclad staff and OpenAI employees who use the product selected 11 tasks, including creating nondisclosure agreements, configuring procurement approvals and revising a reusable contract clause based on a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes per task. The tasks were scored against rubrics containing 8 to 50 criteria, depending on complexity.
OpenAI reported that GPT-6 Astra, run at its maximum setting, met an average 55.0% of the criteria. GPT-5.6 Sol, run at high effort, averaged 41.6%, while an internal OpenAI model used during Astra’s development reached 63.7%. On one example task, Astra met about 94% of the criteria. These are rubric scores across the research tasks, not percentages of tasks fully completed.
For training, Ironclad supplied hosted product copies where models could practise. OpenAI said it generated synthetic tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. The company stated that it did not use OpenAI customer data, its internal contracts or non-public Ironclad customer data. OpenAI estimated Astra’s time per attempt at 19.2 minutes, compared with 37 minutes for Sol, but described those figures as simulated estimates based on assumed processing and generation speeds.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Work Needs More Than a Score
The test addresses a practical question for businesses: can an AI agent carry out multi-step work inside specialized software while preserving company rules and controls? A model may complete many actions correctly and still make a consequential error if it misses one required approval or applies the wrong contract provision. In the procurement example described by OpenAI, a workflow could require Finance approval above a spending threshold, Security review for certain requests and Legal review for nonstandard terms. Missing one condition could undermine the whole process.
That makes the reported 55% average criteria score a measure of progress, not evidence that the system is ready to run contract workflows without review. OpenAI’s account itself says human oversight remains important when an agent may lose track of a business rule. For buyers, the relevant question is not only how many criteria an agent meets on average, but which criteria it misses, how serious those failures are and whether a person can detect them before work is finalized.
The initiative also points to a possible change in how AI systems are developed: software companies could serve as training and testing environments for agents learning professional work. That may make products more useful, but it could also shift the value of software away from its screens and toward the rules, records, audit trails and controls behind them. That is an implication of the approach, not a result established by this test.
As an affiliate, we earn on qualifying purchases.
How Ironclad Became a Test Environment
The October 6 post was titled “Advancing computer use with Ironclad.” The name refers to Ironclad, the contract-management software company, rather than a new AI framework. OpenAI described the project as training models to understand business rules, complete multi-step workflows in specialized software and check results against the original requirements.
OpenAI said it is inviting a small number of software companies to work on tasks that current agents cannot reliably complete. It asked prospective partners to bring a concrete example of an agent failure, people with expertise in the relevant work, a secure test environment and data that can safely be used for research. The stated aim is to study difficult real-world workflows, rather than only general computer-use tasks.
The source account attributes to Ironclad CTO Sunita Verma the point that agents must preserve “the controls teams rely on.” OpenAI’s post also presents the work as evidence that a full contracting platform remains important. Those statements describe the companies’ framing; they do not establish how the partnership will affect Ironclad’s product or its customers.
As an affiliate, we earn on qualifying purchases.
What the Evaluation Cannot Yet Show
The published results cover 11 selected research tasks; they do not show how Astra performs across Ironclad workflows generally, across different customers or under live production conditions. The account does not provide task-by-task details of which rubric criteria were missed, how severe those failures were, or how often a human reviewer would catch them. An average score can conceal a failure in a mandatory control.
The 19.2-minute and 37-minute figures are simulated estimates, according to OpenAI, not observed completion times or measured savings for customers. The comparison therefore does not establish that Astra currently makes contract work faster. It also remains unclear how the reported results would change with different workflows, data or software configurations, and what review procedures would be required before any deployment.
OpenAI says it did not use non-public Ironclad customer data, but the source material does not specify all technical or contractual arrangements for the hosted test environments. Nor does it provide a deployment schedule, commercial terms or details about which other software companies may take part. These points should not be inferred from the research announcement.
As an affiliate, we earn on qualifying purchases.
What Buyers and Partners Should Check
OpenAI’s stated next step is to invite a small group of software companies to propose difficult tasks, provide domain experts and supply secure environments and research-eligible data. The company has not named additional partners or announced a timetable in the source material. Further results would be needed to show whether agent performance improves on a wider range of workflows and whether those gains translate into dependable use.
Organizations considering agents in contract, finance or customer-record systems can ask vendors for criterion-level results, rather than a single average score; details on failed requirements; and evidence from testing under conditions similar to their own. They can also ask how human review, approval thresholds, audit records and access controls work. Until such information is available, this evaluation is best read as a research demonstration of training in specialized software—not proof of safe, hands-off automation or realized productivity gains.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is Ironclad in this announcement?
Ironclad is a contract-management software company. OpenAI’s October 6 post describes training and testing a model inside hosted copies of Ironclad’s product; it is not announcing a framework called Ironclad.
What does Astra’s 55% score mean?
It is the average share of rubric criteria met across the 11 tasks. It does not mean Astra completed 55% of the tasks, and it does not show that each workflow was safe or usable without review.
Did OpenAI show that the agent saves customers time?
No. OpenAI described the reported per-attempt times as simulated estimates based on assumed processing and generation speeds. They are not measured time savings from customer use.
What data did OpenAI say it used?
OpenAI said it created synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database and filtered out personal information. It said it did not use OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data.
Is Astra ready to run contract workflows on its own?
The reported results do not establish that. OpenAI’s account says human oversight remains important, and the average score does not reveal which individual requirements were missed. The evaluation is a limited research test, not evidence of readiness for unsupervised deployment.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
