OpenAI and contract-management company Ironclad have turned 11 professional workflows into training and evaluation tasks for computer-using agents. The examples include configuring nondisclosure agreements, building procurement approvals and updating reusable clauses according to a requester’s jurisdiction.[1]
The interesting part is not that a model can click through specialised software. It is that each finished workflow was checked against 8 to 50 criteria. A procurement process may look complete while routing a high-value purchase around Finance, omitting Security review for a sensitive vendor or applying the wrong legal clause to one jurisdiction. In professional work, one hidden exception can matter more than twenty correct clicks.[1]
What the Ironclad evaluation measured
| Measure | Value | Context |
|---|---|---|
| Research tasks | 11 | Selected across legal, commercial and procurement work with Ironclad employees and OpenAI users of the product.Source 1 |
| Criteria per task | 8 to 50 | The rubric checks whether the completed workflow satisfies the requirements, not merely whether the agent navigated the interface.Source 1 |
| Experienced-user estimate | About 30 to 40 minutes per task | A human estimate supplied for the selected tasks, not measured production telemetry.Source 1 |
| GPT-5.6 Sol result | 41.6% mean rubric score | Run at the reasoning setting where that model scored highest in the comparison.Source 1 |
| GPT-6 Astra result | 55.0% mean rubric score | An internal development model reached 63.7% on the same research task set.Source 1 |
| Estimated attempt time | 37.0 minutes versus 19.2 minutes | Simulated estimates for Sol and Astra; the source explicitly says they are not measured customer time savings.Source 1 |
The unit of success is the rule-preserving workflow
A weak computer-use benchmark can reward visible progress: open the right page, fill the form, press save. A business process is a set of conditional obligations. Purchases above one threshold may require Finance; certain data categories may require Security; nonstandard language may require Legal. The agent must preserve those rules across ordinary cases and exceptions, then leave an auditable record.
That changes how an evaluation should be built. The task instruction is only one input. A high-quality rubric also needs test cases above and below thresholds, conflicting conditions, missing information and attempts to use an unapproved path. The evaluator should inspect the final configuration and exercise it, rather than infer success from the action trace.
Three ways to score a professional agent
| Score | What it rewards | What it can miss |
|---|---|---|
| Action completion | The agent reached the expected screens and performed listed steps | A superficially complete process whose rules are wrongSource 1 |
| Final-state checklist | Required fields, clauses, approvals and records exist | Conditional behaviour that fails when the workflow is exercisedSource 1 |
| Scenario-based rubric | The configured system behaves correctly across normal and exception cases | Unrepresented edge cases and changes after the evaluationSource 1 |
The performance gain is meaningful—and still incomplete
Astra’s 55.0% mean score is 13.4 percentage points above Sol’s 41.6%. Relative to the starting score, the improvement is about 32.2%, calculated as 13.4 divided by 41.6. The estimated attempt time is 17.8 minutes lower, or about 48.1%. Those calculations reproduce the percentages highlighted by OpenAI, but the source warns that time was simulated from assumed processing and generation speeds.[1]
One illustrated task shows Astra meeting about 94% of criteria in an estimated 20 minutes and Sol meeting about 85% in 32 minutes. A near-complete score can still conceal the one requirement that makes a contract process unacceptable. For a production decision, teams need the distribution of misses: which rules failed, how severe they were and whether the same failure recurred across tasks.[1]
Synthetic training data is useful when its boundary is explicit
OpenAI says the simulated tasks were built from contracts publicly available through the SEC’s EDGAR system, with filters designed to remove personal information. It says no OpenAI customer data, internal contracts or nonpublic Ironclad customer data were used for training or evaluation. That boundary reduces one category of confidentiality risk while preserving representative structures for research.[1][3]
Synthetic tasks still need domain review. A plausible contract clause can be legally inappropriate; a workflow copied from one organisation can encode the wrong authority structure for another. The safest use of synthetic data is to create controlled variations around rules supplied by experts, then validate the resulting tasks before they become rewards.
A practical acceptance test for enterprise agents
Before delegating a professional workflow, an organisation can define five artefacts: the source policy, a machine-readable rule set, representative scenarios, a severity-weighted rubric and an audit record linking each action to a requirement. The agent may then create or change the system, but publication or activation waits for the scenarios to pass and an authorised reviewer to examine the exceptions.
The Ironclad collaboration does not establish that contracting can now be automated without legal or operational oversight. It demonstrates a better way to improve agents: bring the software vendor and domain experts into task design, let the model practise in a secure environment and judge the finished work against the business rules that matter. That standard can travel well beyond contracts—to finance, compliance, procurement, insurance and any system where an invisible condition is more important than a visible click.
Sources & further reading
- Advancing computer use with Ironclad OpenAI · 6 Oct 2026 · Accessed 8 Oct 2026
- GPT-6 Astra: A new generation of intelligence OpenAI · 8 Sep 2026 · Accessed 8 Oct 2026
- EDGAR Public Dissemination Service System U.S. Securities and Exchange Commission · 15 Dec 2015 · Accessed 8 Oct 2026