A contract agent can get every click right and still build the wrong approval process

A contract agent can click every correct button and still create the wrong business process. A purchase workflow may route to legal but omit finance above a threshold. A reusable clause may contain the right text but apply it to the wrong jurisdiction. An NDA form may collect fields correctly while giving a user access to an agreement they are not permitted to launch. The hard task is preserving rules across the whole lifecycle, not generating legal language one paragraph at a time.

OpenAI and Ironclad converted this problem into eleven research tasks covering legal, commercial and procurement work. The tasks included configuring confidentiality agreements, building purchase-approval processes and updating reusable clauses for a selected jurisdiction. Each task was evaluated against eight to fifty criteria. OpenAI reports GPT-6 Astra scoring 32% higher on average than GPT-5.6 Sol while using 48% less estimated time per attempt.[1]

The benchmark unit should be a working process

Clause drafting is easy to display because a reviewer can see the text. Workflow configuration contains hidden dependencies: thresholds, roles, fallbacks, permissions, exceptions, required fields, notifications and downstream records. A process that works for the standard case can fail on the first renewal, foreign jurisdiction or out-of-policy request. This is why the criteria count can reach fifty even when the visible user request sounds simple.

Ironclad’s current product design reinforces the point. Conversational intake can suggest an available workflow, explain the recommendation and require the user to confirm before launch. The Workflow Designer Agent is separately enabled by an administrator and works inside the configuration environment. These are authority boundaries as well as interface choices: finding a workflow, designing one, launching one and approving a contract should not collapse into one unreviewed action.[3][4]

Criteria should encode the rule, not reward the appearance

A screenshot can show a polished approval diagram without revealing whether its conditions are mutually exclusive, whether a default path leaks around a control or whether inherited permissions expose the wrong contracts. Evaluation should execute test cases: below and above value thresholds; a missing cost centre; a restricted requester; an unsupported jurisdiction; an amendment tied to the wrong parent; a renewal after the original owner has left.

The strongest benchmark uses the same principle as software testing. Each criterion states an observable requirement, and the environment reveals whether it holds. Some criteria can be deterministic—field required, role permitted, formula correct. Others require professional judgement—whether a clause matches the company’s approved risk position. The final score should expose both categories rather than averaging a serious policy failure into dozens of easy clicks.

The commercial context is part of the model input

Ironclad says its new Contract Knowledge Graph grounds agents in contracts, workflows, prior positions, deals and negotiations. That can make an agent more useful than a generic model reading one PDF. It also means the system’s answer depends on the quality, permissions and currency of institutional records. Past practice may be inconsistent or outdated; repetition does not turn it into policy.[2]

1. Freeze the business specification before asking the agent to configure the process.
2. Separate deterministic requirements from legal or commercial judgement.
3. Run positive, negative, boundary and permission test cases before publication.
4. Show the exact change set and require approval for mutating workflow rules.
5. Preserve the version, requester, evidence, criteria and result for every agent run.
6. Measure reopened contracts, exception handling and downstream correction—not only completion time.

Contract agents will become valuable when they can carry business rules through a complicated system without losing them halfway. That requires more than a stronger model. It requires task definitions from people who know the work, safe practice environments, granular evaluation, permissions, change review and receipts. The winning product will not be the agent that edits the most screens. It will be the one whose completed process still behaves correctly when the first exception arrives.

Report an error or suggest a correction