OR-License tests whether AI optimisation agents can deliver a feasible client solution

An efficient plan is useful only if it satisfies the client’s actual constraints. OR-License tests that full task using 31 de-identified cases from paid operations-research engagements worth $3.68 million across ten industrial scenarios. The benchmark begins with business materials and required deliverables, before an expert has translated everything into a tidy optimisation problem.[1]

Operations research underpins decisions such as how goods should be routed, which orders to fulfil, how resources should be scheduled and how to allocate capacity under multiple constraints. The mathematics can be sophisticated, but the first difficult step often happens before the solver runs. Someone must understand what the client means by 'must', discover unstated rules, check that data tables align and identify which constraints cannot be violated. Conventional AI benchmarks frequently start after an expert has performed that translation. OR-License deliberately tests much of the translation itself.[1]

The information a client forgets to give you

The team retained, after de-identification, the original-style requirements, input data, expected delivery formats and information boundaries associated with the professional projects. It then developed a controlled client simulator, ORSim, which can answer questions about business facts without giving away the optimisation expertise being tested. That arrangement matters because an agent that is never allowed to ask a follow-up question is solving a different task from a consultant who can investigate the brief, request missing figures and clarify operational exceptions.[1]

The hidden evaluation checks whether each proposed solution meets hard constraints and compares the quality of feasible outcomes with the original expert solution accepted by the client. The researchers call their principal combined metric Deployment-Adjusted Quality, or DAQ. An infeasible delivery receives zero rather than a partial reward for an impressive objective score. This reflects a blunt reality: a routing plan that reduces mileage but violates a contractual time window is not 'almost right' merely because it is efficient on paper.[1]

Across eleven frontier language models in coding-agent settings, several configurations exceeded the expert reference on objective quality conditional on delivering a feasible solution. None matched the reference on Deployment-Adjusted Quality, which assigns zero to infeasible deliveries. The strongest primary Claude Code configuration reached 88.39% strict feasibility and DAQ of 0.87, against the expert reference of 1.[1]

Two scores reveal different kinds of progress

The researchers also studied the progression of work from information gathering through formulation, implementation and validation. They found task-equalised research-chain completion rates of 24% to 45% across models on evaluable tasks. This identifies an important distinction: generating a plausible optimisation formulation is not the same as completing an end-to-end professional engagement. Agents can stall during data interpretation, take the wrong branch of a constraint or fail to package the solution in the form a client actually needs.[1]

The human expert's invisible contribution

Traditional optimisation tests may inadvertently measure the capability to solve a problem already made legible by specialists. Real engagements include decisions that are difficult to infer from a spreadsheet: whether a staffing rule is legal, whether a shipment can be split, how to handle a missing depot capacity or which business priority takes precedence when objectives conflict. Successful consulting involves investigating those decisions and documenting the resulting assumptions. An AI system that begins with a wrong formulation can produce a flawless solution to the wrong problem.[1]

A configuration can deliver excellent feasible solutions on some cases and fail the constraints on others. Conditional quality captures the first success; DAQ includes the second failure. The higher conditional scores should therefore not be explained away as rewards for missing constraints—they are computed on feasible deliveries. Reliability across the case set is the remaining gap.[1]

Making the brief part of the evaluation

For an enterprise trial, the equivalent test would begin with the documents and uncertainties staff actually receive. It would preserve required formats, record clarification questions and apply an independent acceptance check. That exposes failures in understanding and delivery that a solver-only benchmark cannot see.[1]

OR-License makes the client brief part of the technical challenge. Its results show real progress in optimisation quality while identifying where usable delivery still breaks down: acquiring information, formulating the right problem and checking the completed work.[1]

Report an error or suggest a correction