What AI must preserve when it translates an old program

A replacement for an old program can look cleaner and still produce the wrong total. In a 19 October evaluation, OpenAI scored COBOL migrations only when every hidden test passed and observable outputs matched exactly. It reported an 86.6% success rate for GPT-6.1 Sol at low reasoning effort, compared with 5.4% for GPT-5.6 Sol. These are the developer’s results for its test, rather than a measured success rate for commercial migration projects.[1]

A penny can encode a rule

The GnuCOBOL manual describes a small but consequential choice. Arithmetic results are truncated when an explicit rounding clause is absent; the ROUNDED clause without a further mode selects the nearest value, with ties going away from zero. Storage precision and the specified mode therefore help determine the result.[3]

Consider our illustrative positive amount of £12.345 stored to two decimal places. Truncation produces £12.34; nearest rounding with a half-up tie rule produces £12.35. Repeating that same hypothetical calculation for 1,000 records creates a £10 difference. We checked this decimal arithmetic; it is not an observed migration loss or a test of a COBOL application.[3]

A migration must preserve whichever rule the existing system is meant to implement. Replacing truncation with a seemingly more natural rounding operation changes the result, even when the new code is shorter and easier to read. Improving a business rule is a separate decision from translating its implementation.

Preserve the change, as well as the answer

OpenAI’s accompanying MultiLangBench takes a related approach. Models build a SQLite-compatible engine, and the benchmark checks the resulting database against SQLite after a query. The comparison reaches beyond whether a program runs to whether it leaves the expected state behind.[2]

SQLite’s documentation explains another part of that behaviour: an atomic transaction makes all its changes together, or none of them. Its rollback design can recover the previous state after an interrupted write. That property depends on a coordinated sequence involving a journal and durable writes; matching a successful output alone says little about interruption recovery.[4]

In a hypothetical transfer, debiting one account and crediting another belong to the same transaction. A replacement that stops after the debit has changed the meaning of the operation. This example illustrates transaction behaviour; it is not a claim that MultiLangBench tests this particular failure.[4]

The old program’s awkward details may be carrying the rules its users depend on.

Report an error or suggest a correction