M4Bench finds better clinical specifications matter more than an agent “skill” wrapper

Clinical researchers routinely turn raw electronic health records into variables such as ventilation duration, medication exposure or a patient’s first qualifying laboratory result. The SQL is only the final expression of a much larger specification: which item codes count, how time windows are defined, what missing values mean, which encounter is selected and what one output row represents.[1][2]

M4Bench tests whether AI coding agents can reproduce that work. The benchmark contains 28 derivation tasks across 15 families using the MIMIC-IV and eICU databases. Its central result is both encouraging and sobering. Clinician-reviewed “skills” improved agents’ partial scores, but most of the gain came from giving the system a complete operational specification. Even then, fully correct runs were uncommon.[1][2]

Separating skill from specification

Agent skills are often presented as reusable packages of instructions, examples and domain knowledge. M4Bench was designed to ask which part of that package actually matters. In a Docker-isolated campaign using GPT 5.4 Mini and GPT 5.5 through Codex, the targeted skills raised average reward by 0.136 over the no-skill baseline, with improvement on 26 of 28 tasks.[1][2]

An eight-task control then supplied the operational specification without the skill wrapper. That alone raised reward by 0.162. Adding the skill on top produced only another 0.024, a difference the authors report as indistinguishable from zero. On a separate ten-task subset, decoy skills scored like no skill, while raw reference SQL nearly matched the targeted skills.[1][2]

Why clinical derivations are difficult

A superficially plausible query can be wrong in ways that alter a cohort without producing an obvious error. It may select measurements after rather than before an intervention, collapse multiple admissions, treat an undocumented event as a negative finding or return the right value at the wrong grain. Those choices can propagate into an observational study, risk model or quality metric.[1][2]

M4Bench’s schema tests are important for the same reason. With GPT 5.4 Mini, schema obfuscation preserved 86% of the skill gain and schema restructuring preserved 70% across 26 MIMIC-IV tasks. The result argues that agents were not relying only on familiar table and column names, although the authors correctly describe this as a robustness check rather than proof that memorisation played no role.[1][2]

Replication strengthens the direction, not the claim of reliability

A descriptive Claude Code replication used Opus 4.7 and Sonnet 4.6 for 560 runs across all tasks. It produced a directionally similar gain of 0.120, with 21 of 28 tasks improving. Cross-system consistency makes the role of specification more credible, but it does not erase the low strict-pass rate.[1][2]

The benchmark’s reward measures partial agreement with a reference. That is useful for diagnosing progress, yet clinical deployment requires a harder standard. A derivation that earns a higher score but returns the wrong patients can still invalidate a study. The authors’ decision to publish strict checks and per-run evidence is therefore more informative than the headline improvement alone.[1][2]

The practical lesson

Organisations building clinical data agents should invest first in versioned, clinician-reviewable specifications. Those documents need explicit code lists, timing logic, missingness rules, output grain, acceptance tests and examples of boundary cases. The model should generate code against that contract, and independent tests should verify both the values and the rows returned.[1][2]

M4Bench does not show that skills are useless. It shows that a skill cannot compensate for an underspecified task. In clinical data work, the most valuable intelligence may be the disciplined translation of expert intent into a specification that humans and machines can both inspect.[1][2]

Report an error or suggest a correction