A robot is an unusually unforgiving place to discover that software which compiles does not work. A node can build without errors yet publish to the wrong topic, miss a parameter, misunderstand a lifecycle transition or fail to answer a service request. Large language models have become proficient at changing code on the screen, but a new benchmark shows that their apparent success can vanish when the program is run inside the coordinated system it is meant to serve.[1]
ROSTrans studies migration from the older Robot Operating System, ROS1, to ROS2. This is not a rename exercise. The middleware, communication mechanisms and lifecycle rules are different, and reliable operation depends on several components interacting correctly. The researchers constructed 108 scaffolded tasks from more than 70 public ROS1 repositories and supplied verified ROS2 implementations with runtime tests and reproducible container environments.[1]
Why the tests are harder than a code-completion prompt
The researchers evaluate interface-level, behaviour-level and system-level correctness. Candidate code is inserted into an otherwise fixed reference package; it cannot quietly edit the tests or build rules in its own favour. The resulting package is built and executed, with tests exercising message topics, services, actions and parameters. This design makes the score more meaningful than asking a model to provide code that looks correct or passes a syntax check.[1]
Across six frontier models, reported pass-at-one rates ranged from 5.7% for DeepSeek V4 Pro to 10.9% for Claude Opus 4.6. GPT-5.5 reached 10.0%; Gemini 3.1 Pro 7.2%; Qwen 3.5 Plus 6.7%; and GLM-5 Turbo 6.5%. These figures refer to the specific 108 migration tasks and execution harness, not a general ranking of the models for all programming or robotics work.[1]
The striking diagnostic finding is that 56.5% of generated samples built but failed at runtime. Another reported figure—32.0% failing runtime assertions on the migrated interfaces—is part of the runtime failure analysis, not a separate class that should simply be added to the first percentage. Compiling tested compatibility with a build system. It did not demonstrate that the migrated robot components behaved correctly together.[1]
Repair changes the result, but not the engineering responsibility
What the robotics benchmark found
| Test or model | Result |
|---|---|
| Scaffolded migration tasks | 108 from over 70 ROS1 repositoriesSource 1 |
| Claude Opus 4.6 pass@1 | 10.9%Source 1 |
| GPT-5.5 pass@1 | 10.0%Source 1 |
| Lowest tested pass@1 | 5.7% (DeepSeek V4 Pro)Source 1 |
| Built but failed at runtime | 56.5% of evaluated samplesSource 1 |
| Claude after three repair rounds | 49.1% successSource 1 |
| GPT-5.5 after three repair rounds | 47.2% successSource 1 |
The authors also tested execution-guided repair. After three feedback rounds, success rose to 49.1% for Claude Opus 4.6 and 47.2% for GPT-5.5. That is a major increase from one attempt, yet more than half the tested tasks still did not pass. The gains should not be described as a deployed robot repairing itself successfully on a factory floor; the setting was a controlled software benchmark.[1]
A migration team could use the benchmark’s method as a practical template. First identify the relevant communication contracts; then preserve the reference environment and expected outputs. Run automated builds, but also launch the nodes, test message exchange and inspect failures. Ask the model to repair code with execution evidence. Finally, keep human review and integration testing before any connection to physical machinery.[1]
One limitation is that a scaffolded task does not capture every difficulty of moving a production robot fleet: undocumented interfaces, physical calibration, safety interlocks and hardware-specific timing may add complexity. Conversely, some benchmark tasks may be more difficult than an ordinary line-by-line migration. The proper inference is narrower: on this test suite, coding competence did not transfer reliably to distributed runtime correctness.[1]
The practical implication extends beyond robots. Warehouse systems, financial services and healthcare platforms also rely on independently developed components agreeing about message format, timing, authentication and state. A change that passes static checks may still fail only after another process responds. ROS makes this visible because the tests deliberately start components and observe interaction. In a production migration, the final sign-off should include an execution trace and a regression scenario for each critical interface, not merely an AI-written explanation of why a changed function should work.[1]
The hidden line between writing code and operating a system
Robotics exposes a weakness present in many AI-assisted software migrations: the text of a function can be plausible while the contract between components is wrong. That is why the most useful benchmark result is the one recorded after execution. A programmer may use an AI to accelerate a migration, but the finished system must earn trust through observable behaviour, not through its ability to compile.[1]
Sources & further reading
- ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration Zhao and colleagues, arXiv · 30 Oct 2026 · Accessed 2 Nov 2026