Moonshot lowers Kimi’s cyber benchmark score before planned open-weight release

Moonshot AI says it deliberately reduced one measure of Kimi K3.1 Pro’s performance before a planned release of its weights: the ability to find and exploit software vulnerabilities. On the company’s internal xdev-bench-hard evaluation, the final model scored 81.0%, compared with 90.5% for an unrestricted checkpoint. The company says much of its general coding ability survived the change.[1]

The 9.5-percentage-point difference describes a controlled company comparison. The benchmark uses expert-curated tasks based on recently disclosed software bugs, with exploitation carried out in sandboxes. Its pass@3 score counts whether a model succeeds within three attempts. Moonshot also says the selected runs had no hard timeout or output-token limit, conditions that matter when comparing the figures with other evaluations.[1]

Changing the training rather than only the interface

Moonshot says it removed cyber-related data during mid-training and excluded corresponding reinforcement-learning tasks while retaining general coding work. A hosted chatbot can refuse a request through its application layer. Once weights are distributed, downstream users can run or modify the model without that application. Reducing an underlying capability is therefore a different intervention from adding a warning or refusal to a chat window.[1]

The boundary is technically difficult because debugging and exploitation draw on overlapping skills. A programmer investigating a memory error might be writing a repair, building a test or developing an attack. Preserving useful programming while suppressing offensive performance means distinguishing tasks that can look similar at the level of code. A lower cyber score is informative only alongside an account of what legitimate work became harder.

K3.1 Pro is available through Moonshot’s hosted products and API, with a stated one-million-token context window. The company plans to release the full weights by 4 November 2026. That schedule creates a clear distinction between what customers can use now and what independent researchers will be able to inspect and modify after distribution.[1]

Two comparisons matter

The restricted model still improves on its predecessor in Moonshot’s internal cyber comparison. It can therefore be less capable than an unrestricted version of itself while increasing the capability available relative to the previous generation. The training intervention and the product’s overall advance answer different questions; reporting only the fall from 90.5% would obscure the second.[1]

Neither figure measures attacks prevented, the security of a randomly chosen website or resilience against every tool an agent could acquire. The evaluation provides evidence of a specific trade-off under disclosed conditions. Independent testing could establish whether the effect persists across different vulnerability tasks and whether ordinary adaptation restores capabilities the company sought to reduce.

For defenders, the useful follow-up would pair offensive tasks with repair and diagnosis: does the model still identify a flaw accurately, propose a correct patch and explain its consequences? Publishing reproducible comparisons of both capabilities would make the cost of the intervention visible. The planned weights release will give researchers a more direct way to investigate that balance than a hosted demonstration allows.

Report an error or suggest a correction