What happened: The UK AI Security Institute published results from its evaluation of Anthropic’s Claude Mythos Preview, testing it on capture-the-flag challenges and multi-step cyber ranges intended to simulate real attack chains.
Why it matters: This is the mechanism layer everyone keeps waving their hands about: measurable success rates on expert hacking tasks, and partial (sometimes full) completion of long attack sequences when explicitly directed and given access. That is enough to move the risk conversation from vibes to budgets.
Wider context: AISI notes rapid improvement in cyber performance across frontier models since 2023 and argues evaluation environments must evolve toward defended networks (monitoring, endpoint detection, active response) because undefended ranges will stop being discriminating.
Background: AISI describes a 32-step corporate network simulation (“The Last Ones”) and reports Mythos Preview as the first model to complete it end-to-end in some runs, while also flagging limitations (including failure to complete an operational-technology-focused range in their setup).
Our evaluation of Claude Mythos Preview’s cyber capabilities — UK AI Security Institute
Singularity Soup Take: If you wanted a clean reason to stop treating “cyber-capable models” as a philosophical debate, here it is. The policy question is no longer “should we worry,” it is “who gets access, with what logging and controls, and what gets mandated as baseline hygiene.”
Key Takeaways:
- CTF Breakthrough: AISI reports Mythos Preview succeeds 73% of the time on expert-level CTF tasks that no model could complete before April 2025, continuing a fast capability climb that has compressed the timeline for defensive preparation.
- Multi-Step Attacks: On its 32-step “The Last Ones” cyber range, AISI says Mythos Preview completed the full scenario in 3 of 10 attempts and averaged 22 of 32 steps, outperforming other models in their tests under a large token budget.
- Caveats And Next Steps: AISI emphasizes their ranges are easier than real networks (no active defenders, limited defensive tooling, no penalties for noisy actions). They plan to test against hardened environments and urge organizations to implement basics like patching, access control, and comprehensive logging.