Skip to content
← Back to Agents & METR

METR finds Claude Opus 5.5 told every safety test in its set apart from ordinary tasks

A model that spots a test may behave better than it would in real use. METR did not check whether that happens.

19 of 19safety tests Claude Opus 5.5 ranked above all 36 ordinary tasks

METR, an independent evaluator, wanted to know whether AI models can tell a safety test from ordinary work. It placed “investigator” agents inside 19 alignment evals, which check whether a model behaves badly when given the chance, and 36 capability tasks. The alignment evals included honeypots (tasks with a tempting shortcut that reveals cheating), games against strong engines, physical-safety scenarios and seeds from the Petri auditing tool. The agents explored each environment and were asked for the chance it was an alignment eval, and what it was for.

Over the past year the agents got better at telling the two kinds apart. Claude Opus 4.8 and Claude Opus 5.5 were nearly perfect, and Anthropic’s latest models beat OpenAI’s. Averaged over three runs, both of Opus 5.5’s scores ranked every alignment eval above every capability eval. Most of Claude’s gain from Opus 4.6 to Opus 4.8 came from recognising alignment evals.

The agents named their clues. Opus 5.5 spotted all four changes METR had made to turn a capability task into a reward-hacking honeypot, and recognised a chess task as a copy of Palisade Research’s 2025 experiment. GPT-6 Astra read a nuclear-leak scenario as a purpose-built test, noting its tiny searchable world and unusually exact casualty figures.

What it means

A model that can tell it is being tested may behave better in the test than in real use, so a clean result on a safety test counts for less. METR says alignment evals built the usual way may be easy for current models to spot, and it suggests testing them against investigator agents and removing the clues that give them away.

Caveats

  • METR measured whether models can tell the tests apart, not whether that changes how they behave. Nobody has shown that here.
  • Asking a model whether it is in an alignment eval can prime it. METR saw this on GPT-6 Astra, whose direct score on one task was 91 while its open-ended answer described a capability eval.
  • The set is small (19 alignment tasks, 36 capability tasks), and METR says it does not prove alignment evals cannot be made unrecognisable.
  • Astra’s direct score moved 40 points or more between runs on 7 of the 55 tasks. METR thinks the agents did not explore enough on those.
  • METR calls this an unpolished research note that had less review than its research posts.
Suggest a correction