No AI model met Epoch's employee standard on any real task; two reached contractor level on one
Epoch AI graded six models by hand on tasks from its own work. GPT-6.2 Astra averaged 65%, 17 points under Claude Fable 5.2.
Epoch AI tested Claude Fable 5.2, GPT-6.2 Astra and four other models on Epoch Automation Reports, tasks taken from Epoch’s own work. Staff graded each output by hand, with 100% meaning the standard Epoch expects of its employees. Fable 5.2 and GPT-6.2 Astra roughly tie for the top Epoch Capabilities Index score, yet no model met the employee standard on any task. GPT-6.2 Astra averaged 65%, 17 points below Fable 5.2.
The best result came on adding an AI data centre to Epoch’s database, which involves reading news and permits and analysing satellite images. Fable 5.2 scored 97.5% and Claude Opus 5.5 87.5%, which Epoch’s analyst judged comparable to contractor work, against 65% for GPT-6.2 Astra. A contractor typically spends about 16 hours per data centre. Opus 5.5 took just under 3 hours at a lower cost, and Fable 5.2 about 6.5 hours at a higher cost than a contractor.
On research tasks all models fell short. They reused existing ideas where the task asked for new ones, mostly kept their first attempt instead of iterating, and struggled to pick and present the key findings. When asked to compare their own output with a human’s, every model preferred its own, except Gemini 4 Argon on one task.
What it means
Top scores on standard benchmarks do not tell a reader how a model fares on open-ended work. Epoch is considering moving contractors’ time towards reviewing AI output on data centre analysis, so part of that task is near automation. The research tasks are not.
Caveats
- Epoch ran each model once per task on a small task suite, and graded subjectively. Epoch says the qualitative findings are more informative than the exact numbers.
- Epoch changed its setup after its 8 October report, so scores are not comparable with that one. Only some models have been re-run.
- Epoch evaluated GPT-6.2 Astra internally, before its release.
- The tasks come from one organisation’s work, so they do not stand for work in general.