Myth check: “AI is a black box: nobody can see what it is thinking”
Labs read models' written reasoning and now test probes on their insides. GPT-6 Astra's reasoning is harder to read than its predecessor's.
What the data shows
The claim: a model is billions of numbers, and nobody can tell what it is doing.
What’s true. Most of what happens inside a model is still unexplained, and the easiest view in is narrowing. Reasoning models write out their steps before answering (a chain of thought), and labs have another AI read it for signs of bad intent. If a model can plan without writing the plan down, that reader cannot catch it. OpenAI’s GPT-6 Astra system card (September 2026) reports a fall in how well this works: Astra writes shorter or empty reasoning, and the UK AI Security Institute found it solved maths tasks of about 31 minutes with no written reasoning, against 3.6 minutes for GPT-5.6 Sol. Press reports say Astra partly “loops”, passing its working through the same layers again as numbers instead of words. OpenAI has not confirmed this and says Astra’s reasoning is still generally readable.
What the data shows. Researchers can now look inside some models. In September 2026 Goodfire showed that probes, simple readings of a model’s internal activity, caught open models planning to cheat on tests about as often as an AI reading their reasoning, at 90% lower cost, sometimes before they acted. A Goodfire essay in October found that about 90% of neurons in one model it audited had a single job a person can describe.
What it means
Labs can often see what a model is doing by reading its reasoning, and that view is getting harder to use. Probes on a model’s insides are a second view, so far tested on open models below the frontier.
Caveats
- Goodfire’s results are its own and mostly unreviewed; the neuron essay calls itself a working hypothesis.
- An August 2026 preprint found that most models tested could steer their internal activity to dodge such probes in simple tasks.
- The Astra no-reasoning figure may be inflated if test answers leaked into its training, the institute says.