Myth check: “AI can't even count the r's in strawberry, so it can't reason”
AI reads words in chunks, not letters, so counting letters is genuinely hard for it. It's a spelling blind spot, not proof it can't reason.
The claim in full “AI can't even count the letters in “strawberry”, so it can't really reason.”
What the data shows
The argument: if a chatbot can’t say how many r’s are in “strawberry”, something a child can do, it can’t really be thinking.
What’s true. The failures are real and measured. Language models don’t see letters. They read text as tokens, chunks like “straw” and “berry”. Fu et al. (2024) found models recognise letters but struggle to count them, worst of all with repeated letters. Cosma et al. (EMNLP 2025) trace the problem to tokens and show letter-level skills appear only late in training. On CharBench (2025), models without step-by-step reasoning scored 62–90% on counting characters (GPT-4o: 89%).
What the data shows. Reasoning models, which work through a problem before answering, do much better on related tasks inside words. On SubTokenTest (January 2026), GPT-5 scored 98–99%, against 48–59% for GPT-4. The same models that stumble on letters solve olympiad maths and research problems (see the maths topic). Letter counting tests how a model reads text, not whether it can reason.
What it means
Check a chatbot’s answers on spelling, letter counts and similar word puzzles: that weakness is real. But failing at them says little about what the same model can do on maths or code. Newer reasoning models make these mistakes far less often on related tasks.
Caveats
- We found no dedicated benchmark of current frontier models on letter counting alone.
- The SubTokenTest results cover related tasks, not letter counting itself, so they show the direction rather than today’s exact error rate.