Myth check: “Model collapse from AI-generated data will end AI progress”

What the data shows

The claim: as AI-written text fills the internet, new models will be trained on it, copies of copies, and get steadily worse until progress stops.

What’s true. Shumailov et al. (Nature, 2024) showed that training each new model generation on the previous one’s output, with no care, causes damage that can’t be undone. Rare content (the “tail”) disappears first. That result is real.

What the data shows. The effect depends on replacing real data. Gerstgrasser et al. (2024) showed that when AI-made data builds up alongside the original real data, errors stay limited however many generations follow. Collapse doesn’t happen in their analysis. In practice, labs select and filter training data rather than training blindly on scraped AI output. And measured abilities kept rising steeply through 2026 (see the METR and FrontierMath entries).

What it means

Model collapse is a real effect in the lab, but it needs a setup labs avoid: throwing away real data and training only on AI output. It is not a reason to expect AI progress to stop. Where training data comes from, and how good it is, still matters.

Caveats

• Shumailov has argued that keeping real data reduces the effect at increasing cost rather than removing it.
• The collapse experiments use simplified setups.
• We couldn’t open the Nature page directly (login wall); details come from the abstract as reported elsewhere.