Shumailov, I., et al. (2024).
Nature, 631(8022), 755–759.
Abstract
Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.
Here are some thoughts:
This paper introduces and analyzes "model collapse," a degenerative process in which generative AI models—such as large language models (LLMs), variational autoencoders (VAEs), and Gaussian mixture models (GMMs)—deteriorate over successive generations when trained on data produced by previous versions of themselves. The authors demonstrate both theoretically and empirically that using model-generated content as training data causes models to gradually forget the true underlying data distribution, particularly losing sensitivity to rare or low-probability events (early model collapse), and eventually collapsing into a narrow, high-probability mode with very low variance (late model collapse). This occurs due to compounding errors from finite sampling, functional approximation, and model expressivity limitations. Experiments with the OPT-125m language model show that even fine-tuned models suffer from increasing perplexity and distorted output distributions over generations. The study warns that as AI-generated content floods the web, future models trained on such data risk becoming increasingly biased, inaccurate, and disconnected from reality. The authors stress the importance of preserving original, human-generated data and tracking data provenance to mitigate this inevitable collapse.
In short, relying on AI-generated content as training data for future AI models leads to a progressive degradation in model quality—a phenomenon called "model collapse." Over successive generations, models trained on synthetic data begin to lose critical information about rare or low-probability events (the "tails" of the distribution), eventually distorting reality and converging on a narrow, oversimplified version of the original data.
