Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification

Feng, Y., et al. (2024, June 11).

arXiv.org.

https://arxiv.org/abs/2406.07515

Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about \emph{model collapse}, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.

Here are some thoughts:

Drawing on psychological principles of learning and evaluation, this paper argues that LLMs suffer from "model collapse" not because synthesized data is inherently useless, but because they are poor at self-evaluating quality. Like humans, LLMs can generate good outputs but struggle to reliably identify the best ones among many (e.g., using perplexity). The core insight is that external verification—using even imperfect "verifiers" to select high-quality synthetic examples—is crucial for scaling. This mirrors how human learning benefits from feedback: selection, not perfect generation, is the key. The authors theoretically prove and empirically demonstrate that a simple proxy (p*) measuring a verifier's ability to distinguish good from bad data strongly predicts model performance, showing that leveraging synthesized data with robust selection prevents collapse and can even surpass original models.

Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

Gerstgrasser, M., Schaeffer, R., et al. (2024).

arXiv (Cornell University).

https://doi.org/10.48550/arxiv.2404.01413

Abstract

The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.

Here are some thoughts:

This research directly addresses a critical concern for psychologists and researchers who rely on AI: the potential degradation of AI models when they are trained on data generated by previous AI models, a phenomenon known as "model collapse." While prior studies, often assuming old data is discarded and replaced with new AI-generated data, painted a dire picture of inevitable performance decline, this paper offers a more optimistic and realistic perspective. The authors argue that in the real world, data accumulates over time—new AI-generated content is added to the existing pool of human-generated data, not substituted for it. Through extensive experiments with language models, image generators, and molecular modeling tools, they demonstrate that this accumulation of data effectively prevents model collapse. Performance remains stable or even improves across successive generations of models trained on the growing, mixed dataset. The paper further supports this finding with a mathematical proof using a simplified linear model, showing that accumulating data bounds the error, preventing it from growing uncontrollably. For psychologists, this suggests that the increasing presence of AI-generated content on the internet may not catastrophically corrupt future AI tools used in research or clinical settings, as long as training datasets continue to incorporate diverse, original human data alongside synthetic content.

AI models collapse when trained on recursively generated data

Shumailov, I., et al. (2024).

Nature, 631(8022), 755–759.

https://doi.org/10.1038/s41586-024-07566-y

Abstract

Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.

Here are some thoughts:

This paper introduces and analyzes "model collapse," a degenerative process in which generative AI models—such as large language models (LLMs), variational autoencoders (VAEs), and Gaussian mixture models (GMMs)—deteriorate over successive generations when trained on data produced by previous versions of themselves. The authors demonstrate both theoretically and empirically that using model-generated content as training data causes models to gradually forget the true underlying data distribution, particularly losing sensitivity to rare or low-probability events (early model collapse), and eventually collapsing into a narrow, high-probability mode with very low variance (late model collapse). This occurs due to compounding errors from finite sampling, functional approximation, and model expressivity limitations. Experiments with the OPT-125m language model show that even fine-tuned models suffer from increasing perplexity and distorted output distributions over generations. The study warns that as AI-generated content floods the web, future models trained on such data risk becoming increasingly biased, inaccurate, and disconnected from reality. The authors stress the importance of preserving original, human-generated data and tracking data provenance to mitigate this inevitable collapse.

In short, relying on AI-generated content as training data for future AI models leads to a progressive degradation in model quality—a phenomenon called "model collapse." Over successive generations, models trained on synthetic data begin to lose critical information about rare or low-probability events (the "tails" of the distribution), eventually distorting reality and converging on a narrow, oversimplified version of the original data.

AI models fed AI-generated data quickly spew nonsense

Elizabeth Gibney

nature.com

News: July 24, 24

Training artificial intelligence (AI) models on AI-generated text quickly leads to the models churning out nonsense, a study has found. This cannibalistic phenomenon, termed model collapse, could halt the improvement of large language models (LLMs) as they run out of human-derived training data and as increasing amounts of AI-generated text pervade the Internet.

“The message is, we have to be very careful about what ends up in our training data,” says co-author Zakhar Shumaylov, an AI researcher at the University of Cambridge, UK. Otherwise, “things will always, provably, go wrong”. he says.” The team used a mathematical analysis to show that the problem of model collapse is likely to be universal, affecting all sizes of language model that use uncurated data, as well as simple image generators and other types of AI.

The researchers began by using an LLM to create Wikipedia-like entries, then trained new iterations of the model on text produced by its predecessor. As the AI-generated information — known as synthetic data — polluted the training set, the model’s outputs became gibberish. The ninth iteration of the model completed a Wikipedia-style article about English church towers with a treatise on the many colours of jackrabbit tails (see ‘AI gibberish’).

The info is here.

Here are some thoughts:

This article highlights a concerning phenomenon known as model collapse, which occurs when artificial intelligence (AI) models are trained on text generated by other AI models. This recursive training leads to a degradation in the quality of outputs, ultimately resulting in nonsensical responses. Researchers demonstrated that as AI-generated content increasingly permeates the internet, the reliance on this synthetic data could stifle the advancement of large language models (LLMs) due to a lack of high-quality human-derived training data. The study revealed that even before complete collapse, models trained on AI-generated texts tend to forget less frequent information, which poses significant risks for fair representation of marginalized groups.

Said differently: AI garbage in, AI garbage out.