Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

Gerstgrasser, M., Schaeffer, R., et al. (2024).

arXiv (Cornell University).

https://doi.org/10.48550/arxiv.2404.01413

Abstract

The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.

Here are some thoughts:

This research directly addresses a critical concern for psychologists and researchers who rely on AI: the potential degradation of AI models when they are trained on data generated by previous AI models, a phenomenon known as "model collapse." While prior studies, often assuming old data is discarded and replaced with new AI-generated data, painted a dire picture of inevitable performance decline, this paper offers a more optimistic and realistic perspective. The authors argue that in the real world, data accumulates over time—new AI-generated content is added to the existing pool of human-generated data, not substituted for it. Through extensive experiments with language models, image generators, and molecular modeling tools, they demonstrate that this accumulation of data effectively prevents model collapse. Performance remains stable or even improves across successive generations of models trained on the growing, mixed dataset. The paper further supports this finding with a mathematical proof using a simplified linear model, showing that accumulating data bounds the error, preventing it from growing uncontrollably. For psychologists, this suggests that the increasing presence of AI-generated content on the internet may not catastrophically corrupt future AI tools used in research or clinical settings, as long as training datasets continue to incorporate diverse, original human data alongside synthetic content.

Self-Consuming Generative Models GO MAD

Alemohammad, S., et al. (n.d.).

OpenReview.

Link to article

Abstract:

Seismic advances in generative AI algorithms for imagery, text, and other data types have led to the temptation to use AI-synthesized data to train next-generation models. Repeating this process creates an autophagous ("self-consuming") loop whose properties are poorly understood. We conduct a thorough analytical and empirical analysis using state-of-the-art generative image models of three families of autophagous loops that differ in how fixed or fresh real training data is available through the generations of training and whether the samples from previous-generation models have been biased to trade off data quality versus diversity. Our primary conclusion across all scenarios is that without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease. We term this condition Model Autophagy Disorder (MAD), by analogy to mad cow disease, and show that appreciable MADness arises in just a few generations.

Here are some thoughts:

This study explored the potential consequences of autophagous loops in generative models, where models train future models using synthetic data. This phenomenon, known as Model Autophagy Disorder (MAD), can lead to a degradation of model quality and diversity, ultimately poisoning the entire Internet's data quality and diversity if left uncontrolled.

The researchers identified three families of autophagous loops and found that sampling bias plays a crucial role in the development of MAD. Without sufficient fresh real data, future generative models will inevitably suffer from MAD, leading to decreased quality and diversity. This has significant implications for practitioners working with generative models, particularly those using synthetic training data.

To mitigate the risks of MAD, practitioners can take steps to control the ratio of real-to-synthetic training data and identify synthetic data through watermarking or other methods. However, watermarking introduces hidden artifacts that can be amplified by autophagy, highlighting the need for autophagy-aware watermarking techniques. Future research should focus on developing these techniques, examining the effects of MADness on downstream tasks, and exploring the implications for other data types, such as language models.

The study's conclusions serve as a warning for practitioners, highlighting the need for careful consideration of the potential risks and consequences of autophagous loops. As generative models become increasingly ubiquitous, it is essential to address the risks associated with MAD to prevent a decline in data quality and diversity. By understanding the causes and consequences of MAD, practitioners can take steps to prevent its occurrence and ensure the continued development of high-quality generative models.