Welcome to the Nexus of Ethics, Psychology, Morality, Philosophy and Health Care

Welcome to the nexus of ethics, psychology, morality, technology, health care, and philosophy
Showing posts with label scaling laws. Show all posts
Showing posts with label scaling laws. Show all posts

Monday, November 3, 2025

Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check

Lourie, N., Hu, M. Y., & Cho, K. (2025).
ArXiv.org.

Abstract

Downstream scaling laws aim to predict task performance at larger scales from pretraining losses at smaller scales. Whether this prediction should be possible is unclear: some works demonstrate that task performance follows clear linear scaling trends under transformation, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, finding that close fit to linear scaling laws only occurs in a minority of cases: 39% of the time. Furthermore, seemingly benign changes to the experimental setting can completely change the scaling trend. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To fully model the relationship between pretraining loss and downstream task performance, we must embrace the cases in which scaling behavior deviates from linear trends.

Here is a summary:

This paper challenges the reliability of downstream scaling laws—the idea that you can predict how well a large language model will perform on specific tasks (like question answering or reasoning) based on its pretraining loss at smaller scales. While some prior work claims a consistent, often linear relationship between pretraining loss and downstream performance, this study shows that such predictable scaling is actually the exception, not the rule.

Key findings:
  • Only 39% of 46 evaluated tasks showed smooth, predictable (linear-like) scaling.
  • The rest exhibited irregular behaviors: inverse scaling (performance gets worse as models grow), nonmonotonic trends, high noise, no trend, or sudden “breakthrough” improvements (emergence).
  • Validation dataset choice matters: switching the corpus used to compute pretraining perplexity can flip conclusions about which model or pretraining data is better.
  • Experimental details matter: even with the same task and data, small changes in setup (e.g., prompt format, number of answer choices) can qualitatively change scaling behavior.
Conclusion: Downstream scaling laws are context-dependent and fragile. Researchers and practitioners should not assume linear scaling holds universally—and must validate scaling behavior in their own specific settings before relying on extrapolations.

Wednesday, July 30, 2025

AlphaGo Moment for Model Architecture Discovery

Liu, Y., Nan, Y., Xu, W. et al. (2025)
arXiv.org

Abstract

While AI systems demonstrate exponentially improving capabilities, the pace of AI research itself remains linearly bounded by human cognitive capacity, creating an increasingly severe development bottleneck. We present ASI-Arch, the first demonstration of Artificial Superintelligence for AI research (ASI4AI) in the critical domain of neural architecture discovery--a fully autonomous system that shatters this fundamental constraint by enabling AI to conduct its own architectural innovation. Moving beyond traditional Neural Architecture Search (NAS), which is fundamentally limited to exploring human-defined spaces, we introduce a paradigm shift from automated optimization to automated innovation. ASI-Arch can conduct end-to-end scientific research in the domain of architecture discovery, autonomously hypothesizing novel architectural concepts, implementing them as executable code, training and empirically validating their performance through rigorous experimentation and past experience. ASI-Arch conducted 1,773 autonomous experiments over 20,000 GPU hours, culminating in the discovery of 106 innovative, state-of-the-art (SOTA) linear attention architectures. Like AlphaGo's Move 37 that revealed unexpected strategic insights invisible to human players, our AI-discovered architectures demonstrate emergent design principles that systematically surpass human-designed baselines and illuminate previously unknown pathways for architectural innovation. Crucially, we establish the first empirical scaling law for scientific discovery itself--demonstrating that architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process. We provide comprehensive analysis of the emergent design patterns and autonomous research capabilities that enabled these breakthroughs, establishing a blueprint for self-accelerating AI systems.


Here are some thoughts:

ASI-ARCH is a revolutionary AI system that helps other AIs invent new ways to learn. Instead of humans telling AI what to do, ASI-ARCH figures out new designs for AI brains all by itself. This means AI can now discover breakthroughs much faster than humans can, speeding up how quickly we make smart AI. It's like an "AlphaGo Moment" because the AI is showing us new, better ways to build AI that we hadn't thought of before, which can be terrifying.