Scaling Laws — Emergent Performance on Unseen Tasks

Kaplan et al. (OpenAI, 2020) establish empirical power-law relationships between language-model cross-entropy loss and model size, dataset size, and training compute, with some trends spanning more than seven orders of magnitude. The measurements are on held-out text prediction and therefore demonstrate predictable improvements in language-modelling loss as scale increases. They do not directly test performance on previously unseen tasks, nor do they establish that the observed held-out-loss improvements reflect task generalisation rather than memorisation. This is foundational evidence for the scaling mechanism that later task-level studies build on, but only indirect evidence for this record's task-generalisation claim.

Brown et al. (2020) demonstrate that GPT-3 (175B parameters) performs a wide range of evaluated tasks from instructions and/or few examples without gradient updates or task-specific fine-tuning — in-context learning. Performance on translation, arithmetic, commonsense reasoning, and other evaluations improves with scale in the reported settings. This is strong supportive evidence that scaling improves task-agnostic few-shot/in-context performance without task-specific parameter updates. It does not establish that the evaluated task types, specific instances, or relevant structural analogues were absent from GPT-3's pretraining corpus, so it does not by itself prove performance on previously unseen tasks in the stronger training-data sense tracked by BN-001. The mechanism is debated — including whether in-context learning constitutes task performance or sophisticated pattern matching — but the reported evaluation performance is documented across diverse task types.

Wei et al. (2022, TMLR) document that certain capabilities appear discontinuously as model scale increases — absent below a threshold, present above it. Examples include multi-step arithmetic, word unscrambling, and chain-of-thought reasoning. These emergent abilities are not predictable from the smooth scaling curves documented by Kaplan et al.; they appear as sharp transitions. The paper strengthens the claim that scaling produces performance on unseen tasks, but introduces a complication: if emergence is discontinuous, scaling law extrapolation cannot reliably predict when new task performance will appear. This is an interior mechanism question — it does not threaten the claim's truth, but it constrains the predictive power of scaling as a mechanism.

Schaeffer, Miranda, and Koyejo (2023, NeurIPS) present an alternative explanation for apparent emergent abilities. For particular tasks and model families, holding model outputs fixed, they show that nonlinear or discontinuous evaluation metrics can produce apparently sharp emergence while linear or continuous metrics yield smoother, more predictable changes with scale. Their experiments include the InstructGPT/GPT-3 family and BIG-Bench tasks. This is contesting evidence against treating every apparent emergence threshold as a fundamental discontinuity in capability. It does not establish that all scaling improvements are continuous and predictable, nor does it deny that larger models can achieve better task performance.

Golchin and Surdeanu (2023) develop black-box methods for detecting training-data contamination and report evidence that GPT-4 is contaminated with AG News, WNLI, and XSum. This establishes contamination as a material threat to the evidential value of benchmark results when training corpora are undisclosed. It does not establish the legacy claim that scaling gains on MMLU, HellaSwag, and ARC were partially caused by test-set overlap, nor does this source establish a general benchmark-saturation trend. The instance therefore contests the strength of benchmark-based generalisation evidence, rather than demonstrating that scaling-driven performance gains are themselves contamination-driven.

Hoffmann et al. (DeepMind, 2022) study compute-optimal language-model training and find that model size and training-token count should be scaled together under a fixed compute budget. Their 70B-parameter Chinchilla model, trained on substantially more data, outperforms much larger models including Gopher and GPT-3 across a broad range of downstream evaluations. The result refines the allocation of training scale between parameters and data and shows that additional data can materially improve downstream performance at fixed training compute. It does not independently establish that those gains are on genuinely previously unseen tasks or resolve contamination and task-novelty questions central to this record.

Snell et al. (2024) study inference-time computation as a distinct scaling axis. They evaluate search with process-based verifier reward models and adaptive modification of a model's response distribution, finding that the effectiveness of test-time compute depends on prompt difficulty. A compute-optimal strategy improves test-time scaling efficiency by more than fourfold relative to a best-of-N baseline, and in FLOPs-matched evaluations a smaller model with additional test-time compute can outperform a model 14 times larger on problems where the smaller model already has non-trivial success. This establishes test-time compute as a real capability-scaling mechanism. It does not establish the broader legacy bundle about o1/o3, DeepSeek-R1, an industry consensus shift, training-scale saturation, or performance specifically on uncontaminated previously unseen tasks. Because this record's claim is explicitly about scaling language-model training, the result also sits partly outside the claim's present scope rather than directly supporting it.