Scaling Laws — Emergent Performance on Unseen Tasks

Training data contamination. For closed models, training data content is not fully disclosed. When a model performs well on a benchmark, the possibility that benchmark instances appeared in training data cannot be fully excluded. This mechanism creates a structural limitation on the evidential value of benchmark performance gains: positive results are always partially contestable on contamination grounds. The mechanism is more severe for the claim tracked here than for most, because the claim specifically asserts performance on previously unseen tasks — contamination directly challenges whether any given positive result satisfies the "previously unseen" criterion. The contamination resistance mechanism is not eliminable for closed models; it can only be reduced by purpose-built anti-contamination evaluation designs.

Metric-dependence of apparent emergence. As documented by Schaeffer et al. (INST-004), apparent discontinuities in scaling curves are sensitive to metric choice. What appears as an emergent ability under one evaluation metric may appear as a smooth continuous improvement under another. This creates a resistance mechanism specific to the claim's language: "increases performance" can be measured in multiple ways, and the picture of what scaling achieves differs depending on measurement choice. The mechanism does not deny that scaling helps; it complicates how and when improvement is observed, which is a genuine interior difficulty for the claim's evidential base.

No agreed definition of "previously unseen." The claim requires that performance improvements occur on tasks the model has not been specifically trained on. But "previously unseen" is ambiguous across several dimensions: whether a task type appeared in training data, whether specific instances appeared, whether structural analogues appeared, and whether the model's in-context learning constitutes encountering the task during inference rather than training. Without an agreed operational definition, positive and contesting evidence cannot be cleanly compared. This is a bottleneck because it cannot be resolved by more experiments — it requires definitional agreement that currently does not exist. This bottleneck sits at the interior of the record, in measurement methodology, not at the claim identity or closure boundary.

Open-weights models with disclosed training data. The contamination resistance mechanism (RM-001) and the "previously unseen" bottleneck (BN-001) both weaken substantially for models where training data is fully disclosed and verifiable. Open-weights models with documented training sets — LLaMA, Mistral, and similar — permit principled contamination analysis. As the open-weights ecosystem matures and evaluation methodology improves, the evidentiary base for the claim can become cleaner. The attractor is not a single experiment but a methodological development: a corpus of models where the "previously unseen" criterion can be operationally verified. This is an interior attractor — it resolves an evidence quality problem, not a boundary ambiguity.