Scaling Mechanism Coherence — Continuity Across Model Sizes

Olsson et al. (Anthropic, 2022) identify "induction heads" — attention-head circuits that implement a form of pattern completion — as a candidate mechanism for in-context learning across transformer sizes. The paper provides strong causal evidence in small attention-only models, including ablation and mechanistic reverse engineering, while evidence in larger models with MLPs is mainly correlational and partly extrapolative. Induction heads recur across the model sweep and are associated with the onset of in-context learning, supporting mechanistic continuity, but the paper does not causally establish that induction heads explain in-context learning across the full model-size range. This remains important supportive evidence for the claim, with the strength of causal attribution decreasing at larger scale.

Wei et al. (2022) define emergent abilities as abilities absent in smaller models but present in larger models and document behavioural transitions that are not predictable by straightforward extrapolation from smaller-model performance. That result concerns observed capability, not the identity of the underlying mechanism, so behavioural discontinuity does not by itself establish that a new mechanism has appeared. Michaud et al. (2023) propose the Quantization Model, in which knowledge and skills are learned as discrete quanta; the model is intended to reconcile power-law scaling with sudden capability emergence and is supported by toy-data tests plus tentative language-model analyses. It is a mechanistic hypothesis rather than a demonstration that the same mechanism operates across model sizes. Together these sources show that emergence can be modelled without assuming a wholly new mechanism, while leaving the record's cross-scale mechanism-identity claim unresolved.

Elhage et al. (2022) use deliberately simplified toy neural networks to show how sparse features can be represented in superposition, allowing a model to encode more features than it has available dimensions and producing polysemantic neurons. The work demonstrates a concrete representational mechanism and phase changes within the toy setting. It does not compare language models across scale and therefore does not establish the legacy claim that larger language models encode more concepts per neuron while smaller models exhibit less superposition. The result is relevant background for how representation can change as capacity pressure changes, but it provides no direct positive or negative evidence about cross-scale mechanism identity in deployed language models.

Power et al. (2022) document grokking on small algorithmic datasets: networks can reach near-perfect training performance and only much later transition from chance-level to strong generalisation. Nanda et al. (2023) then reverse-engineer grokking in small transformers trained on modular addition, identifying a Fourier-based algorithm and three continuous training phases — memorisation, circuit formation, and cleanup. This is strong mechanistic evidence that an apparently sudden behavioural transition can arise from gradual internal circuit development in the studied small-model setting. Neither paper performs the cross-model-size comparison needed to support the legacy claim that larger models form the same circuits earlier or more reliably. The evidence therefore informs how emergence can occur without demonstrating continuity across model sizes.

The legacy instance attributed a specific bundle of cross-scale representation-geometry findings — greater linear separability, more distinct concept subspaces, and more structured attention in larger models — to unspecified 2023–24 mechanistic-interpretability work. LPR-001-D06 could not confidently reconstruct a source set supporting that exact historical representation. Those claims are therefore withdrawn from the current evidential basis rather than retrofitted to a substitute source. IN-005 remains visible as explicit legacy provenance debt and contributes no substantive evidence for or against cross-scale mechanism continuity pending a normal governed Record Review.

Yang et al. (ICML 2025) identify a three-stage symbolic architecture for abstract rule induction: symbol-abstraction heads, symbolic-induction heads, and retrieval heads. Their experiments span 13 models from four model families and report that the specialized mechanisms become more evident with model capability and scale, while smaller models that perform poorly on the tasks show little evidence of the same specialized mechanism. This is material contesting evidence for a simple continuity claim: for at least one reasoning capability, the evidence is consistent with a structured mechanism becoming established with sufficient scale rather than an unchanged circuit merely strengthening continuously. The result does not settle the record because mechanism identity remains abstraction-dependent and the study covers specific reasoning tasks rather than capabilities generally.

Xu (2026) introduces Subspace PGA to test whether the geometry of intermediate hidden representations is organised around the unembedding readout subspace used for next-token prediction. Across seven Pythia models from 70M to 6.9B, trained on the same data and tokenizer, the study reports a scale-dependent regime: smaller models (hidden dimension at or below 1024) progressively lose predictive alignment in late layers during training, while larger models (dimension at or above 2048) preserve it across intermediate layers. Cross-family checks on Phi-1.5, OLMo-1B, and Gemma-2-2B broadly support the capacity interpretation while also showing architecture-specific variation. The paper attributes the small-model effect to a capacity trade-off in which dominant directions move away from the readout subspace; importantly, removing those directions restores alignment, so the predictive structure is described as masked rather than destroyed. This is direct cross-scale evidence that models of different sizes can organise internal geometry differently while retaining related underlying predictive structure. It therefore pressures a simple 'same mechanism means same organisation' reading, but does not establish a wholly different computational mechanism at larger scale. The result is a 2026 preprint under review and the proposed metric has not yet received independent replication.

Keeman (2026) tests emotion-related internal representations across six model variants spanning Llama-3.2-1B, Llama-3-8B, and Gemma-2-9B in base and instruct forms, using linear probing, causal activation patching, knockout experiments, and representational geometry on keyword-free clinical vignettes. The study reports affect reception — detection of emotionally significant content — with near-perfect probe performance and an early-layer pattern replicated across all six variants, while emotion categorisation is partly keyword-dependent and improves with scale. Causal activation patching indicates that keyword-rich and keyword-free stimuli share affective-salience representation rather than emotion-category identity. This is bounded supportive evidence that a mechanistic component can recur across different model sizes while behavioural/category performance changes with scale. It is not a direct demonstration of the record's full claim because the study does not establish an emotion capability that is absent at small scale and emerges only through scaling, and the compared models also differ by family and training regime rather than forming a controlled single-family scale sweep. The result is a single-author 2026 arXiv preprint and should be treated as unreplicated pending independent confirmation.