Large language models can perform multi-step reasoning that generalises beyond memorised training examples.
Evidence is materially increasing confidence in—or pressure on—the claim; significant uncertainty remains. State definitions
No evidence has been admitted since this assessment.
Retained questions may have been narrowed or reframed by the current assessment.
Verification position derived from the record’s assessments; dates show when Faultline first recorded each stage.
Normal Record Review is the Observatory’s assessment process for newly considered evidence, not external scientific peer review. LPR identifies an internal source-review record. PROG identifies a programme: AI (Artificial Intelligence), AM (Advanced Materials), BT (Biotechnology), or QE (Quantum Engineering).
How the claim could hold, what obstructs it, and possible ways forward. The State Warrant above remains the authoritative current assessment.
Benchmark contamination. For closed models, the content of training data is not publicly disclosed. Every benchmark performance result for a closed model carries an unresolvable uncertainty about whether the model encountered near-identical instances during training. This is a structural resistance mechanism: positive evidence on any established benchmark is always partially contestable on contamination grounds. The mechanism weakens as purpose-built anti-contamination benchmarks (ARC-AGI, FrontierMath) accumulate results, but is not eliminable for closed systems.
Distribution shift sensitivity. Performance on reasoning benchmarks consistently degrades under surface-level modifications that preserve logical structure. This is evidence that model performance tracks training distribution proximity rather than abstract reasoning rules. The mechanism has been robustly documented across multiple models and benchmark families (INST-002, INST-004). Its significance is contested — human performance also degrades under unfamiliar presentation — but it has not been refuted as a characterisation of current model behaviour.
Generalisation threshold is undefined. The claim requires that reasoning "generalises beyond memorised training examples" but no agreed standard specifies what degree of generalisation is sufficient, what comparison class defines memorisation, or what problem distribution constitutes "beyond training." Different researchers apply different implicit thresholds, producing genuine disagreement from shared evidence. This bottleneck differs from BN-001 in FR-QE-0002: it is not a domain scope ambiguity but a threshold ambiguity within a single domain. It is not resolvable by further evidence alone; it requires definitional consensus that does not currently exist.
Historical narrative recorded for this claim. It does not override the current State Warrant.
Questions retained in this record. The current State Warrant may have narrowed or reframed them. Recorded dates refer to question history, not necessarily the age of the current wording; known revision dates are shown separately.
Does ARC-AGI performance at 87.5% constitute sufficient evidence to confirm the claim, or does confirmation require performance across a wider distribution of anti-contamination benchmarks, including those not yet constructed?
Question recorded 2024-01-15Does the o1/o3-class computation support generalisation beyond memorised or structurally familiar training examples, and what mechanism accounts for that performance? Separately, does its verbalised chain-of-thought faithfully describe the computation producing its answers? AS-002 and IN-006 require these questions to be distinguished: an unfaithful trace does not by itself establish failure to generalise, while a faithful trace does not by itself establish generalisation or decide between the mechanism hypotheses.
Question recorded 2024-01-15Does the distribution shift sensitivity documented in INST-002 persist in o1/o3-class models? If it does, the claim must be scoped to specific problem classes rather than stated at the class level.
Question recorded 2024-01-15Can an agreed operational definition of "generalisation" be established that is acceptable to both the capability-affirming and capability-contesting research communities? Without this, BN-001 cannot be closed regardless of further evidence.
Question recorded 2024-01-15Is this a class-level claim or a system-level claim? The record tracks LLMs as a category, but the strongest positive evidence is from a specific architecture (o1/o3). If the generalisation mechanism is specific to the extended chain-of-thought training regime, the claim as stated may be too broad.
Question recorded 2024-01-15| Mutation | Date | Field | Prior value | Current value |
|---|---|---|---|---|
| M-019 | 2026-10-25 | Source reviewprovenance_review | LPR-001-D33 | LPR-001-D61What this update recorded for M-019Second-cycle Legacy Provenance Review completed as PASS across all seven existing evidence instances. IN-001, IN-002, IN-004, IN-005, IN-006 and IN-007 were re-verified against their recorded sources; IN-003 remains explicitly and correctly retained as an unverifiable legacy report rather than attributable evidence. Existing structured provenance is sufficient; no source was guessed and no routine enrichment was required. No new factual, interpretive, attribution or verification-stage discrepancy was identified. Potential later reasoning-capability reports encountered during source checking were not admitted through LPR-001 and do not alter this provenance closure. Canonical outcome verified / not_required; Pressure State and Verification Stage unchanged. |
| M-018 | 2026-10-24 | Editorial correctioneditorial_correction | AT-001 — Extended chain-of-thought as mechanism candidate | RETIRED — substantive question retained in OQ-002 / BN-001 / governed assessmentsWhat this update recorded for M-018Operator-authorised bounded redundancy retirement following FR-AI-0001 AT-001 Redundancy Review. Remove current AT-001 because it duplicates the governed generalisation/mechanism question already carried by OQ-002, BN-001 and assessments. No new evidence, reassessment, claim change, Pressure State change, Verification Stage change or historical assessment rewrite. Historical references to AT-001 remain append-only history. Receipt: docs/reviews/FR-AI-0001-AT-001-RETIREMENT-2026-10-24.json. |
| M-017 | 2026-10-19 | Reference correctionreference_corrected | Existing source-review limitation not fully exposed in evidence-list status | Explicit reviewed evidence status on IN-003What this update recorded for M-017Operator-authorised corpus reading audit Batch 3 / R02. Render the existing entry-level review disposition from explicit status metadata linked to its retained review and mutation. A withdrawn attribution is not a retracted publication; missing sources and Partial polarity do not automatically determine status. Preserve all original titles, descriptions, citations, counts, assessments, review markers and prior mutations. No substitute source, new evidence or reassessment. Receipt: docs/reviews/EVIDENCE-STATUS-BATCH-3-2026-10-19.md. |
| M-016 | 2026-10-19 | Reference correctionreference_corrected | Generic historical assessment targets retained without structured links | Reviewed associations: M-007 → AS-002What this update recorded for M-016Operator-approved bounded metadata correction. Add source-backed associations in assessmentMutationReferences.js; preserve all original mutations, dates, assessments and current stage/state. This is not a new assessment or evidence admission. Receipt: docs/reviews/LEGACY-ASSESSMENT-REFERENCE-CORRECTION-2026-10-19.md. |
| M-015 | 2026-10-12 | Assessment issuedassessment_issued | AS-002 | AS-003What this update recorded for M-015Operator-authorised evidence-gap reassessment considers IN-007 admitted 2026-09-29. AS-003 appended at ESCALATING / VS-03; current posture reaffirmed. No prior assessment, claim, evidence, mechanism, lineage, open question or provenance-review metadata changed. Review receipt: docs/reviews/EVIDENCE-GAP-REASSESSMENT-2026-10-12.md. |
| M-014 | 2026-10-08 | Reference correctionreference_corrected | Investigated clarification candidate and misnamed mechanistic-study model | Bounded wording aligned with source limits and existing assessmentWhat this update recorded for M-014Editorial Correction (GP-001), investigated clarification repair: openQuestions.1.question, instances.5.description corrected after source checks. IN-006 changes only Claude 3.5 Sonnet to Claude 3.5 Haiku; no other instance wording or provenance changed. No new evidence, reassessment, claim-scope change or provenance-cycle completion. Claims, assessment histories/current judgments, status, question IDs and raised dates, review metadata and prior mutation entries preserved. Previous values and source basis: docs/reviews/MCP-CLARIFICATION-REPAIR-2026-10-08.json. View correction receipt (fields, prior wording and source basis) ↗ |
| M-013 | 2026-09-29 | Instances appendedinstances_appended | — | IN-007What this update recorded for M-013Normal Record Review, retrospective route authorized by operator. Added OpenAI-reported IMO 2025 gold-medal-level performance as bounded supportive evidence, with first-party retrospective reporting and official IMO gold threshold provenance. Limitations on independent score validation and evaluation detail are explicit. Claim, historical assessments, current ESCALATING / VS-03 state, mechanisms and open questions unchanged. |
| M-012 | 2026-09-29 | Provenance repairprovenance_repair | LPR-001-D33 pending | LPR-001-D33 completedWhat this update recorded for M-012Operator-approved bounded correction completed. IN-003 now labels the unattributed GPT-4/IMO/AIME account as an unverified legacy report and no longer presents it as supportive evidence; no source was inferred. IN-006 now distinguishes behavioral chain-of-thought faithfulness studies from Lindsey et al.'s mechanistic-interpretability analysis, narrows the evidence's relation to OQ-002, and reflects the 2023–25 source chronology. Existing structured sources remain. Historical assessments, pressure state ESCALATING, verification stage VS-03, mechanisms, lineage and open questions remain unchanged. No new evidence was admitted. |
| M-011 | 2026-09-06 | Description restoreddescription_restored | Legacy ingestion cutoffs: mechanisms:RM-001, mechanisms:RM-002, mechanisms:BN-001, mechanisms:AT-001, lineage:2017–20, lineage:2022, lineage:2023, lineage:2024 | Source-restored complete descriptionsWhat this update recorded for M-011Editorial Correction (GP-001), RENDER-PILOT-001 content restoration: restored RM-001, RM-002, BN-001, AT-001; lineage 2017–20, 2022, 2023, 2024 from FR_AI_0001_LLM_reasoning_generalisation.html (Drive file 1P9qpi6wq4V6nyn4ZpgqiYiz-cVrl9pIq). Each damaged value was a verified prefix of the recovered source after the existing FR-MF to FR-AM identifier migration. Restored the omitted remainder using the original converter text normalization; no inferred completion. Existing later corrections retained. Restoration recovers historical wording and does not reaffirm it as the current assessment. Assessments, evidence instances, open questions, claim, status and rendering eligibility unchanged. Source hashes and field receipt: docs/reviews/render-pilot-content-restoration.json; current-assessment compatibility review: docs/reviews/RENDER-PILOT-001-CONTENT-RESTORATION.md. |
| M-010 | 2026-08-31 | Provenance review completedprovenance_review_completed | — | LPR-001-D02What this update recorded for M-010Legacy provenance review completed. IN-001, IN-002, IN-004 and IN-005 corrected for source fidelity and enriched with structured sources[]; IN-003 remains deliberately unenriched because its intended source could not be confidently established. No assessment or evidence-state change. |
| M-009 | 2026-08-29 | Provenance enrichedprovenance_enriched | — | PROVENANCE-ENRICHEDWhat this update recorded for M-009PA-002 Provenance Enrichment: structured sources[] added to IN-005 and IN-006; evidentiary prose and assessment unchanged. |
| M-008 | 2026-07-09 | Description reordereddescription_reordered | — | DESCRIPTION-REORDEREDWhat this update recorded for M-008Editorial Correction (GP-001): IN-006 description reordered per EP-001 — existing closing synthesis sentence moved to opening, no wording added or removed. |
| M-007 | 2026-06-27 | Assessment issuedassessment_issued | — | ASSESSMENT-ISSUEDWhat this update recorded for M-007ASSESSMENT-002 issued. Pressure state: ESCALATING (sustained). Triggering instance: INST-006. Part of RELEASE-004 / TRIAL-001. |
| M-006 | 2026-06-27 | Instances loggedinstances_logged | — | INSTANCES-LOGGEDWhat this update recorded for M-006INST-006 added (chain-of-thought faithfulness research; OQ-002 sharpened, not closed). |
| M-005 | 2024-01-15 | Mechanisms recordedmechanisms_recorded | — | MECHANISMS-RECORDEDWhat this update recorded for M-005RM-001, RM-002 (Resistance); BN-001 (Bottleneck); AT-001 (Attractor) added. |
| M-004 | 2024-01-15 | Assessment issuedassessment_issued | — | ASSESSMENT-ISSUEDWhat this update recorded for M-004ASSESSMENT-001 issued. Pressure state: ESCALATING. Triggering instance: INST-005. |
| M-003 | 2024-01-15 | Scope note addedscope_note_added | — | SCOPE-NOTE-ADDEDWhat this update recorded for M-003Class-level claim notice added. Definitional dispute recorded as property of frontier. |
| M-002 | 2024-01-15 | Instances loggedinstances_logged | — | INSTANCES-LOGGEDWhat this update recorded for M-002INST-001 through INST-005 added. |
| M-001 | 2024-01-15 | Record createdrecord_created | — | RECORD-CREATEDWhat this update recorded for M-001FR-AI-0001 opened. Programme: PROG-AI. First record in AI series. |
Logged entries are reports on record, not counts of independent studies or patients. Several entries can concern the same underlying evidence. Supportive and Contesting describe an entry’s relation to the claim. Partial indicates limited or mixed bearing; read the details for limitations or legacy status. These labels are not study-quality ratings.
Wei et al. (2022, NeurIPS) demonstrate that prompting large language models with intermediate reasoning steps ("chain-of-thought") substantially improves performance on arithmetic, commonsense, and symbolic reasoning benchmarks. On GSM8K, PaLM 540B improves from 17.9% accuracy under standard prompting to 58.1% with chain-of-thought prompting. The paper presents this as evidence that chain-of-thought can elicit multi-step reasoning in sufficiently large language models. The benchmark result does not by itself resolve whether the underlying capability generalises beyond memorised or structurally familiar training examples.
Shi et al. (2023) introduce GSM-IC, a variant of grade-school mathematics problems that adds irrelevant information to the problem description while preserving the underlying arithmetic task. They find that large-language-model problem-solving accuracy can decrease dramatically when irrelevant context is included, and test mitigation approaches including self-consistency and explicit instructions to ignore irrelevant information. The result is direct evidence of distractibility under a bounded distribution shift; it does not establish that all surface-form changes produce the same degradation or by itself distinguish lack of abstract generalisation from other robustness failures.
Unverified legacy attribution: the source and reported result could not be traced. This entry does not establish the claimed finding.
The earlier record asserted that independent evaluators tested GPT-4 on IMO 2023 and AIME 2023 problems published after its training cutoff and found it solved a non-trivial fraction. The intended evaluations, exact problems, scoring result, and basis for the attribution to researchers at MIT and elsewhere could not be identified from available sources. This is retained as an unverified legacy report, not as attributable evidence that GPT-4 solved post-cutoff problems. Without a traceable source, the claimed novelty and training-contamination status cannot be assessed.
Dziri et al. (2023, NeurIPS) investigate transformer limits across three representative compositional tasks: multi-digit multiplication, logic-grid puzzles, and a classic dynamic-programming problem. They formulate these tasks as computation graphs, decompose them into intermediate sub-procedures, and report empirical results consistent with models reducing multi-step compositional reasoning to linearized subgraph matching rather than developing systematic problem-solving skills. Their theoretical analysis further shows how autoregressive-generation performance can rapidly decay as task complexity increases. This is strong contesting evidence about systematic compositional generalisation, but it is bounded to the studied task families and does not establish a universal inability to generalise.
ARC Prize reports that OpenAI's December 2024 o3-preview system scored 75.7% on the ARC-AGI-1 Semi-Private Evaluation set under the public-leaderboard compute limit and 87.5% in a low-efficiency configuration using 1,024 samples and roughly 172 times the compute. ARC Prize also reports that OpenAI trained the tested o3-preview system on 75% of the ARC-AGI-1 Public Training set and had not disclosed enough detail to determine how much of the result was attributable to that exposure. The result remains strong supportive evidence of adaptation to novel ARC tasks, but the 87.5% figure should not be read as an efficiency-neutral or contamination-free measure of generalisation. Whether the system's test-time search and extended reasoning constitute a qualitatively different general mechanism remains unresolved.
Studies of chain-of-thought faithfulness identify cases where a stated reasoning trace fails to reflect computation used to produce an answer. Chen et al. (2025) and Arcuschin et al. (2025) examine contemporary reasoning models, while Lanham et al. (2023) and Turpin et al. (2023) provide earlier behavioral evaluations of chain-of-thought faithfulness. Lindsey et al. (2025) use mechanistic interpretability to identify cases in Claude 3.5 Haiku where answer-relevant information is read directly from the prompt rather than from preceding chain-of-thought text. Together, this work establishes faithfulness as a separate issue when interpreting reasoning traces. Only Lindsey et al. is mechanistic-interpretability research, and these studies do not resolve whether o1/o3-class models implement a qualitatively different reasoning mechanism or scaled pattern matching. The evidence informs OQ-002 without deciding either mechanism hypothesis, and adds faithfulness as a distinct axis alongside contamination (RM-001) and distribution-shift sensitivity (RM-002).
OpenAI reported on 20 February 2026 that, in July 2025, a general-purpose reasoning model achieved gold-medal-level performance on the International Mathematical Olympiad, scoring 35/42. The official IMO 2025 results set the gold-medal threshold at 35 points. This is bounded supportive evidence of high performance on competition-style mathematics as reported by OpenAI. The sources recorded here do not independently validate a model score or provide enough evaluation detail to assess model identity, training exposure, inference and compute conditions, or scoring and independent grading. The result therefore does not by itself establish generalisation beyond memorised examples or resolve the record's class-level claim.