AI-assisted medical diagnosis achieves specialist-level accuracy on defined imaging tasks.
Different aspects of the evidence point in different directions or support competing explanations. State definitions
No evidence has been admitted since this assessment.
Retained questions may have been narrowed or reframed by the current assessment.
Verification position derived from the record’s assessments; dates show when Faultline first recorded each stage.
Stage under review. AS-004 assigns VS-05 using evidence already on this record. The July ruling covers the initial assessment only. A bounded review has examined whether the current evidence supports Operation; an operator decision is pending. The displayed stage is unchanged pending that decision.
Editorial rendering for clarity; the assessment is unchanged.
This is a verification-stage correction using evidence already admitted, not a newly observed clinical advance. The claim concerns specialist-level accuracy on defined imaging tasks. The completed MASAI trial (IN-006) applied AI-supported reading to a real population mammography programme and demonstrated non-inferior interval-cancer performance, higher sensitivity and unchanged specificity against standard double reading. That supports Operation (VS-05) for a defined clinical workflow under the ratified evidence-depth meanings. The July 2026 stage review likewise distinguished defined-task operation from general deployment robustness. The later source corrections withdrew the inference that regulatory clearance alone proves clinical accuracy; they did not amend the claim into a universal promise across all imaging domains. AS-002 and AS-003 retained VS-03 without reconciling that distinction. VS-05 now replaces that inherited current-stage assignment. FRAGMENTING remains: success in one Swedish mammography programme does not establish dependable transfer across imaging domains, populations, acquisition systems and implementations. Operation is not a finding that the broad generalisation problem is resolved. No new evidence instance is admitted, and no historical assessment is rewritten.
Operator-authorised record-reading coherence repair, 18 October 2026. Basis: the ratified VS semantic ruling and Legacy VS Assignment Review of 22–23 July 2026; existing MASAI evidence IN-004 and IN-006, including Gommers et al., The Lancet, doi:10.1016/S0140-6736(25)02464-X. This correction preserves the historical VS-03 assessments and the ratified historical review. The initial assessment's converter date is not treated as a verified date of clinical operation.
Normal Record Review is the Observatory’s assessment process for newly considered evidence, not external scientific peer review. LPR identifies an internal source-review record. VSC identifies a verification-stage challenge and its review reference. RN identifies an institutional note. PROG identifies a programme: AI (Artificial Intelligence), AM (Advanced Materials), BT (Biotechnology), or QE (Quantum Engineering).
How the claim could hold, what obstructs it, and possible ways forward. The State Warrant above remains the authoritative current assessment.
Research-dataset performance is an imperfect proxy for clinical deployment performance. The claim is defined around accuracy on imaging tasks, but the evidential meaning of an AUC, sensitivity, specificity or F1 result depends on the population, acquisition environment, reference standard and workflow in which it is measured. Cross-site studies show that performance can change under distribution shift, while prospective trials such as MASAI show that some AI-supported workflows can succeed clinically. The bottleneck is therefore not whether prospective clinical success is possible — MASAI now demonstrates that it is — but how reliably performance transfers across imaging domains, populations, acquisition environments and implementations.
Distribution shift and confounding can degrade transfer. Medical-imaging models may exploit correlations associated with acquisition site, equipment, population or workflow that do not remain stable elsewhere. Zech et al. directly demonstrate site-associated confounding in chest radiography, and cross-dataset work shows domain shift can reduce performance. This is a documented resistance mechanism, not a claim that every deep-learning imaging system necessarily fails for the same reason.
A recorded possible resolution path, not automatically a sufficient test for closing the claim.
Replicated prospective multi-site clinical performance across distinct imaging domains, populations and implementations. MASAI now provides mature randomised evidence that an AI-supported mammography workflow can maintain or improve clinically meaningful accuracy in population screening. Resolution of the record's deployment-depth question requires comparable evidence beyond a single domain and programme: stable performance across diverse sites, patient populations, acquisition systems and clinical workflows, with patient-relevant outcomes and independent replication.
History labels compare stages originally stored at each assessment. Retrospective reviews are shown separately.
Assessments on the same date follow their assessment ID order. An evidence-basis correction and a review of new evidence can be separate events.
Editorial rendering for clarity; the assessment is unchanged.
This is a verification-stage correction using evidence already admitted, not a newly observed clinical advance. The claim concerns specialist-level accuracy on defined imaging tasks. The completed MASAI trial (IN-006) applied AI-supported reading to a real population mammography programme and demonstrated non-inferior interval-cancer performance, higher sensitivity and unchanged specificity against standard double reading. That supports Operation (VS-05) for a defined clinical workflow under the ratified evidence-depth meanings. The July 2026 stage review likewise distinguished defined-task operation from general deployment robustness. The later source corrections withdrew the inference that regulatory clearance alone proves clinical accuracy; they did not amend the claim into a universal promise across all imaging domains. AS-002 and AS-003 retained VS-03 without reconciling that distinction. VS-05 now replaces that inherited current-stage assignment. FRAGMENTING remains: success in one Swedish mammography programme does not establish dependable transfer across imaging domains, populations, acquisition systems and implementations. Operation is not a finding that the broad generalisation problem is resolved. No new evidence instance is admitted, and no historical assessment is rewritten.
Operator-authorised record-reading coherence repair, 18 October 2026. Basis: the ratified VS semantic ruling and Legacy VS Assignment Review of 22–23 July 2026; existing MASAI evidence IN-004 and IN-006, including Gommers et al., The Lancet, doi:10.1016/S0140-6736(25)02464-X. This correction preserves the historical VS-03 assessments and the ratified historical review. The initial assessment's converter date is not treated as a verified date of clinical operation.
Historical narrative recorded for this claim. It does not override the current State Warrant.
Questions retained in this record. The current State Warrant may have narrowed or reframed them. Recorded dates refer to question history, not necessarily the age of the current wording; known revision dates are shown separately.
BN-001 is the first measurement validity bottleneck in PROG-AI. RN-005 was developed from PROG-BT and PROG-AI (FR-AI-0007) evidence. Does its appearance here in an applied deployment claim — rather than a frontier capability claim — suggest that measurement validity is a broader AI phenomenon than previously established, or does it reflect a property specific to the benchmark-deployment gap in applied AI?
Question recorded 2024-01-15The deployment generalisation gap is a depth question structurally different from foundational uncertainty. Is this a new depth-layer category within PROG-AI, or is it the same surface/depth inversion described differently? If AI systems cannot reliably generalise their demonstrated capabilities to deployment environments, that is a depth failure even for surface claims.
Question recorded 2024-01-15MASAI now demonstrates mature prospective success in one population-screening workflow. What breadth of replication is required to close BN-001: independent multi-site replication within mammography, successful transfer across populations and acquisition systems, or comparable prospective evidence across several imaging domains?
Original date uncertain (legacy) · wording revised 2026-09-06| Mutation | Date | Field | Prior value | Current value |
|---|---|---|---|---|
| M-017 | 2026-10-19 | Reference correctionreference_corrected | Existing source-review limitation not fully exposed in evidence-list status | Explicit reviewed evidence status on IN-005What this update recorded for M-017Operator-authorised corpus reading audit Batch 3 / R02. Render the existing entry-level review disposition from explicit status metadata linked to its retained review and mutation. A withdrawn attribution is not a retracted publication; missing sources and Partial polarity do not automatically determine status. Preserve all original titles, descriptions, citations, counts, assessments, review markers and prior mutations. No substitute source, new evidence or reassessment. Receipt: docs/reviews/EVIDENCE-STATUS-BATCH-3-2026-10-19.md. |
| M-016 | 2026-10-19 | Reference correctionreference_corrected | IN-004 / IN-006 MASAI relationship implicit in descriptions and sources | IN-006 explicitly linked to IN-004 as a report from the same trialWhat this update recorded for M-016Make the already recorded MASAI same-trial relationship explicit and machine-readable. Preserve both reports, descriptions, sources, chronology, vectors and all assessments. This is an editorial relationship correction, not new evidence, replication or a stage decision. |
| M-015 | 2026-10-18 | Verification Stage challenge openedverification_stage_challenge_opened | AS-004 / stage presented without an active challenge | VSC-001 / AS-004 stage under review; operator decision pendingWhat this update recorded for M-015Operator-authorised bounded challenge and interim disclosure. No assessment, historical overlay, stage, Pressure State or evidence instance changed. The review finding is a recommendation, not a ratified replacement stage. |
| M-014 | 2026-10-18 | Source or history correctionprovenance_correction | OQ-003 legacy raisedDate alone | OQ-003 revisedDate 2026-09-06; legacy date explicitly qualifiedWhat this update recorded for M-014M-011 records the MASAI-related revision of OQ-003 on 6 September 2026. Preserve its stored original date and question text; do not describe the current wording as 2y 9m old. No question was opened or closed. |
| M-013 | 2026-10-18 | Assessment correctionassessment_correction | AS-003 / FRAGMENTING / VS-03 | AS-004 / FRAGMENTING / VS-05What this update recorded for M-013Operator-authorised coherence repair. Apply the ratified defined-task interpretation of Operation to the existing clinical-use evidence rather than conflating verification depth with cross-domain generalisation. No new evidence or pressure-state transition; preserve AS-001–AS-003 and the July stage-review overlay. Initial legacy chronology is explicitly uncertain in the display overlay. |
| M-012 | 2026-10-04 | Source reviewprovenance_review | LPR-001-D08 | LPR-001-D41What this update recorded for M-012Second-cycle Legacy Provenance Review completed as PASS across all six existing evidence instances. Current source-to-claim representations remain source-faithful after the governed D08 correction and subsequent Normal Record Review of IN-006. IN-005 remains explicit unsupported legacy provenance debt and is already withdrawn from the current evidential basis, so no source was guessed or retrofitted. No factual, interpretive, attribution, or verification-stage discrepancy was identified; no genuinely new scientific evidence was admitted or flagged for Normal Record Review. Canonical outcome verified / not_required; Pressure State and Verification Stage unchanged. |
| M-011 | 2026-09-06 | Assessment issuedassessment_issued | AS-002 | AS-003What this update recorded for M-011Normal Record Review of the final MASAI interval-cancer analysis. FRAGMENTING / VS-03 retained, but the positive prospective-deployment pole is materially strengthened. BN-001, AT-001, 2026 lineage and OQ-003 updated to reflect that prospective success is demonstrated in mammography while cross-domain transferability remains unresolved. |
| M-010 | 2026-09-06 | Instance addedinstance_added | IN-005 | IN-006What this update recorded for M-010Normal Record Review admitted Gommers et al. (2026) final MASAI primary interval-cancer analysis with structured primary provenance. New evidence only; no retroactive repair or replacement of legacy instances. |
| M-009 | 2026-09-06 | Assessment and dependencies correctedassessment_and_dependencies_corrected | AS-001 / legacy BN-RM-AT-lineage-OQ wording | AS-002 / corrected dependenciesWhat this update recorded for M-009Append-only AS-002 issued after the operator-approved LPR-001-D08 repair. FRAGMENTING / VS-03 retained on a narrower evidential basis. BN-001, RM-001, AT-001, lineage and OQ-003 aligned to the corrected evidence: regulatory authorisation is not treated as uniform prospective validation; distribution shift is documented but not universalised; foundation-model resolution is no longer presumed. Historical AS-001 preserved. The 2026 final MASAI interval-cancer analysis remains outside this repair as a normal Record Review candidate. |
| M-008 | 2026-09-06 | Source or history correctionprovenance_correction | LPR-001-D08 discrepancies_found | LPR-001-D08 discrepancies_correctedWhat this update recorded for M-008Operator-approved correction of IN-001 through IN-005. Primary structured provenance added where source fidelity was established. IN-001 bounded to the actual landmark benchmark designs; IN-002 separates device-specific FDA authorisation from prospective-validation claims; IN-003 corrects chronology and bounds the generalisation inference; IN-004 replaces conflated STHLM3/NHS wording with verified MASAI prospective randomised evidence; IN-005 is retained as explicit unsupported legacy provenance rather than retrofitted to substitute evidence. No new scientific evidence admitted through LPR-001. |
| M-007 | 2026-09-06 | Source reviewprovenance_review | — | LPR-001-D08What this update recorded for M-007Legacy provenance review completed. All five evidence instances examined. Material source-fidelity discrepancies identified in IN-001 through IN-005, so no structured provenance was silently added to those instances. One genuinely new scientific result — the 2026 final MASAI interval-cancer analysis — was identified and held out for normal Record Review rather than admitted through LPR-001. No factual, interpretive, assessment, pressure-state, or verification-stage wording changed. Review marked pending governed correction. |
| M-006 | 2026-07-09 | Description reordereddescription_reordered | — | DESCRIPTION-REORDEREDWhat this update recorded for M-006Editorial Correction (GP-001): IN-002, IN-003, IN-004, IN-005 descriptions reordered per EP-001 — existing closing synthesis sentence moved to opening, no wording added or removed. |
| M-005 | 2024-01-15 | Diagnosis helddiagnosis_held | — | DIAGNOSIS-HELD |
| M-004 | 2024-01-15 | Mechanisms recordedmechanisms_recorded | — | MECHANISMS-RECORDED |
| M-003 | 2024-01-15 | Assessment issuedassessment_issued | — | ASSESSMENT-ISSUED |
| M-002 | 2024-01-15 | Instances loggedinstances_logged | — | INSTANCES-LOGGED |
| M-001 | 2024-01-15 | Record createdrecord_created | — | RECORD-CREATED |
Logged entries are reports on record, not counts of independent studies or patients. Several entries can concern the same underlying evidence. Supportive and Contesting describe an entry’s relation to the claim. Partial indicates limited or mixed bearing; read the details for limitations or legacy status. These labels are not study-quality ratings.
Three landmark studies establish strong performance on defined medical-imaging tasks, but under different datasets and comparator designs. Esteva et al. (2017) trained a skin-lesion classifier on 129,450 images and tested it against 21 board-certified dermatologists on two binary classification tasks, reporting dermatologist-level performance. Gulshan et al. (2016) validated a diabetic-retinopathy algorithm on the EyePACS-1 and Messidor-2 datasets, reporting AUCs of 0.991 and 0.990 and high sensitivity/specificity at prespecified operating points against expert-derived reference standards. Rajpurkar et al. (2017) trained CheXNet on ChestX-ray14 and reported F1 performance above the average of four radiologists on a pneumonia test set. Together these studies support the record's surface claim in bounded research evaluations. They do not establish uniform real-world deployment performance, and the legacy statement that all three used retrospective single-site datasets is withdrawn.
FDA authorisation establishes that AI imaging systems can satisfy device-specific regulatory requirements, but it is not equivalent to universal prospective clinical validation. IDx-DR received De Novo authorisation in April 2018 as a diabetic-retinopathy detection device. Viz.ai Contact was authorised in February 2018 as clinical decision-support software that analyses CT images and alerts specialists to potential large-vessel occlusion; FDA explicitly states that it is not a replacement for full patient evaluation or a standalone diagnosis. Viz.ai's submission included a retrospective study of 300 CT images plus real-world evidence concerning notification time. The large number of later FDA-authorised AI/ML devices, many in radiology, demonstrates regulatory adoption across defined intended uses, but authorisation pathways and evidentiary designs vary. The legacy inference that clearance itself proves prospective specialist-level accuracy is withdrawn.
Evidence predating the legacy 2021–23 dating already showed that strong internal benchmark performance need not transfer unchanged across clinical environments. Zech et al. (2018) evaluated pneumonia models across three hospital systems and found variable cross-site generalisation; the models could identify acquisition site with high accuracy and exploit confounding information. Pooch et al. (2019) evaluated domain shift across major chest-radiograph datasets and found substantial performance degradation when training and testing distributions differed. Nagendran et al. (2020) systematically reviewed 81 non-randomised medical-imaging studies comparing deep-learning systems with clinicians and found few prospective studies or randomised trials, high risk of bias, and limited real-world evidence. These sources support a deployment-generalisation and evidence-quality problem. They do not establish that every failure is caused by a single structural property of deep learning or that all models necessarily learn dataset-specific rather than task-relevant features.
Same trial as IN-006; these are different reports from one trial, not independent replications. View IN-006
MASAI supplies prospective randomised evidence for AI-supported mammography screening in the Swedish national screening programme. In the protocol-defined screening-performance analysis of 105,915 analysed participants, AI-supported screening detected 338 cancers versus 262 under standard double reading, corresponding to 6.4 versus 5.0 cancers per 1,000 screened. Recall and false-positive rates were not significantly higher, while screen-reading workload was reduced by 44.2%. This is strong evidence that an AI-supported workflow can improve a defined imaging-screening process in a real clinical programme. It does not establish that all medical-imaging AI generalises successfully, and it is an AI-supported screening workflow rather than autonomous replacement of specialist diagnosis. The legacy STHLM3-MRI and unspecified NHS-trial attributions are withdrawn because they do not support the instance as written.
No verified source for this legacy attribution; excluded from the current evidence basis.
The legacy instance grouped GPT-4V, Med-PaLM 2 and BioMedCLIP as evidence that medical-imaging foundation models had already demonstrated substantially better cross-domain generalisation and reduced distribution-shift sensitivity than task-specific predecessors. LPR-001-D08 could not verify that bundled claim from the cited evidence. In particular, Singhal et al.'s Med-PaLM 2 work evaluates medical question answering rather than establishing cross-site medical-imaging generalisation. The stronger imaging-generalisation inference is therefore withdrawn from the current evidential basis rather than retrofitted to different studies. IN-005 remains visible as legacy provenance debt and contributes no substantive evidence that foundation models have resolved the deployment-generalisation problem pending separately governed evidence review.
Same MASAI trial as IN-004. This is a later outcome report from that trial, not an independent replication. View IN-004
Gommers et al. (2026) report the protocol-defined primary interval-cancer analysis of the completed MASAI randomised population-based screening trial. Among 105,915 analysed participants, interval-cancer rates were 1.55 per 1,000 in the AI-supported group and 1.76 per 1,000 under standard double reading, meeting the prespecified non-inferiority criterion. Sensitivity was significantly higher with AI-supported screening (80.5% versus 73.8%), while specificity was identical at 98.5%. The AI-supported group also had descriptively fewer interval cancers with unfavourable characteristics, including invasive, T2+ and non-luminal-A tumours. Together with the previously reported reduction in reading workload, this provides strong prospective evidence that one AI-supported mammography workflow can preserve or improve clinically relevant screening accuracy in a real population programme. The result remains domain- and implementation-specific: it used one Swedish screening programme and a defined AI-supported workflow, so it does not establish cross-domain generalisation for medical imaging as a whole.