← Observatory › The Record › FR-AI-0002
PROG-AI
FR-AI-0002

LLM Knowledge-Work Utility — Economically Valuable Task Performance

Large language models can perform economically valuable knowledge-work tasks with limited human supervision.

EscalatingVS-02·Assessed 2026-10-12

Evidence is materially increasing confidence in—or pressure on—the claim; significant uncertainty remains. State definitions

Where the record stands

Assessed 2026-10-12
Current assessmentEscalating
Evidence since assessment

No evidence has been admitted since this assessment.

Questions on recordView 3 retained questions →

Retained questions may have been narrowed or reframed by the current assessment.

Assessment history
EscalatingPressure State unchanged · latest assessment 2026-10-12
Verification Matrix

Verification position derived from the record’s assessments; dates show when Faultline first recorded each stage.

VS-01
Assertion
—
VS-02
Published evidence
Current from 2024-01-15 — present
VS-03
Audit
—
VS-04
Replication
—
VS-05
Operation
—
Stage first recorded Current verification position Not yet recorded
State Warrant
Current stateEscalatingVS-02
Current assessmentIN-008 adds process-level contesting evidence at the low-supervision agentic boundary. OSWorld-Pro evaluates sequential GUI subgoals and exposes irrelevant actions and click failures that final-output scores can obscure. Its completion metric is not a measure of workplace productivity, economic value or the supervision cost needed to make deployment reliable; the OSWorld comparison is not a like-for-like decline. Together with IN-007's root-cause-analysis failures, it strengthens the case that extended autonomous workflows remain an unresolved boundary of the claim. The corrected controlled writing and coding studies and deployed customer-support evidence continue to support bounded human-reviewed utility, with task- and version-specific limits; legal deployment alone supplies no quantified productivity effect. The assessment remains ESCALATING / VS-02. IN-008 makes RM-002 and OQ-001 more observable without identifying a general supervision threshold, independently replicating professional-work outcomes or closing OQ-002 and OQ-003.
Assessment contextReassessment of IN-008 admitted 2026-09-26, taking the subsequent provenance corrections to IN-003, IN-005 and IN-006 into account. OSWorld-Pro arXiv:2609.24890v1 reviewed 2026-10-12. Benchmark feasibility/judging and Linux task-selection limits, preprint status and NVIDIA author affiliation constrain generalisation. No evidence admission, state transition, stage transition or question closure.

Normal Record Review is the Observatory’s assessment process for newly considered evidence, not external scientific peer review. LPR identifies an internal source-review record. PROG identifies a programme: AI (Artificial Intelligence), AM (Advanced Materials), BT (Biotechnology), or QE (Quantum Engineering).

State entered2024-01-15
Latest assessment2026-10-12
Mechanisms and barriers

How the claim could hold, what obstructs it, and possible ways forward. The State Warrant above remains the authoritative current assessment.

Resistance MechanismRM-001

Hallucination under low supervision. LLMs produce confident, fluent, and factually incorrect output at non-trivial rates. In supervised settings, human review catches and corrects errors before they produce harm. As supervision decreases, uncorrected errors propagate. The mechanism creates a hard constraint on the supervision level at which the claim holds: the claim is true under adequate supervision; the supervision threshold required varies by task stakes. This is not a reason to contest the claim but a reason to specify its domain of validity precisely.

Resistance MechanismRM-002

Agentic error compounding. In single-turn assisted tasks, errors are local and recoverable. In multi-step agentic workflows, errors compound: a mistaken intermediate step propagates through subsequent steps, producing outputs that are difficult to audit and expensive to correct. The mechanism is evidenced by INST-006. It is structurally distinct from hallucination — it is not primarily about factual accuracy but about error propagation in sequential decision-making. As the frontier moves toward agentic deployment, this mechanism becomes the primary constraint on the claim's "limited supervision" condition.

BottleneckBN-001

Knowledge-work task scope is undefined. The claim spans a range from low-stakes, single-turn drafting assistance to high-stakes, multi-step autonomous professional work. The evidence is positive at the low end and negative or qualified at the high end. The bottleneck is not evidential — more evidence will not resolve it — it is definitional. A scoped version of the claim (e.g. restricted to single-turn assisted tasks in low-stakes domains) would already be confirmable. A broader version (including autonomous high-stakes professional work) would not. The Observatory records this without resolving it: the claim as stated spans both, and both are worth tracking.

AttractorAT-001

A recorded possible resolution path, not automatically a sufficient test for closing the claim.

Agentic capability threshold as resolution point. The primary open question for this record is whether LLMs can perform valuable knowledge-work autonomously over extended task sequences — the agentic frontier. Current evidence shows strong performance in supervised single-turn settings and weaker performance in autonomous multi-step settings. If agentic systems demonstrate reliable multi-step task completion in commercially deployed settings, the claim's upper scope boundary moves significantly. This is AT-001: the agentic capability frontier is drawing the evidence trail toward a potential resolution of BN-001. It is not yet reached; it is the attractor the record is currently moving toward.

Assessment History
2024-01-15
AS-001 · Initial assessment — Escalating
The claim is supported by the current evidence in a qualified but meaningful sense. Three independent lines of evidence — controlled experiments (INST-001, INST-002), commercial deployment at scale (INST-003), and natural experiment in deployed settings (INST-005) — all find that LLMs produce measurable economic value in knowledge-work contexts under conditions approximating limited supervision. The effect sizes are not marginal: 14–55% productivity improvements in relevant task domains, with quality improvements accompanying rather than trading off against speed in at least two of the three studies (INST-002, INST-005). Contesting evidence is concentrated at the boundary of the claim rather than at its core: documented hallucination failures in high-stakes domains (INST-004) and agentic multi-step task failures (INST-006) show that the 'limited human supervision' condition holds reliably in single-turn, reviewed contexts but not yet in extended autonomous workflows. The pressure state is ESCALATING: the core claim is well supported within a supervision boundary that has not yet been precisely defined (OQ-001).
AS-001 stage: VS-02 preserved — historically unverified.
2026-08-17
AS-002 · Assessment updated; Pressure State and Verification Stage unchanged — Escalating
The claim remains ESCALATING and bounded by the supervision threshold identified in AS-001. IN-007 adds a stronger operational test at that boundary: on live-system root-cause analysis, general-purpose coding agents achieve low accuracy on realistic and hard tasks, hallucinate causes, and often miss concurrent incidents. This is meaningful contesting evidence against reliable delegation of extended, high-stakes multi-step knowledge work under low supervision. It does not overturn the claim's supported core because the record is not a universal claim of autonomous competence: controlled studies and deployed settings continue to show economically valuable performance where humans review outputs and the task is bounded. IN-007 therefore narrows confidence at the agentic frontier rather than reversing the underlying utility finding. Pressure State remains ESCALATING and Verification Stage remains VS-02; OQ-001 remains the decisive unresolved boundary.
Issued during OHR-2026-09 catch-up review to close the unassessed-evidence gap created by IN-007. The reassessment preserves the distinction between bounded human-reviewed utility and extended low-supervision agentic work.
2026-10-12
AS-003 · Assessment updated; Pressure State and Verification Stage unchanged — Escalating
IN-008 adds process-level contesting evidence at the low-supervision agentic boundary. OSWorld-Pro evaluates sequential GUI subgoals and exposes irrelevant actions and click failures that final-output scores can obscure. Its completion metric is not a measure of workplace productivity, economic value or the supervision cost needed to make deployment reliable; the OSWorld comparison is not a like-for-like decline. Together with IN-007's root-cause-analysis failures, it strengthens the case that extended autonomous workflows remain an unresolved boundary of the claim. The corrected controlled writing and coding studies and deployed customer-support evidence continue to support bounded human-reviewed utility, with task- and version-specific limits; legal deployment alone supplies no quantified productivity effect. The assessment remains ESCALATING / VS-02. IN-008 makes RM-002 and OQ-001 more observable without identifying a general supervision threshold, independently replicating professional-work outcomes or closing OQ-002 and OQ-003.
Reassessment of IN-008 admitted 2026-09-26, taking the subsequent provenance corrections to IN-003, IN-005 and IN-006 into account. OSWorld-Pro arXiv:2609.24890v1 reviewed 2026-10-12. Benchmark feasibility/judging and Linux task-selection limits, preprint status and NVIDIA author affiliation constrain generalisation. No evidence admission, state transition, stage transition or question closure.
Claim Lineage

Historical narrative recorded for this claim. It does not override the current State Warrant.

2020–22
GPT-3 and early commercial deployment. OpenAI API opens GPT-3 for commercial access. Early adopters deploy in copywriting, summarisation, and customer service contexts. Results are mixed; the capability exists but hallucination rates and inconsistency limit reliable deployment. The utility claim is in an EMERGING state: existence demonstrated, economic value unconfirmed at scale.
2022–23
ChatGPT release and mass adoption. ChatGPT reaches 100 million users in two months. Spontaneous mass deployment across knowledge-work contexts — drafting, coding, analysis, research — produces an enormous but methodologically uncontrolled evidence trail. Anecdotal reports of value are ubiquitous; systematic measurement is absent. The deployment scale constitutes a form of evidence but is not sufficient for the Observatory to issue a formal assessment.
2023
Controlled studies and commercial deployments. The Peng et al. and Noy and Zhang studies provide the first rigorous experimental evidence. Simultaneously, major professional services firms (law, consulting, finance) begin commercial deployment. The claim transitions to ESCALATING.
2023–24
Hallucination incidents and agentic frontier. High-profile hallucination failures in legal and medical contexts establish the supervision threshold as a genuine constraint. Agentic deployment attempts reveal the compounding error problem as a distinct resistance mechanism. The claim's boundary conditions become observable.
2023–25
Deployed customer-support study. Brynjolfsson et al. circulate working-paper results in 2023–24 and publish the final peer-reviewed QJE version in 2025. The staggered workplace rollout adds observational evidence of productivity gains in an operating company, with version-specific sample and effect estimates and no significant overall surveyed-satisfaction improvement.
Questions on record

Questions retained in this record. The current State Warrant may have narrowed or reframed them. Recorded dates refer to question history, not necessarily the age of the current wording; known revision dates are shown separately.

OQ-001

Where exactly is the supervision threshold boundary? The claim holds at supervised single-turn assistance and fails at autonomous multi-step high-stakes work. The boundary between these is not characterised. As agentic systems improve, does the boundary move, or is it a structural property of LLM architecture?

Question recorded 2024-01-15
OQ-002

The productivity gains documented in INST-001, INST-002, and INST-005 are concentrated in tasks with clear outputs and measurable quality criteria (code, documents, customer service responses). Do the same gains hold for knowledge-work tasks with harder-to-measure outputs: strategic judgment, novel problem formulation, relationship management?

Question recorded 2024-01-15
OQ-003

FR-AI-0001 (reasoning) and FR-AI-0002 (utility) are both ESCALATING. Is their co-escalation coincidental — both happen to be progressing simultaneously — or is there a causal relationship? If reasoning improvements (FR-AI-0001) drive utility improvements (FR-AI-0002), the two records are dependent. If utility advances independently of reasoning (through better prompting, deployment infrastructure, task design), they are not. This relationship is currently unobserved.

Question recorded 2024-01-15
Mutation Log
MutationDateFieldPrior valueCurrent value
M-0152026-10-24Source reviewprovenance_reviewLPR-001-D32LPR-001-D60
What this update recorded for M-015

Second-cycle Legacy Provenance Review completed as PASS across all eight existing evidence instances. Underlying sources and claim representations verified; all instances already carry structured sources and no provenance enrichment or unverifiable attribution was required. No factual, interpretive, attribution, or verification-stage discrepancy identified. No genuinely new scientific evidence requiring Normal Record Review was identified. Canonical outcome verified / not_required; Pressure State ESCALATING and Verification Stage VS-02 unchanged.

M-0142026-10-12Assessment issuedassessment_issuedAS-002AS-003
What this update recorded for M-014

Operator-authorised evidence-gap reassessment considers IN-008 admitted 2026-09-26. AS-003 appended at ESCALATING / VS-02; current posture reaffirmed. No prior assessment, claim, evidence, mechanism, lineage, open question or provenance-review metadata changed. Review receipt: docs/reviews/EVIDENCE-GAP-REASSESSMENT-2026-10-12.md.

M-0132026-09-29Provenance review completedprovenance_review_completedLPR-001-D32 review_required / pendingLPR-001-D32 discrepancies_corrected / completed
What this update recorded for M-013

Approved bounded correction and closure of the second-cycle provenance review. IN-003 narrowed to attributable 2023 legal deployment without an unmeasured time-saving claim; IN-005 aligned to the 2025 QJE sample, 15% productivity result and distinct sentiment versus surveyed-satisfaction outcomes; IN-006 narrowed to SWE-bench and SWE-agent results without an unsupported human baseline or broad agent-failure claims. Structured primary sources added; lineage date aligned. AS-001 and AS-002 preserved as dated historical assessments, with no new scientific evidence, pressure-state or verification-stage change.

M-0122026-09-26Instance appendedinstance_appendedIN-001–IN-007IN-001–IN-008
What this update recorded for M-012

Normal Record Review of Frontline Scout flag 2026-09-26-01 admitted OSWorld-Pro arXiv:2609.24890v1 as bounded process-level contesting evidence at the low-supervision agentic boundary. AS-002 remains current; Pressure State ESCALATING, Verification Stage VS-02, mechanisms and open questions unchanged. The benchmark does not establish economic productivity or real-world supervision cost.

M-0112026-09-06Description restoreddescription_restoredLegacy ingestion cutoffs: mechanisms:RM-001, mechanisms:RM-002, mechanisms:BN-001, mechanisms:AT-001, lineage:2020–22, lineage:2022–23, lineage:2023–24, lineage:2024Source-restored complete descriptions
What this update recorded for M-011

Editorial Correction (GP-001), RENDER-PILOT-001 content restoration: restored RM-001, RM-002, BN-001, AT-001; lineage 2020–22, 2022–23, 2023–24, 2024 from FR_AI_0002_LLM_knowledge_work_utility.html (Drive file 1kmjQHLUxpNKFNStSoFKR-AE7P-ogFAIZ). Each damaged value was a verified prefix of the recovered source after the existing FR-MF to FR-AM identifier migration. Restored the omitted remainder using the original converter text normalization; no inferred completion. Existing later corrections retained. Restoration recovers historical wording and does not reaffirm it as the current assessment. Assessments, evidence instances, open questions, claim, status and rendering eligibility unchanged. Source hashes and field receipt: docs/reviews/render-pilot-content-restoration.json; current-assessment compatibility review: docs/reviews/RENDER-PILOT-001-CONTENT-RESTORATION.md.

M-0102026-08-30Provenance review completedprovenance_review_completedNo governed provenance-review completion markerLPR-001-D01 completed; discrepancies corrected
What this update recorded for M-010

LPR-001 Day 1 completion recorded after the bounded correction of IN-001, IN-002 and IN-004. The marker records review completion separately from repair outcome so the deterministic oldest/never-reviewed queue can operate. No evidence instance, assessment, Pressure State or Verification Stage was added or changed by this marker.

M-0092026-08-30Reference correctionreference_correctedIN-001 inferred comparable code quality; IN-002 used a secondary 37% speed expression; IN-004 misclassified Omiye et al. as a systematic reviewIN-001 reports task success and the stated code-quality boundary; IN-002 uses the published 40% time reduction and 18% quality increase; IN-004 accurately describes Mata and the bounded Omiye et al. study; structured sources recorded
What this update recorded for M-009

LPR-001 Day 1 Editorial Correction. Corrected the three linked legacy evidence descriptions and added canonical structured provenance. IN-001, IN-002 and IN-004 retain their identifiers, dates and vectors. AS-001 and AS-002 remain preserved; current Pressure State ESCALATING and Verification Stage VS-02 are unchanged.

M-0082026-08-17Assessment issuedassessment_issuedAS-001AS-002
What this update recorded for M-008

AS-002 issued during OHR-2026-09 catch-up review to close the evidence-assessment gap created by IN-007. ORCA-bench strengthens the existing contesting evidence at the low-supervision agentic boundary but does not overturn bounded, human-reviewed knowledge-work utility. Pressure State remains ESCALATING; Verification Stage remains VS-02; mechanisms and open questions remain unchanged.

M-0072026-08-01Instance appendedinstance_appendedIN-006IN-007
What this update recorded for M-007

IN-007 appended — ORCA-bench (arXiv:2607.28545v1), authorised through Post-Scout flag 2026-08-01-02. Bounded CONTESTING evidence strengthens the existing low-supervision agentic boundary with operational root-cause-analysis measurements. Instance only: AS-001 remains current; Pressure State ESCALATING, Verification Stage VS-02, mechanisms, and open questions unchanged. No new assessment issued.

M-0062026-07-09Description reordereddescription_reordered—DESCRIPTION-REORDERED
What this update recorded for M-006

Editorial Correction (GP-001): IN-004 description reordered per EP-001 — existing closing synthesis sentence moved to opening, no wording added or removed.

M-0052024-01-15Programme panel addedprogramme_panel_added—PROGRAMME-PANEL-ADDED
M-0042024-01-15Mechanisms recordedmechanisms_recorded—MECHANISMS-RECORDED
M-0032024-01-15Assessment issuedassessment_issued—ASSESSMENT-ISSUED
M-0022024-01-15Instances loggedinstances_logged—INSTANCES-LOGGED
M-0012024-01-15Record createdrecord_created—RECORD-CREATED
Evidence Sources
8 logged entriesShow sources ↓Hide ↑

Logged entries are reports on record, not counts of independent studies or patients. Several entries can concern the same underlying evidence. Supportive and Contesting describe an entry’s relation to the claim. Partial indicates limited or mixed bearing; read the details for limitations or legacy status. These labels are not study-quality ratings.

IN-001
GitHub Copilot productivity study — Peng et al. (Microsoft Research)
Read evidence details and limits for IN-001

Peng et al. (2023) report a controlled experiment in which 95 professional developers recruited through Upwork were randomly assigned to implement a JavaScript HTTP server with or without GitHub Copilot. Conditioning on completing the task, the Copilot group completed it 55.8% faster. Task success was seven percentage points higher in the treatment group, but the difference was not statistically significant. A 12-check test suite defined successful task completion; the authors explicitly state that the study did not examine code quality. The result is a direct productivity measurement in a bounded software-development task under conditions approximating limited supervision. It is a randomised controlled experiment rather than a retrospective survey, but it does not establish effects on code quality or general software-engineering work.

1. Peng, S., Kalliamvakou, E., Cihon, P., and Demirer, M. (2023), The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, arXiv:2302.06590. · Study Design; Results; Discussion
Supportive
IN-002
Noy and Zhang — Experimental evidence on productivity effects of generative AI in professional writing
Read evidence details and limits for IN-002

A preregistered randomised controlled trial (Noy and Zhang, 2023, Science) assigned occupation-specific writing tasks to 453 college-educated professionals, including marketers, grant writers, consultants, data analysts, human-resource professionals, and managers, with half randomly exposed to ChatGPT. The published study reports that average completion time decreased by 40% and output quality as rated by blinded evaluators increased by 18%. Inequality effects were notable: lower-ability workers benefited more, compressing the quality distribution. The study directly tests the claim across several bounded professional-writing domains under limited-supervision conditions; it does not establish the same effect for knowledge work generally.

1. Noy, S. and Zhang, W. (2023), Experimental evidence on the productivity effects of generative artificial intelligence, Science 381(6654), 187–192. DOI 10.1126/science.adh2586 · Abstract and experimental results
Supportive
IN-003
AI in legal practice — contract review and due diligence deployment
Read evidence details and limits for IN-003

Commercial legal use of LLM-based tools was observable in 2023. Allen & Overy said that about 3,500 lawyers had made around 40,000 Harvey queries during its trial and announced a wider launch; the firm explicitly said that lawyers must carefully review the output. Casetext launched CoCounsel for tasks including document review, legal research and contract analysis, and Thomson Reuters acquired Casetext for $650 million in August 2023. These disclosures establish organisational adoption and commercial commitment in a high-value knowledge-work domain, but do not measure associate time saved, net economic return, or safe performance without careful human review. They provide bounded deployment evidence rather than a controlled productivity estimate.

1. Allen & Overy (2023), A&O announces exclusive launch partnership with Harvey. · 15 February announcement; trial usage and lawyer-review requirement2. Thomson Reuters (2023), Thomson Reuters Completes Acquisition of Casetext, Inc. · 17 August announcement; acquisition value and CoCounsel task descriptions
Supportive
IN-004
Hallucination and reliability failure documentation — legal and medical contexts
Read evidence details and limits for IN-004

These events do not falsify the claim — which does not require error-free performance — but they constitute direct evidence that the "limited human supervision" condition is not safely achievable across all knowledge-work domains. Mata v. Avianca (2023) documents attorneys submitting fabricated judicial opinions and citations generated by ChatGPT; the court imposed sanctions after finding that the respondents abandoned their responsibilities when they submitted and then defended the false material. Separately, Omiye et al. (2023) tested four commercial LLMs with nine questions concerning race-based medicine and repeated each question five times. Every model produced some responses that promoted race-based medicine, racist tropes, or unsubstantiated claims, with outputs varying across repeated runs. This was a bounded empirical study of harmful and inaccurate race-based medical content, not a systematic review of general medical question-answering. Together the sources contest reliable low-supervision use in high-stakes legal and clinical contexts without establishing a universal failure rate.

1. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), Opinion and Order on Sanctions. · Opinion and Order on Sanctions, 22 June 20232. Omiye, J. A. et al. (2023), Large language models propagate race-based medicine, npj Digital Medicine 6, 195. DOI 10.1038/s41746-023-00939-z · Abstract; Results; Discussion
Contesting
IN-005
Brynjolfsson, Li, and Raymond — Generative AI at work (customer service study)
Read evidence details and limits for IN-005

The final peer-reviewed study by Brynjolfsson, Li and Raymond (QJE, 2025) examines the staggered introduction of an AI conversational assistant using data from 5,172 customer-support agents. Access increased issues resolved per hour by 15% on average, with larger gains among less experienced and lower-skilled workers. Transcript-based customer sentiment improved, but the study found no significant overall difference in surveyed customer satisfaction; effects also varied by worker group. This large-scale observational deployment study measures work in an operating company rather than a laboratory task, while its results remain specific to customer support and AI-assisted agents rather than autonomous knowledge work generally. Earlier 2023–24 working-paper versions reported about 14% and a sample of roughly 5,000 agents; those figures must not be presented as the final QJE result.

1. Brynjolfsson, E., Li, D. and Raymond, L. (2025), Generative AI at Work, Quarterly Journal of Economics 140(2), 889–942. DOI 10.1093/qje/qjae044 · Abstract; Table I; surveyed satisfaction results; Section VII.A customer sentiment
Supportive
IN-006
SWE-bench and SWE-agent — bounded autonomous software issue resolution
Read evidence details and limits for IN-006

SWE-bench introduced 2,294 real GitHub issues whose resolution can require coordinated edits across functions, classes and files. Its initial language-model evaluations solved only a small fraction of issues. A 2024 agent-system study, SWE-agent, improved performance but resolved about 12.5% of the full SWE-bench test set in its reported setting. This is bounded evidence that autonomous multi-step software repair remained unreliable on that benchmark in 2024, despite substantial improvement from an agent interface. The studies do not measure performance against human developers, document the broader claimed failure modes of task drift or inappropriate escalation, or establish a general supervision threshold across knowledge work. The instance therefore marks a task-specific agentic boundary rather than a general refutation of valuable human-reviewed assistance.

1. Jimenez, C. E. et al. (2024), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024; arXiv:2310.06770. · Abstract; benchmark construction and baseline evaluations2. Yang, J. et al. (2024), SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024; arXiv:2405.15793. · Abstract; full SWE-bench pass@1 result and agent-interface method
Partial
IN-007
ORCA-bench — live-system root-cause analysis under limited supervision
Read evidence details and limits for IN-007

ORCA-bench provides bounded CONTESTING evidence at the record's established agentic frontier by testing general-purpose coding agents on root-cause analysis in a live OpenTelemetry microservice environment (arXiv:2607.28545v1, submitted July 2026). The benchmark contains 1,079 tasks varying report specificity, detection delay, and concurrent incidents; its ground truth was curated with site-reliability-engineering input, and its automated judge achieved weighted Cohen's kappa of 0.90 against human rescoring. The best reported root-cause-analysis accuracy was 25.3% on realistic medium tasks and 10% on hard tasks. Hallucinated root causes appeared in 7%–40% of non-empty reports, while removing source-code access reduced accuracy by 9–16 percentage points. Agents made partial diagnostic progress but were distracted by louder signals and frequently missed concurrent causes, leaving performance far below reliable autonomous delegation for production incident diagnosis. The result strengthens rather than redraws AS-001's boundary: economically valuable bounded, human-reviewed assistance remains supported, while extended high-stakes multi-step work under low supervision remains unreliable. It does not measure economic value directly, evaluate actual enterprise deployment, or establish the performance of specialised AI-SRE systems. Generalisation is limited by the benchmark harness, prompt design, preprint status, and relevant vendor interest because several authors are affiliated with Traversal, a company offering AI-SRE systems.

1. Gong, A. et al. (2026), ORCA-bench: How Ready Are Language Model Agents for Oncall?, arXiv:2607.28545v1. · Abstract; benchmark construction; evaluation and human rescoring; results and limitations
Contesting
IN-008
OSWorld-Pro — process evaluation of computer-use agents
Read evidence details and limits for IN-008

OSWorld-Pro (arXiv:2609.24890v1, submitted 21 September 2026) offers bounded evidence at the low-supervision agentic boundary by evaluating computer-use agents across 305 Linux GUI tasks with 2,814 sequentially dependent subgoals and 67,264 human-annotated step labels. Its reported task metric is the share of tasks completing all subgoals deemed feasible by human annotators: the highest tabulated model scores 77.7% and Claude Opus 5 scores 75.7%. The paper identifies subgoal-irrelevant actions and click errors that final-output scoring alone may obscure. This supports a process-level reliability constraint in extended multi-app workflows and makes RM-002/OQ-001 more observable, but does not measure workplace productivity, economic value, the amount of human supervision actually required, or performance in deployed professional settings. The cited 83.4% OSWorld score concerns a different benchmark and must not be treated as a like-for-like decline. The preprint's automated subgoal judge agrees with human task-level completion labels more closely than with feasibility/progress labels, and task selection includes uncommon applications and Linux environments; independent replication is not established. Authors are affiliated with NVIDIA, an interested AI infrastructure and agent research vendor. This evidence is distinct from IN-007's live-service root-cause analysis. It reinforces the existing boundary without changing AS-002's ESCALATING / VS-02 assessment.

1. Wang et al. (2026), OSWorld-Pro: Process-based Evaluation for Computer Use Agents, arXiv:2609.24890v1. · Sections 2–6; Tables 1–2; action-level failure analysis
Contesting