Long-Running AI Agents: Reliability, Recovery and Oversight

Long-running AI agents, systems that execute many tool calls over minutes, hours, or days rather than a single conversational turn, have become measurably more capable, but independent research shows their reliability has not kept pace. METR's time-horizon methodology finds that the length of task frontier agents can complete with 50% reliability has doubled roughly every seven months since 2019, reaching around 110 minutes for OpenAI's o3 as of a March 2025 study (^[1]), while its 80%-reliability horizon remains far shorter, showing that occasional long-task success does not mean dependable long-task success. A 2026 Princeton-affiliated study evaluating 14 models across GAIA and tau-bench found that roughly eighteen months of model development produced only small reliability gains even as capability rose, and a large duration-stratified benchmark found mean success rates falling by over 24 percentage points between short and very-long tasks, findings detailed with their primary sources in the Methodology and Benchmark Data sections below.

Recovery mechanisms vary by framework and configured storage. LangGraph uses checkpointers that persist a thread's graph state as checkpoints for functions including fault tolerance; its in-memory saver loses checkpoints when the process restarts (^[2]). Human-in-the-loop oversight is treated as a core design feature rather than a stopgap: Anthropic's own telemetry across roughly one million API tool calls found 73% involved a human in the loop in some form and only 0.8% of actions were irreversible (^[3]). No standardized formula for "intervention rate" yet exists, though a proposed Autonomy Index, detailed below, defines it as the share of task steps completed without human involvement.

Enterprise observability platforms from LangChain, Arize, Datadog, Langfuse, Weights & Biases, OpenAI, AWS, and Microsoft all instrument agent execution at the trace level, but none publishes a standardized "uptime" figure for agentic systems, since a technically running agent can still fail its assigned task (^[4]). This gap between deployment and confidence shows up across enterprise surveys: Gartner projects over 40% of agentic AI projects will be canceled by the end of 2027 (^[5]), Deloitte finds only 21% of organizations report mature agentic governance despite 74% expecting moderate agent usage by 2027 (^[6]), and Capgemini finds trust in fully autonomous agents fell from 43% to 27% year over year even as 90% of respondents viewed human involvement as positive or cost-neutral (^[7]).

This report documents the method behind each of these figures, publishes a comparative benchmark table (METR, GAIA, SWE-bench Verified, WebArena, OSWorld 2.0, tau-bench, AgentBench, and a duration-stratified reliability suite), a comparative table of enterprise observability platforms, and a table of enterprise adoption and trust survey data, so that a reader can trace any number back to its originating source rather than a restated figure. The benchmarks and vendor materials cited here address different tasks, configurations, and implementation choices; together, they support evaluating long-running agents with task-specific reliability measures and deliberately designed human oversight.

Long-running AI agents are systems that operate over minutes, hours, or days rather than a single conversational turn, issuing many tool calls, tolerating interruptions, and resuming work after a crash or a pause for human input. Interest in these systems has shifted from raw task-completion accuracy toward reliability engineering: how an agent behaves the tenth time it runs the same workflow, what happens when a network call fails midway through step 40 of 200, and how much human supervision autonomous operation actually requires in practice. Independent measurement from the nonprofit AI evaluation group METR shows that the length of tasks frontier agents can complete with 50% reliability has doubled approximately every seven months since 2019, reaching around 110 minutes for OpenAI's o3 model as of METR's March 2025 study, detailed with its full methodology in Table 1 below (^[1]). That trend describes rising capability, not rising reliability, and the distinction between the two is the organizing theme of this report.

This report is a companion to, not a restatement of, an earlier analysis published on this site of durable orchestration for agentic workflows, which focused on architectural patterns such as Temporal for coordinating long-running, failure-prone processes across systems (intuitionlabs.ai). Readers seeking a deep treatment of workflow-engine mechanics should consult that piece. This report instead assembles independently reproducible measurements published since: what current benchmarks say about reliability at long time horizons, how open-source and vendor frameworks recover from mid-task failure, how much human intervention autonomous agents actually require in production, and how enterprises are attempting to monitor "uptime" for a class of system that can be technically running while still failing its assigned task.

Every quantitative claim below carries the date its source observed it, because benchmark leaderboards, vendor documentation, and survey figures change quickly; a number reported as current in early 2026 may already be superseded by the time a reader encounters it. As of September 2026, no standards body publishes an agreed definition of "AI agent uptime" or "human intervention rate": this report documents the competing definitions currently in use, traces each to its originating source, and states plainly where the evidence is vendor-reported rather than independently verified.

Reliability research on AI agents separates two questions that demos tend to blur. Capability is whether a model can complete a task at all on its best attempt, typically reported as pass@1; reliability is whether it completes the same class of task consistently across repeated attempts and across increasing task duration (^[8]). A model that scores well on a five-minute version of a task can score far worse on a multi-hour version of the identical task type, which is why single-shot accuracy scores are a poor proxy for production reliability.

METR's headline metric, the 50% time horizon, is defined as the duration of task (measured in how long the task takes a skilled human) that an agent completes successfully half the time; it is explicitly a reliability threshold rather than a best-case capability number, and METR reports the figure separately for each model it evaluates, for example finding Claude 3.7 Sonnet has a time horizon of approximately one hour, distinct from other models' figures, rather than as a single industry-wide constant (^[9]). METR's own data show that its 80%-reliability time horizon has a similar doubling rate to the 50% horizon but is roughly four to six times shorter in absolute duration, meaning models that can sometimes succeed at hour-plus tasks still fail the majority of attempts at that same duration (^[10]). A related benchmark, HCAST, calibrates task difficulty against human completion time directly and finds agents succeed 70 to 80% of the time on tasks a human completes in under an hour, falling to under 20% success on tasks that take a human more than four hours (^[11]).

Two 2026 academic frameworks formalize this measurement further. A Princeton-affiliated paper, "Towards a Science of AI Agent Reliability," defines twelve metrics across four dimensions (consistency, robustness, predictability, and safety) and evaluates 14 models on GAIA and tau-bench, running each task multiple times with different random seeds specifically to separate genuine capability from run-to-run noise. Despite roughly eighteen months of intervening model development, the study finds only small reliability improvements over that period, even as raw capability scores rose (^[12]). A second framework, "Beyond pass@1," constructs a 396-task suite spanning four duration buckets and three domains, then runs 10 open-source models across it with three repeats and two agent scaffolds each, across thousands of repeated episodes, explicitly to measure degradation as task duration increases rather than a single-point accuracy score. Any reader attempting to reproduce reliability numbers should note the assumptions embedded in these designs: fixed temperature settings, a specific scaffold or harness, and a specific task suite, all of which can shift results independent of the underlying model.

A cluster of benchmarks introduced or substantially revised between 2023 and 2026 now targets multi-step, long-horizon reliability specifically rather than single-turn accuracy. Table 1 below summarizes the major public benchmarks referenced in current reliability research, their task counts, and their headline reliability findings.

Several benchmarks in Table 1 report lower success or consistency as task duration, step count, or trial repetition increases; their task designs and domain results differ, so those findings should not be treated as a universal pattern. On the "Beyond pass@1" suite, software-engineering tasks fell from a Graceful Degradation Score of 0.90 on short tasks to 0.44 on very-long tasks, while document-processing tasks stayed nearly flat, from 0.74 to 0.71 (^[22]), suggesting that reliability engineering effort should be prioritized by task type rather than applied uniformly. Vendor-affiliated research groups have begun publishing targeted scaffolding techniques that partially close this gap for specific model and task combinations, though these results remain vendor-reported rather than independently replicated, and should be read alongside the independent benchmark figures in Table 1 rather than in place of them.

Long-running agents fail in ways that short-lived chat sessions do not: context windows fill and truncate ("context rot"), sessions lose state across a container restart, and models have no reliable way to verify their own progress (^[23]). Recent engineering writing groups the resulting failure dynamics into four compounding mechanisms: context drift (relevant information falls out of the working context), hallucination cascades (an early error is treated as fact and built upon), goal drift (the agent's objective slowly mutates across many turns), and a catastrophic failure mode termed "meltdown", whose precursors include entropy spikes, repeated tool calls with slightly varied arguments, and contradictions across consecutive turns (^[24]). The recommended response to meltdown is not to retry from the failure point but to abort, salvage, and restart from a known-good checkpoint; a cleanly aborted run with partial results is treated as a first-class success state, not a failure, in this framing (^[25]).

Recovery architecture is framework- and configuration-specific. In LangGraph, replay from a prior checkpoint re-executes nodes after that checkpoint, including LLM calls, API requests, and interrupts, which may produce different results (^[26]). Vendor implementations of this idea differ in mechanism. Temporal, a workflow orchestration engine, records a full event history of every workflow step and activity call so that a crashed application instance resumes exactly where it left off "implicitly," without the developer writing checkpoint logic (^[27]); a single Temporal Worker process can be tuned to look after hundreds or thousands of such long-running workflows concurrently, evicting inactive ones from memory and reconstituting them from event history on demand (^[28]). LangChain's production agent runtime writes a checkpoint to PostgreSQL after each "super-step" of graph execution, keyed by a persistent thread identifier, so a crashed worker's lease is released and another worker resumes from the latest checkpoint (^[29]); the same runtime supports "time travel," letting an operator select any past checkpoint, optionally edit its state, and resume forward on a forked branch while the original history stays intact (^[30]).

This convergence on durable, replayable state is a caution against treating any single mitigation, including added memory or scratchpad scaffolding, as unambiguously beneficial without dedicated measurement, since scaffolding choices can have counter-intuitive effects on long-horizon reliability that only repeated-trial testing surfaces.

Human-in-the-loop (HITL) design, pausing an agent for human approval, feedback, or correction at defined checkpoints, is treated by major vendors as a core building block of agent architecture rather than an optional safety add-on (^[31]). Anthropic's safety framework states the central design tension explicitly: humans should retain control over how their goals are pursued, particularly before high-stakes or hard-to-reverse decisions are made (^[32]), and its coding agent, Claude Code, defaults to read-only permissions that require explicit human approval before any action that modifies code or systems (^[33]).

Anthropic has published one of the only large-sample, first-party measurements of intervention rate currently available: across roughly one million sampled API tool calls, the company found 80% of tool calls came from agents with at least one kind of safeguard in place, 73% had a human in the loop in some form, and only 0.8% of actions were irreversible (^[3]); human involvement was measured at 87% of tool calls for minimal-complexity tasks versus 67% for high-complexity tasks (^[34]). Separately, Anthropic found that as Claude Code users gained experience, the average number of human interventions per coding session fell from 5.4 to 3.3 while task success rose, a rare before/after intervention-rate data point (^[35]). Anthropic itself cautions that these figures are an upper bound, since its detection classifier over-counts human involvement (^[36]), and that autonomy is an emergent property of model behavior, user oversight strategy, and product design together, not a fixed attribute of a model.

No industry-standard formula for "human intervention rate" currently exists; at least two competing quantitative definitions have been proposed in 2026. A Writer, Inc. white paper defines an Autonomy Index (AIx) as the proportion of task steps completed without human intervention, with a worked example: if 5 of 50 steps require intervention, AIx equals 1 minus 5/50, or 90% autonomy (^[37]). Separately, a five-level, user-centered autonomy taxonomy (Operator, Collaborator, Consultant, Approver, Observer) frames the degree of human involvement as a design decision rather than a capability ceiling; at its "Approver" level, the human is engaged only reactively, when the agent hits a blocker it cannot resolve itself (^[38]). A separate industry proposal extends this idea with a six-level taxonomy for agentic AI, explicitly modeled on the SAE J3016 scale used to classify self-driving car autonomy (^[39]). Microsoft's Agent Framework implements the underlying pattern operationally: when an agent attempts to call a tool marked as requiring approval, the workflow pauses and waits for external input before proceeding (^[40]), with pending approval requests preserved inside the same checkpoint used for crash recovery, so a restored workflow re-emits any request that was outstanding when it stopped (^[41]). Practical costs of this oversight can be small when designed well: engineering guidance on these approval patterns emphasizes that a well-placed pause point costs far less than an unrecoverable error, though systematic public data on typical intervention latency remains limited.

Traditional infrastructure "uptime," the fraction of time a service responds, does not capture whether a long-running agent is doing its job correctly: a process can be technically running while the task it was given silently fails. One vendor's applied framework defines agent reliability as whether the complete model-plus-harness system repeatedly completes its intended task under real conditions, preserves the constraints that matter, and leaves valid evidence that the outcome exists, recommending measurement of task success, repeated-run consistency, recovery rate, false-completion rate, cost, latency, and escalation rate, segmented by workflow and risk (^[4]) (^[42]).

Table 2 below summarizes the observability platforms enterprises currently use to instrument these metrics in production, each drawn from the provider's own documentation as of the access date given.

Every platform in Table 2 converges on the same primitive: a structured, per-step trace of an agent's execution, from which latency, cost, and error metrics are derived after the fact. None of the eight platforms documents a single agreed "uptime" percentage for agentic systems analogous to a server SLA; each instead exposes granular, per-span or per-trace error and success signals that a team must aggregate into its own reliability definition. This is consistent with a large 2025 field survey of 306 production AI-agent practitioners and 20 in-depth case studies, the first large-scale systematic study of its kind, which found teams manage reliability primarily by constraining agent autonomy rather than through novel technical measurement: 68% execute at most 10 steps before requiring human intervention, 70% rely on prompting off-the-shelf models rather than fine-tuning, and 74% depend primarily on human evaluation rather than automated scoring (^[52]).

Enterprise survey data from 2025 and early 2026 shows adoption of agentic AI running well ahead of confidence in its reliability. Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls, based on a January 2025 poll of 3,412 webinar attendees in which only 19% reported significant organizational investment in agentic AI and 42% reported only conservative investment (^[5]) (^[53]); the same analysis estimates only about 130 of the thousands of self-described agentic AI vendors have genuine agentic capability (^[54]). MIT NANDA and MLQ's July 2025 report describes findings from a survey of 153 leaders, 52 in-depth interviews, and analysis of more than 300 public AI implementations; it reports that 95% of organizations were getting zero measurable return on their GenAI pilots (^[55]).

Other surveys find more optimistic adoption figures alongside similar trust gaps. PwC's May 2025 survey of 300 senior executives found 79% report AI agents already being adopted, with two-thirds (66%) of adopters saying the agents deliver measurable value, but trust dropped sharply for higher-stakes activities, to 20% for financial transactions and 22% for autonomous employee interactions (^[56]) (^[57]). Deloitte's global 2026 survey of 3,235 IT and business leaders across 24 countries found only 21% report a mature governance model for agentic AI despite 74% expecting at least moderate agent usage by 2027 (^[6]) (^[58]). IBM's June 2025 survey of 2,900 executives found respondents expect AI-enabled workflows to grow from 3% to 25% of operations within a single year, while citing data concerns (49%), trust issues (46%), and skills shortages (42%) as the top barriers (^[59]) (^[60]).

Governance maturity and trust appear to be moving in different directions across sources. McKinsey's early-2026 AI Trust Maturity Survey (fielded December 2025 to January 2026 across roughly 500 organizations) found average responsible-AI maturity rose to 2.3 in 2026 from 2.0 in 2025, yet only about 30% of organizations reached a maturity level of three or higher on governance dimensions relevant to agentic control (^[61]) (^[62]). Capgemini's 2025 survey of 1,500 executives found trust in fully autonomous agents fell sharply year over year, from 43% to 27%, even as 90% of respondents viewed human involvement in AI-driven workflows as positive or cost-neutral rather than a drag on efficiency (^[7]) (^[63]). Market-sizing research reflects continued investment despite these trust gaps: Grand View Research projects the global AI agents market will reach $50.31 billion by 2030 at a 45.8% compound annual growth rate from 2025 (^[64]), while MarketsandMarkets projects the enterprise agentic AI segment specifically will grow from $6.76 billion in 2025 to $46.04 billion by 2030, a 47% compound annual growth rate (^[65]). Table 3 collects these figures for reference.

Read together, these sources suggest reliability and governance concerns, not raw model capability, are now the binding constraint on agentic AI deployment. Multiple independent surveys converge on a similar shape: adoption and piloting are widespread, financial return and production deployment are rare, and organizational trust for autonomous, high-stakes action remains low even as trust for supervised or bounded-scope use grows.

(Hypothetical Example) Regulated-document processing pipeline. Consider a life-sciences organization running a multi-day agentic pipeline that extracts, cross-references, and drafts summaries from clinical or regulatory documents. Applying the patterns surfaced in this report, the pipeline would checkpoint after each document processed (not only at the end of the batch), route any extraction the agent scores as low-confidence to a human reviewer rather than auto-publishing it, and log every tool call to an append-only event store so that a failure on document 340 of 500 can be diagnosed and resumed without repeating the first 339. This hypothetical illustrates how the checkpoint-and-recovery and human-in-the-loop patterns described above compose in a regulated setting; it is an illustration, not a reported deployment.

The evidence above illustrates several recovery options rather than one vendor-independent architecture. LangGraph documents thread-scoped graph-state checkpoints, while Microsoft Agent Framework offers in-process, local-file, and Azure Cosmos DB checkpoint-storage implementations with different durability and deployment characteristics (^[2]; Microsoft Agent Framework checkpoint documentation). Second, reliability measurement is maturing faster than reliability itself improves: the Princeton framework's twelve metrics, METR's time-horizon methodology, and the Autonomy Index all give practitioners vocabulary and formulas that did not exist two years ago, even as the underlying finding, that capability gains only weakly translate into reliability gains, has not changed. Third, human oversight is being redesigned as a first-class architectural feature rather than a stopgap: request/response interrupt patterns, persistent checkpointed approval state, and level-based autonomy taxonomies all treat the question of "how much human involvement" as a deliberate, tunable design parameter rather than an admission of AI limitation, consistent with survey findings that a large majority of executives view human involvement as neutral or positive for outcomes rather than a cost.

For regulated industries such as life sciences, where audit trails, validation, and human accountability are pre-existing regulatory expectations rather than optional engineering choices, the durability and human-in-the-loop patterns described in this report map closely onto compliance requirements that already exist for other software systems. Consultancies operating adjacent to this space, without functioning as agent-platform vendors themselves, are positioned to translate general-purpose reliability engineering practices (checkpointing, event logging, staged human review) into the specific validation and audit-trail language regulated organizations already use. IntuitionLabs, a Veeva-focused life-sciences technology consultancy founded in 2023 (^[66]), describes its own role in these terms as a trusted advisor on AI and Veeva CRM implementations rather than as a builder of agent orchestration infrastructure (^[67]), illustrating how an adjacent advisory practice can sit alongside, rather than compete with, the orchestration and observability vendors discussed above.

Looking forward, the widening gap between the 50% and 80% time-horizon curves documented by METR suggests that "reliability at length" will likely remain the harder problem relative to "capability at length" for the near term: it is easier to teach a model to occasionally succeed at a longer task than to make that success dependable. Enterprises evaluating long-running agents for production should expect to budget for human-in-the-loop review capacity, durable execution infrastructure, and per-task-type reliability testing (rather than a single aggregate benchmark score) as ongoing operating costs, not one-time integration costs.

What makes an AI agent "long-running" rather than a normal chatbot session? A long-running agent persists across multiple context windows, tool-call cycles, or even process restarts, and is expected to make forward progress over hours or days rather than complete its task within one exchange, requiring durable state and failure-recovery mechanisms a single-turn chatbot does not need (^[68]).

How is AI agent uptime different from server uptime? Server uptime measures whether a service responds; agent "uptime" in practice is measured through task-level metrics, such as completion rate, recovery rate, and false-completion rate, because an agent can be technically running while still failing the task it was given (^[42]).

What is a human intervention rate? It is a measure of how often a human must step in during autonomous agent execution. At least two competing formal definitions are in current use, a per-session intervention count and a proportion-of-task-steps formula, both detailed in the Human-in-the-Loop Oversight and Intervention Rates section above; no single industry-standard definition yet exists.

How do long-running agents recover from mid-task failures? Recovery behavior depends on the framework and the configured storage. LangGraph checkpointers persist a thread's graph state as checkpoints for fault tolerance, and its in-memory saver does not persist between process restarts; durable recovery therefore requires an appropriate persistent checkpointer (^[2]).

Which benchmarks actually measure reliability rather than one-shot accuracy? tau-bench's pass^k metric, METR's 50% and 80% time-horizon curves, and the "Beyond pass@1" duration-stratified suite, all summarized in Table 1 above, are purpose-built to measure consistency and degradation across repeated trials or increasing task length, rather than best-attempt accuracy alone.

How often do enterprise agentic AI projects get canceled or fail to reach production? Gartner's cancellation projection and the MIT NANDA/MLQ report both describe obstacles to scaling AI initiatives. The MIT NANDA/MLQ report says its findings draw on a survey of 153 leaders, 52 in-depth interviews, and analysis of more than 300 public AI implementations, and reports 95% zero measurable return on GenAI pilots (^[55]).

Long-running AI agents have become measurably more capable, but the evidence assembled in this report shows reliability lagging well behind that capability curve. METR's time-horizon data, the Princeton reliability framework, and the "Beyond pass@1" duration-stratified benchmark all independently find the same shape: agents that succeed most of the time on short tasks succeed far less often as task length, step count, or repetition increases, and eighteen months of model progress produced only small reliability gains despite substantial capability gains. The practical responses described in the cited materials include checkpoint-based recovery, human-in-the-loop approval, and observability that measures task-level success and recovery rather than server-style uptime; their storage and replay semantics vary by framework and configuration (^[2]; Microsoft Agent Framework checkpoint documentation).

Enterprise survey data suggests organizations have absorbed this lesson unevenly. Adoption and piloting of agentic AI are widespread, but governance maturity, measurable return, and production deployment all lag well behind pilot activity, and trust for genuinely autonomous, high-stakes action remains a minority position even among enthusiastic adopters. For any organization evaluating a long-running agent deployment, this report's evidence argues for publishing the same kind of artifact this report itself relies on: reproducible event traces, explicit stopping and intervention rules, and dated, sourced reliability numbers, rather than an assertion of autonomous reliability drawn from a demo. As of September 2026, the tools to build such systems durably exist and are converging across vendors; the tools to measure whether they are actually reliable are newer, less standardized, and still primarily the work of independent researchers and a handful of first-party vendor disclosures.