Skip to main content

The Incident Load Is Rising, but Recovery Times Aren't Moving

Sanjeev Sharma
StackGen

Two things happened to production incidents between 2023 and now, and they did not happen at the same speed.

The first is that a class of dependency that barely existed three years ago now accounts for one incident in ten. Incidents disclosed by AI model and AI application providers rose from 1.7% of all disclosed unplanned incidents in 2023 to 10.7% in 2026 year to date, roughly a sixfold rise; that counts only incidents at AI companies themselves, so the true share is higher.

The second is that the time to close an incident has not come down. Across the companies disclosing in every year since 2023, the median has gone from 3.2 hours to 3.6, touching 3.9 in 2025. Split those companies by the kind of service they run and the picture holds: application-layer businesses sit near 1.6 to 1.8 hours, infrastructure businesses between 3.1 and 3.8, and neither group has improved since 2023.

Read those together. More incidents are arriving, a growing share of them from a dependency nobody carried three years ago, and the time to close each one has not improved. That is not teams getting slower. It is teams absorbing a larger, less familiar workload at the rate they absorbed a smaller, more familiar one. The distance between those two lines is the capacity squeeze.

Both halves come from the State of Reliability 2026 report: 177,960 public status-page entries (roughly 109,100 unplanned incidents) from more than 390 companies, 2018 to June 2026.

Resolution time is the disclosed window between the first status-page posting and the incident being marked resolved, a proxy for internal MTTR. Nothing is self-reported, which is also the limitation: incidents never posted publicly are invisible to it. Roughly 65% of the variation is incident-level, which engineer picked up the page and whether an existing runbook fitted.

Where the Extra Load Is Coming From

A second lens, built on failure mechanism rather than company type, moves the same way: AI-related failure modes (output quality degradation plus agentic and tool-use failures) grew from 0.5% of classified incidents in 2023 to 5.1% in 2026 YTD. Nor is it an artifact of tracking more AI companies: fixed to the cohort already disclosing in 2023, the share still rises to 5.7%.

That failure shape is now familiar to anyone running models in production: the service is up, the model is answering, and the answers are wrong. Finding out why can mean walking a thousand-span trace by hand.

It is also concentrated: a small number of providers, Anthropic and OpenAI among them, account for roughly 75% of incidents, and they sit underneath a great many products.

That is the squeeze's other half, and it has a name: third-party cascade. More than one in four classified incidents start upstream, at a provider the affected team does not run. Those cascades take a median of 247 minutes against 96 for internally-caused configuration failures, roughly three times longer.

For a category that large, the open question is who owns the remedy: switchability when a provider degrades, or upstream providers getting faster.

Why the Relief Hasn't Arrived

If the load is rising, the counterweight should be faster response. In one sector it is. AI Model Providers are the fastest-resolving group in the dataset and still improving, from a median of 75 minutes in 2023 to 49 minutes in 2026 YTD: a small, well-resourced group, not a proxy for the wider market, but proof the number can move.

Across everyone else it has not. The most-applied primary remediation in the report's coded post-mortem sample is not a fix at all: waiting for an upstream provider, at 13.6%, ahead of restart at 10.0% and rollback at 8.9%.

Read that carefully. Restarting and rolling back are not evidence that AI is missing from the response; they are often the right call, and an AI-assisted team would still do both. What should change is how fast a team gets there, and eventually how many incidents happen at all, which this dataset does not measure. Falling resolution time would be the first sign, and it has not appeared. The report's own reading: the gains promised by AI-based SRE tooling "have not yet been felt in a material way."

Recovery Times Vary More Between Companies than Between Industries

That spread is not abstract. At one end of the range are infrastructure teams that resolve most outages inside twenty minutes and are not drowning in critical alerts. At the other end, in the same sector, incidents can run for days before anyone can call them definitively closed. Neither is an unusual month for the teams living it; they are simply the ends of the range rather than the middle.

Which raises the question of who absorbs it better, and why. The report decomposes what explains variation in median resolution time: roughly 35% ties to identifiable factors. The largest is the company itself, at about 20 percentage points of variance, against 8 for industry membership and 7 for failure mode and root cause combined.

Two SRE teams in the same industry, handling similar workloads, can plausibly differ by 3x in recovery speed.

Industry still tells you something: the degree of difficulty that comes with the territory. Payments and cloud infrastructure carry coordination demands and blast radius that developer tooling does not. But it is a weaker signal than how a team operates its SRE function: its observability, the automation and runbooks around it, on-call practice, and the stack underneath. We also saw that firms that recover quickly from one type of incident tend to recover quickly from the others.

If sector explained most of the variation, benchmarking against your industry would tell you roughly where you were going to land. It does not. The companies whose numbers most resemble yours are not the ones that look like you on paper. They are the ones that operate the way you operate. Peers running similar workloads in the same sector differ threefold, and that difference is not inherited.

That is the hopeful reading, and also the demand. The load compounds: more incidents, a growing share from dependencies nobody owns, and a failure class where the service stays up and the answer is wrong. Whether SRE teams stay ahead depends on whether the AI technology driving that growth can be turned onto the response itself, faster than it adds complexity. Nothing in this data says that race is lost. Nothing says it is won. On the evidence so far, it has barely started.

Sanjeev Sharma is Field CTO at StackGen

The Latest

AI observability serves several roles across an organization. AI engineers use it to build, evaluate, and improve AI systems. SREs use it to operate those systems reliably and efficiently in production, including managing their cost. Leaders need it to connect AI behavior and operational performance to business outcomes. The common need is a clear understanding of whether AI systems are delivering the intended result ...

Part 4 outlines the challenges enterprise face when adopting AI observability. The main challenge is that AI complexity compounds at every stage of maturity. LLM-based applications introduce new behavior, dependencies, and quality considerations, while Agentic AI adds further failure modes as systems make decisions and take actions ...

Part 3 reports on the current state of AI observability. Most experts agree: enterprises are moving faster with AI than they are with the operational discipline needed to run it safely ...

When discussing AI Observability, one of the first questions that arises: Is this just an extension of traditional observability or is this a whole new type of tool? Part 2 compares traditional observability with AI observability ...

In this extensive 14-part series that will post over the next three weeks, experts across observability and AI will provide insight and opinions on the need for AI Observability, the challenges, the benefits, the current state, and predictions for the future ...

Amid rising complexity, organizations are moving quickly toward agent-driven operations: AgenticOps, according to The Impact of Agentic AI on Network Operations, a new report from Cisco ...

AI, automation, and enterprise software give organizations an extraordinary amount of leverage. Work that once required hours can happen in minutes. Information buried across systems can be analyzed almost instantly. Complex workflows can be coordinated across teams with far less manual intervention. That leverage creates a temptation to start with the technology ...

Everyone wants to talk about models. Compute clusters. Benchmark scores that climb a fraction of a point and make headlines anyway. It's an easy conversation to have because it's measurable. But it's also increasingly beside the point ...

For more than a decade, the default infrastructure play has been simple. Move more workloads to the cloud, retire on-premises gear and treat the data center as a cost to shrink. That logic held while public cloud offered near-limitless capacity at predictable prices, and AI demand is now testing both halves of that assumption ...

Most enterprise IT leaders can tell you exactly how many applications are running across their organization. They can surface a dashboard showing telecom spend by region, pull a report on SaaS license utilization, or call up a real-time view of mobile device inventory. The data is there, and IT leaders have largely delivered on the promise of visibility. Yet, in most organizations, that visibility has not translated into business value ...

The Incident Load Is Rising, but Recovery Times Aren't Moving

Sanjeev Sharma
StackGen

Two things happened to production incidents between 2023 and now, and they did not happen at the same speed.

The first is that a class of dependency that barely existed three years ago now accounts for one incident in ten. Incidents disclosed by AI model and AI application providers rose from 1.7% of all disclosed unplanned incidents in 2023 to 10.7% in 2026 year to date, roughly a sixfold rise; that counts only incidents at AI companies themselves, so the true share is higher.

The second is that the time to close an incident has not come down. Across the companies disclosing in every year since 2023, the median has gone from 3.2 hours to 3.6, touching 3.9 in 2025. Split those companies by the kind of service they run and the picture holds: application-layer businesses sit near 1.6 to 1.8 hours, infrastructure businesses between 3.1 and 3.8, and neither group has improved since 2023.

Read those together. More incidents are arriving, a growing share of them from a dependency nobody carried three years ago, and the time to close each one has not improved. That is not teams getting slower. It is teams absorbing a larger, less familiar workload at the rate they absorbed a smaller, more familiar one. The distance between those two lines is the capacity squeeze.

Both halves come from the State of Reliability 2026 report: 177,960 public status-page entries (roughly 109,100 unplanned incidents) from more than 390 companies, 2018 to June 2026.

Resolution time is the disclosed window between the first status-page posting and the incident being marked resolved, a proxy for internal MTTR. Nothing is self-reported, which is also the limitation: incidents never posted publicly are invisible to it. Roughly 65% of the variation is incident-level, which engineer picked up the page and whether an existing runbook fitted.

Where the Extra Load Is Coming From

A second lens, built on failure mechanism rather than company type, moves the same way: AI-related failure modes (output quality degradation plus agentic and tool-use failures) grew from 0.5% of classified incidents in 2023 to 5.1% in 2026 YTD. Nor is it an artifact of tracking more AI companies: fixed to the cohort already disclosing in 2023, the share still rises to 5.7%.

That failure shape is now familiar to anyone running models in production: the service is up, the model is answering, and the answers are wrong. Finding out why can mean walking a thousand-span trace by hand.

It is also concentrated: a small number of providers, Anthropic and OpenAI among them, account for roughly 75% of incidents, and they sit underneath a great many products.

That is the squeeze's other half, and it has a name: third-party cascade. More than one in four classified incidents start upstream, at a provider the affected team does not run. Those cascades take a median of 247 minutes against 96 for internally-caused configuration failures, roughly three times longer.

For a category that large, the open question is who owns the remedy: switchability when a provider degrades, or upstream providers getting faster.

Why the Relief Hasn't Arrived

If the load is rising, the counterweight should be faster response. In one sector it is. AI Model Providers are the fastest-resolving group in the dataset and still improving, from a median of 75 minutes in 2023 to 49 minutes in 2026 YTD: a small, well-resourced group, not a proxy for the wider market, but proof the number can move.

Across everyone else it has not. The most-applied primary remediation in the report's coded post-mortem sample is not a fix at all: waiting for an upstream provider, at 13.6%, ahead of restart at 10.0% and rollback at 8.9%.

Read that carefully. Restarting and rolling back are not evidence that AI is missing from the response; they are often the right call, and an AI-assisted team would still do both. What should change is how fast a team gets there, and eventually how many incidents happen at all, which this dataset does not measure. Falling resolution time would be the first sign, and it has not appeared. The report's own reading: the gains promised by AI-based SRE tooling "have not yet been felt in a material way."

Recovery Times Vary More Between Companies than Between Industries

That spread is not abstract. At one end of the range are infrastructure teams that resolve most outages inside twenty minutes and are not drowning in critical alerts. At the other end, in the same sector, incidents can run for days before anyone can call them definitively closed. Neither is an unusual month for the teams living it; they are simply the ends of the range rather than the middle.

Which raises the question of who absorbs it better, and why. The report decomposes what explains variation in median resolution time: roughly 35% ties to identifiable factors. The largest is the company itself, at about 20 percentage points of variance, against 8 for industry membership and 7 for failure mode and root cause combined.

Two SRE teams in the same industry, handling similar workloads, can plausibly differ by 3x in recovery speed.

Industry still tells you something: the degree of difficulty that comes with the territory. Payments and cloud infrastructure carry coordination demands and blast radius that developer tooling does not. But it is a weaker signal than how a team operates its SRE function: its observability, the automation and runbooks around it, on-call practice, and the stack underneath. We also saw that firms that recover quickly from one type of incident tend to recover quickly from the others.

If sector explained most of the variation, benchmarking against your industry would tell you roughly where you were going to land. It does not. The companies whose numbers most resemble yours are not the ones that look like you on paper. They are the ones that operate the way you operate. Peers running similar workloads in the same sector differ threefold, and that difference is not inherited.

That is the hopeful reading, and also the demand. The load compounds: more incidents, a growing share from dependencies nobody owns, and a failure class where the service stays up and the answer is wrong. Whether SRE teams stay ahead depends on whether the AI technology driving that growth can be turned onto the response itself, faster than it adds complexity. Nothing in this data says that race is lost. Nothing says it is won. On the evidence so far, it has barely started.

Sanjeev Sharma is Field CTO at StackGen

The Latest

AI observability serves several roles across an organization. AI engineers use it to build, evaluate, and improve AI systems. SREs use it to operate those systems reliably and efficiently in production, including managing their cost. Leaders need it to connect AI behavior and operational performance to business outcomes. The common need is a clear understanding of whether AI systems are delivering the intended result ...

Part 4 outlines the challenges enterprise face when adopting AI observability. The main challenge is that AI complexity compounds at every stage of maturity. LLM-based applications introduce new behavior, dependencies, and quality considerations, while Agentic AI adds further failure modes as systems make decisions and take actions ...

Part 3 reports on the current state of AI observability. Most experts agree: enterprises are moving faster with AI than they are with the operational discipline needed to run it safely ...

When discussing AI Observability, one of the first questions that arises: Is this just an extension of traditional observability or is this a whole new type of tool? Part 2 compares traditional observability with AI observability ...

In this extensive 14-part series that will post over the next three weeks, experts across observability and AI will provide insight and opinions on the need for AI Observability, the challenges, the benefits, the current state, and predictions for the future ...

Amid rising complexity, organizations are moving quickly toward agent-driven operations: AgenticOps, according to The Impact of Agentic AI on Network Operations, a new report from Cisco ...

AI, automation, and enterprise software give organizations an extraordinary amount of leverage. Work that once required hours can happen in minutes. Information buried across systems can be analyzed almost instantly. Complex workflows can be coordinated across teams with far less manual intervention. That leverage creates a temptation to start with the technology ...

Everyone wants to talk about models. Compute clusters. Benchmark scores that climb a fraction of a point and make headlines anyway. It's an easy conversation to have because it's measurable. But it's also increasingly beside the point ...

For more than a decade, the default infrastructure play has been simple. Move more workloads to the cloud, retire on-premises gear and treat the data center as a cost to shrink. That logic held while public cloud offered near-limitless capacity at predictable prices, and AI demand is now testing both halves of that assumption ...

Most enterprise IT leaders can tell you exactly how many applications are running across their organization. They can surface a dashboard showing telecom spend by region, pull a report on SaaS license utilization, or call up a real-time view of mobile device inventory. The data is there, and IT leaders have largely delivered on the promise of visibility. Yet, in most organizations, that visibility has not translated into business value ...