Skip to main content

3 Tips for Flexible, Adaptive Incident Management

Emily Arnott
Blameless

Incidents should be your best friend. It sounds like a controversial statement. It sounds like a lot of unnecessary work. The truth is, for companies engaged in delivering any online or digital experience, taking this point of view is absolutely E-S-S-E-N-T-I-A-L. Apart from the cost of an outage in production, unplanned work created by incidents will begin to hamper feature velocity if you don't approach addressing them in the right way and there's no faster way to damage your customer relationships than recurring product outages.

Whether we like it or not, responding effectively to unexpected incidents is central to modern IT Operations. Having an integrated, evolving approach to managing incidents can unlock the agility and velocity of a DevOps team and can improve the overall quality of the software they're developing. A rigid, dogmatic approach can leave that same team mired in tech debt and struggling to stay above water.

The key is in viewing incidents as an opportunity to learn something new about your product and your process. If delivering a reliable product that customers will love is your goal, then how you build and operate the product is just as important as what you build. Having the right structure and process can help your engineering team stay aligned at scale. Good incident management practices can be a mechanism for interrogating the effectiveness of that structure. That's true for companies embracing ITIL, DevOps or SRE.

Developing a strong incident response process is key to minimizing downtime and learning from each incident. This takes time, practice and the right tooling. So to help you get started, we've got 3 tips for creating a more flexible, adaptive framework for incident management.

1. Where You Manage Incidents Matters

There is no shortage of software solutions that claim to support incident management. That should be no surprise, managing incidents involves a complex set of tasks that include monitoring, alerting, and paging. However, to really be effective at managing incidents, a command center of sorts is needed to organize the people responsible for achieving resolution. There is no better place to locate that command center, than in the team's preferred chat bot. These offer unparalleled flexibility to recruit and coordinate the right experts. This is where targeted incident management solutions begin to separate themselves from more generic IT solutions like ITIL software.

"Incident Management solutions help DevOps or SRE teams create consistent incident workflows that map to their unique needs. Those workflows can then be easily activated within their chat system and can have wide cascading effects across multiple other systems once they're activated" says Kurt Andersen, SRE architect at Blameless.

2. Never forget "Communication is key"

"The worst case scenario for many SRE leaders is a large Sev0 incident with multiple customers impacted. CEO, VPs, and CS are all reporting customer issues and asking for status updates, while it looks like there are no engineers building or executing a plan to restore service. Then the scenario repeats the next day," says Aaron Bento, Principal SRE for Arkose Labs

When an engineering incident is underway, ensuring stakeholder communication is the most important responsibility of an incident commander, next to resolving the incident itself. They can handle the communications themselves or delegate to a communications lead. This may sound simple but it's anything but. Large organizations are likely to have a diverse set of stakeholders who need to be informed, not the least important of which are their customers.

"Having too many cooks in the kitchen can cripple your incident response. That's why it's so important to communicate effectively, to the right stakeholders throughout the incident" says Vincent Rivellino, Head of Reliability and Developer Platforms at Mission Lane.

"Also, If customers are impacted there can be a serious hit to your company's reputation. We lean into IM even for incidents where we're not breaking technology SLAs. We often need swift incident resolution followed by coordinated execution of customer remediation. For us that often involves non-technical stakeholders who are communicating with our customers. At the end of the day, the most important thing is our customers know we have their back."

Whether managing internal stakeholder communications or communicating with customers, having clearly defined expectations for update cadences and automated reminders to follow up is really helpful. These are unique capabilities of modern incident management tools like Blameless that alternatives don't provide.

3. Treat incidents as opportunities

"The benefit of a more mature incident management process is identifying where the hot spots are in your product and where you as an engineering leader need to invest your team's engineering hours or budget," says Elisa Binette, Director of Engineering and Site Reliability at VMWare.

If your team is interested in driving development velocity, it's not enough to try to eliminate toil from the incident response process. You need to go a step further and begin to leverage incidents proactively to identify points of weakness in your product and engineering process. This means running clear, effective retrospectives, tagging and capturing all the relevant incident data available and surfacing that back to the right stakeholders. Over time, this can help reduce the load on your entire team by making your process more efficient, your product more robust, and reducing the number of repeat incidents that your team has to manage.

"If you look at incidents as an opportunity to learn about what's weak or broken in your product, and commit the right resources to addressing those weaknesses, you can quickly begin to reduce the number of repeat incidents your team encounters. Says Aaron Bento, Principal SRE for Arkose Labs. "Repeat incidents can be a killer for morale because they're a sign that we're not identifying the source of our problem. Taking a more proactive approach to incident management can really make a big difference."

To maximize the value of the incident management process, your team needs opportunities to experiment, learn and iterate. With the right tooling and the right approach, you'll soon be turning disruptive incidents into valuable insights.

Emily Arnott is Community Relations Manager at Blameless
APM

Hot Topics

The Latest

A major architectural shift is underway across enterprise networks, according to a new global study from Cisco. As AI assistants, agents, and data-driven workloads reshape how work gets done, they're creating faster, more dynamic, more latency-sensitive, and more complex network traffic. Combined with the ubiquity of connected devices, 24/7 uptime demands, and intensifying security threats, these shifts are driving infrastructure to adapt and evolve ...

Image
Cisco

The development of banking apps was supposed to provide users with convenience, control and piece of mind. However, for thousands of Halifax customers recently, a major mobile outage caused the exact opposite, leaving customers unable to check balances, or pay bills, sparking widespread frustration. This wasn't an isolated incident ... So why are these failures still happening? ...

Cyber threats are growing more sophisticated every day, and at their forefront are zero-day vulnerabilities. These elusive security gaps are exploited before a fix becomes available, making them among the most dangerous threats in today's digital landscape ... This guide will explore what these vulnerabilities are, how they work, why they pose such a significant threat, and how modern organizations can stay protected ...

The prevention of data center outages continues to be a strategic priority for data center owners and operators. Infrastructure equipment has improved, but the complexity of modern architectures and evolving external threats presents new risks that operators must actively manage, according to the Data Center Outage Analysis 2025 from Uptime Institute ...

As observability engineers, we navigate a sea of telemetry daily. We instrument our applications, configure collectors, and build dashboards, all in pursuit of understanding our complex distributed systems. Yet, amidst this flood of data, a critical question often remains unspoken, or at best, answered by gut feeling: "Is our telemetry actually good?" ... We're inviting you to participate in shaping a foundational element for better observability: the Instrumentation Score ...

We're inching ever closer toward a long-held goal: technology infrastructure that is so automated that it can protect itself. But as IT leaders aggressively employ automation across our enterprises, we need to continuously reassess what AI is ready to manage autonomously and what can not yet be trusted to algorithms ...

Much like a traditional factory turns raw materials into finished products, the AI factory turns vast datasets into actionable business outcomes through advanced models, inferences, and automation. From the earliest data inputs to the final token output, this process must be reliable, repeatable, and scalable. That requires industrializing the way AI is developed, deployed, and managed ...

Almost half (48%) of employees admit they resent their jobs but stay anyway, according to research from Ivanti ... This has obvious consequences across the business, but we're overlooking the massive impact of resenteeism and presenteeism on IT. For IT professionals tasked with managing the backbone of modern business operations, these numbers spell big trouble ...

For many B2B and B2C enterprise brands, technology isn't a core strength. Relying on overly complex architectures (like those that follow a pure MACH doctrine) has been flagged by industry leaders as a source of operational slowdown, creating bottlenecks that limit agility in volatile market conditions ...

FinOps champions crucial cross-departmental collaboration, uniting business, finance, technology and engineering leaders to demystify cloud expenses. Yet, too often, critical cost issues are softened into mere "recommendations" or "insights" — easy to ignore. But what if we adopted security's battle-tested strategy and reframed these as the urgent risks they truly are, demanding immediate action? ...

3 Tips for Flexible, Adaptive Incident Management

Emily Arnott
Blameless

Incidents should be your best friend. It sounds like a controversial statement. It sounds like a lot of unnecessary work. The truth is, for companies engaged in delivering any online or digital experience, taking this point of view is absolutely E-S-S-E-N-T-I-A-L. Apart from the cost of an outage in production, unplanned work created by incidents will begin to hamper feature velocity if you don't approach addressing them in the right way and there's no faster way to damage your customer relationships than recurring product outages.

Whether we like it or not, responding effectively to unexpected incidents is central to modern IT Operations. Having an integrated, evolving approach to managing incidents can unlock the agility and velocity of a DevOps team and can improve the overall quality of the software they're developing. A rigid, dogmatic approach can leave that same team mired in tech debt and struggling to stay above water.

The key is in viewing incidents as an opportunity to learn something new about your product and your process. If delivering a reliable product that customers will love is your goal, then how you build and operate the product is just as important as what you build. Having the right structure and process can help your engineering team stay aligned at scale. Good incident management practices can be a mechanism for interrogating the effectiveness of that structure. That's true for companies embracing ITIL, DevOps or SRE.

Developing a strong incident response process is key to minimizing downtime and learning from each incident. This takes time, practice and the right tooling. So to help you get started, we've got 3 tips for creating a more flexible, adaptive framework for incident management.

1. Where You Manage Incidents Matters

There is no shortage of software solutions that claim to support incident management. That should be no surprise, managing incidents involves a complex set of tasks that include monitoring, alerting, and paging. However, to really be effective at managing incidents, a command center of sorts is needed to organize the people responsible for achieving resolution. There is no better place to locate that command center, than in the team's preferred chat bot. These offer unparalleled flexibility to recruit and coordinate the right experts. This is where targeted incident management solutions begin to separate themselves from more generic IT solutions like ITIL software.

"Incident Management solutions help DevOps or SRE teams create consistent incident workflows that map to their unique needs. Those workflows can then be easily activated within their chat system and can have wide cascading effects across multiple other systems once they're activated" says Kurt Andersen, SRE architect at Blameless.

2. Never forget "Communication is key"

"The worst case scenario for many SRE leaders is a large Sev0 incident with multiple customers impacted. CEO, VPs, and CS are all reporting customer issues and asking for status updates, while it looks like there are no engineers building or executing a plan to restore service. Then the scenario repeats the next day," says Aaron Bento, Principal SRE for Arkose Labs

When an engineering incident is underway, ensuring stakeholder communication is the most important responsibility of an incident commander, next to resolving the incident itself. They can handle the communications themselves or delegate to a communications lead. This may sound simple but it's anything but. Large organizations are likely to have a diverse set of stakeholders who need to be informed, not the least important of which are their customers.

"Having too many cooks in the kitchen can cripple your incident response. That's why it's so important to communicate effectively, to the right stakeholders throughout the incident" says Vincent Rivellino, Head of Reliability and Developer Platforms at Mission Lane.

"Also, If customers are impacted there can be a serious hit to your company's reputation. We lean into IM even for incidents where we're not breaking technology SLAs. We often need swift incident resolution followed by coordinated execution of customer remediation. For us that often involves non-technical stakeholders who are communicating with our customers. At the end of the day, the most important thing is our customers know we have their back."

Whether managing internal stakeholder communications or communicating with customers, having clearly defined expectations for update cadences and automated reminders to follow up is really helpful. These are unique capabilities of modern incident management tools like Blameless that alternatives don't provide.

3. Treat incidents as opportunities

"The benefit of a more mature incident management process is identifying where the hot spots are in your product and where you as an engineering leader need to invest your team's engineering hours or budget," says Elisa Binette, Director of Engineering and Site Reliability at VMWare.

If your team is interested in driving development velocity, it's not enough to try to eliminate toil from the incident response process. You need to go a step further and begin to leverage incidents proactively to identify points of weakness in your product and engineering process. This means running clear, effective retrospectives, tagging and capturing all the relevant incident data available and surfacing that back to the right stakeholders. Over time, this can help reduce the load on your entire team by making your process more efficient, your product more robust, and reducing the number of repeat incidents that your team has to manage.

"If you look at incidents as an opportunity to learn about what's weak or broken in your product, and commit the right resources to addressing those weaknesses, you can quickly begin to reduce the number of repeat incidents your team encounters. Says Aaron Bento, Principal SRE for Arkose Labs. "Repeat incidents can be a killer for morale because they're a sign that we're not identifying the source of our problem. Taking a more proactive approach to incident management can really make a big difference."

To maximize the value of the incident management process, your team needs opportunities to experiment, learn and iterate. With the right tooling and the right approach, you'll soon be turning disruptive incidents into valuable insights.

Emily Arnott is Community Relations Manager at Blameless
APM

Hot Topics

The Latest

A major architectural shift is underway across enterprise networks, according to a new global study from Cisco. As AI assistants, agents, and data-driven workloads reshape how work gets done, they're creating faster, more dynamic, more latency-sensitive, and more complex network traffic. Combined with the ubiquity of connected devices, 24/7 uptime demands, and intensifying security threats, these shifts are driving infrastructure to adapt and evolve ...

Image
Cisco

The development of banking apps was supposed to provide users with convenience, control and piece of mind. However, for thousands of Halifax customers recently, a major mobile outage caused the exact opposite, leaving customers unable to check balances, or pay bills, sparking widespread frustration. This wasn't an isolated incident ... So why are these failures still happening? ...

Cyber threats are growing more sophisticated every day, and at their forefront are zero-day vulnerabilities. These elusive security gaps are exploited before a fix becomes available, making them among the most dangerous threats in today's digital landscape ... This guide will explore what these vulnerabilities are, how they work, why they pose such a significant threat, and how modern organizations can stay protected ...

The prevention of data center outages continues to be a strategic priority for data center owners and operators. Infrastructure equipment has improved, but the complexity of modern architectures and evolving external threats presents new risks that operators must actively manage, according to the Data Center Outage Analysis 2025 from Uptime Institute ...

As observability engineers, we navigate a sea of telemetry daily. We instrument our applications, configure collectors, and build dashboards, all in pursuit of understanding our complex distributed systems. Yet, amidst this flood of data, a critical question often remains unspoken, or at best, answered by gut feeling: "Is our telemetry actually good?" ... We're inviting you to participate in shaping a foundational element for better observability: the Instrumentation Score ...

We're inching ever closer toward a long-held goal: technology infrastructure that is so automated that it can protect itself. But as IT leaders aggressively employ automation across our enterprises, we need to continuously reassess what AI is ready to manage autonomously and what can not yet be trusted to algorithms ...

Much like a traditional factory turns raw materials into finished products, the AI factory turns vast datasets into actionable business outcomes through advanced models, inferences, and automation. From the earliest data inputs to the final token output, this process must be reliable, repeatable, and scalable. That requires industrializing the way AI is developed, deployed, and managed ...

Almost half (48%) of employees admit they resent their jobs but stay anyway, according to research from Ivanti ... This has obvious consequences across the business, but we're overlooking the massive impact of resenteeism and presenteeism on IT. For IT professionals tasked with managing the backbone of modern business operations, these numbers spell big trouble ...

For many B2B and B2C enterprise brands, technology isn't a core strength. Relying on overly complex architectures (like those that follow a pure MACH doctrine) has been flagged by industry leaders as a source of operational slowdown, creating bottlenecks that limit agility in volatile market conditions ...

FinOps champions crucial cross-departmental collaboration, uniting business, finance, technology and engineering leaders to demystify cloud expenses. Yet, too often, critical cost issues are softened into mere "recommendations" or "insights" — easy to ignore. But what if we adopted security's battle-tested strategy and reframed these as the urgent risks they truly are, demanding immediate action? ...