Most organizations suffer from some form of alert noise, Adam Blau, Senior Director of Product Marketing at BigPanda, explains: "Alert noise is only going to increase as organizations support cloud-native applications spanning multiple public and private clouds, including ephemeral deployments and more. It's not going to get easier for organizations to understand the signal from all those alerts being sent." Blau said. So what can be done about it?
In an on-demand session from Pandapalooza, Blau and BigPanda CTO Jason Walker discussed how we got to this worrisome level of unhelpful alert noise and what organizations can do about it.
Blau describes a few reasons why most organizations are dealing with unmanageable levels of low-quality alert noise. Part of the problem is that "it's no one's job to monitor alert quality," he said. There is no feedback loop. "It gets painful because the NOC is receiving all these events and alerts, and there's no way to go back [to the monitoring team] to tell them what is actionable and what is just noise."
The impact of this problem is that the proportion of noise to signal increases each year. And for some organizations, only 1% of all alerts are actionable. The goal is to prevent users from having to filter and sort through low- and medium-quality alerts so they can focus on meaningful alerts.
Walker stepped in to offer practical tips and best practices in the domain of monitoring quality and how to start it:
Best practice 1: Focus on the domain within your control
Concentrate on obvious, low-quality alerts where you have high levels of technical and business understanding in order to connect the dots across your organization. These obvious, low-quality alerts should be on your hit list so you can eliminate them from the operations pipeline as early as possible.
Best practice 2: Be guided by business context
Enrich alerts with real-world business context that has been defined, reviewed and agreed-upon by multiple teams.
Best practice 3: Drive effectiveness with cross-functionality review processes
Directly involving stakeholders helps establish a culture of ownership and commitment to improving alert and incident quality. Communication among teams about what percentage of alerts are noise and what are meaningful promotes an overall healthier pipeline.
Best practice 4: Prioritize monitoring quality hygiene
"You have to guard this jealously," said Jason. "You have to have something in place that tells you what your monitoring quality is and maintains the integrity of your pipeline."
To learn more about these best practices and how BigPanda's AIOps platform can help streamline the flood of alert noise in your organization, check out the full session here.
The Latest
Developers need a tool that can be portable and vendor agnostic, given the advent of microservices. It may be clear an issue is occurring; what may not be clear is if it's part of a distributed system or the app itself. Enter OpenTelemetry, commonly referred to as OTel, an open-source framework that provides a standardized way of collecting and exporting telemetry data (logs, metrics, and traces) from cloud-native software ...
As SLOs grow in popularity their usage is becoming more mature. For example, 82% of respondents intend to increase their use of SLOs, and 96% have mapped SLOs directly to their business operations or already have a plan to, according to The State of Service Level Objectives 2023 from Nobl9 ...
Observability has matured beyond its early adopter position and is now foundational for modern enterprises to achieve full visibility into today's complex technology environments, according to The State of Observability 2023, a report released by Splunk in collaboration with Enterprise Strategy Group ...
Before network engineers even begin the automation process, they tend to start with preconceived notions that oftentimes, if acted upon, can hinder the process. To prevent that from happening, it's important to identify and dispel a few common misconceptions currently out there and how networking teams can overcome them. So, let's address the three most common network automation myths ...
Many IT organizations apply AI/ML and AIOps technology across domains, correlating insights from the various layers of IT infrastructure and operations. However, Enterprise Management Associates (EMA) has observed significant interest in applying these AI technologies narrowly to network management, according to a new research report, titled AI-Driven Networks: Leveling Up Network Management with AI/ML and AIOps ...
When it comes to system outages, AIOps solutions with the right foundation can help reduce the blame game so the right teams can spend valuable time restoring the impacted services rather than improving their MTTI score (mean time to innocence). In fact, much of today's innovation around ChatGPT-style algorithms can be used to significantly improve the triage process and user experience ...
Gartner identified the top 10 data and analytics (D&A) trends for 2023 that can guide D&A leaders to create new sources of value by anticipating change and transforming extreme uncertainty into new business opportunities ...
The only way for companies to stay competitive is to modernize applications, yet there's no denying that bringing apps into the modern era can be challenging ... Let's look at a few ways to modernize applications and consider what new obstacles and opportunities 2023 presents ...
As online penetration grows, retailers' profits are shrinking — with the cost of serving customers anytime, anywhere, at any speed not bringing in enough topline growth to best monetize even existing investments in technology, systems, infrastructure, and people, let alone new investments, according to Digital-First Retail: Turning Profit Destruction into Customer and Shareholder Value, a new report from AlixPartners and World Retail Congress ...
