Skip to main content

Finding the Needle in the Haystack: How Machine Learning Will Revolutionize Root Cause Analysis

Ajay Singh
Zebrium

When a website or app fails or falters, the standard operating procedure is to assemble a sizable team to quickly "divide and conquer" to find a solution. The details of the problem can usually be found somewhere among millions of log events and metrics, leading to slow and painstaking searches that can take hours and often involve handoffs between experts in different areas of the software. The immediate goal in these situations is not to be comprehensive, but rather to troubleshoot until you find a solution that remedies the symptom, even if the underlying root cause is not addressed.

The entire troubleshooting process takes time — generally lots and lots of it — and experience. Development teams tend to be chronically short-staffed and overworked, so adding the burden to hunt for the cause of an app problem means a substantial opportunity cost among other things. To help with the task, most companies leverage multiple best-of-breed observability tools including application performance management (APM), tracing, monitoring and log management. These are used to detect and find a solution to the problem being experienced. Although each tool provides useful data, in total, it can be hard for a person to interpret what is important and what is less so.

Instead of a disruptive and often frenzied, big team approach, this kind of challenge is a perfect application for machine learning (ML) to sift through volumes of data and find meaningful patterns or anomalies that can explain the root cause.

AIOps — using AI for IT operations — has emerged as a possible solution for correlating data from multiple tools to reduce noise and translate events into something more meaningful for a user. On the plus side, AIOps solutions are designed to handle events from a wide range of tools, making them versatile. On the negative side, most AIOps solutions require very long training periods (typically many months) against labeled data sets. These solutions also fall short, because they are designed to correlate events against known problems rather than find the root cause of new or unknown failure modes. This is a particular weakness in fast changing cloud-native environments, where new failure modes crop up on a regular basis.

In order to find the root cause of new failure modes, a different type of AI approach is needed. Since logs often contain the source of truth when a software failure occurs, one approach is to use ML on logs. The concept is to identify just the anomalous patterns in the logs that explain the details of the problem. This can be challenging since logs are mostly unstructured and "noisy." On top of that, log volumes are typically huge with data coming from many different log streams, each with a large number of log lines. Historical approaches have focused on basic anomaly detection which not only produce verbose results that require human interpretation, but also don't explain correlations across micro-services, often entirely missing key details of the problem.

It turns out, the most effective way to perform ML on logs is to use a pipeline with multiple different ML strategies depending on stage of the process. Specialized ML starts by self-learning (i.e. unsupervised) how to structure and categorize the logs — this produces a solid foundation for the remaining ML stages. Next, the ML learns the patterns of each type of log event. Once this learning has occurred, the ML system can identify anomalous log events within each log stream (events that break pattern).

Finally, to pull out the signal from the noise, the system needs to find correlations between anomalies and errors across multiple log streams. This process provides an effective way of uncovering just the sequence of log lines that describe the problem and its root cause. In doing so, it allows for accurate detection of new types of failure modes as well as the information needed to identify root cause.

Such a methodology is similar to the approach taken by skilled engineers — understanding the logs, identifying rare and high-severity events and then finding correlations between clusters of these events across multiple log streams. But it requires considerable time for humans to do this. In practice, the task would be spread out across multiple people in a divide and conquer mode in attempt to accelerate the process and lessen the load for each person. While this inherently makes sense, it creates an additional challenge of requiring team members to communicate with each other in such a way that all are aware of all anomalies and errors, and the observations and learnings are all known and shared across the group. In essence, the team needs to function as a single entity.

A multi-staged ML approach works as a single automated entity, and it should not require any manual training, whether in reviewing correlations for tuning algorithms or massaging data sets. The system should free up DevOps teams, so that they only have to respond to actual findings of root cause. A system should only need a few hours of log data to achieve proper levels of accuracy.

While AIOps is useful for reducing the overall event "noise" from the many observability tools in use in an organization, applying multi-stage unsupervised ML to logs is a great way of both detecting new types of failure modes as well as their root cause. Rather than just triaging a problem and coming up with a quick fix or workaround, the system can determine the true root cause and likely avoid other such problems in the future.

Ajay Singh is Founder and CEO of Zebrium

Hot Topics

The Latest

AI is the catalyst for significant investment in data teams as enterprises require higher-quality data to power their AI applications, according to the State of Analytics Engineering Report from dbt Labs ...

Misaligned architecture can lead to business consequences, with 93% of respondents reporting negative outcomes such as service disruptions, high operational costs and security challenges ...

A Gartner analyst recently suggested that GenAI tools could create 25% time savings for network operational teams. Where might these time savings come from? How are GenAI tools helping NetOps teams today, and what other tasks might they take on in the future as models continue improving? In general, these savings come from automating or streamlining manual NetOps tasks ...

IT and line-of-business teams are increasingly aligned in their efforts to close the data gap and drive greater collaboration to alleviate IT bottlenecks and offload growing demands on IT teams, according to The 2025 Automation Benchmark Report: Insights from IT Leaders on Enterprise Automation & the Future of AI-Driven Businesses from Jitterbit ...

A large majority (86%) of data management and AI decision makers cite protecting data privacy as a top concern, with 76% of respondents citing ROI on data privacy and AI initiatives across their organization, according to a new Harris Poll from Collibra ...

According to Gartner, Inc. the following six trends will shape the future of cloud over the next four years, ultimately resulting in new ways of working that are digital in nature and transformative in impact ...

2020 was the equivalent of a wedding with a top-shelf open bar. As businesses scrambled to adjust to remote work, digital transformation accelerated at breakneck speed. New software categories emerged overnight. Tech stacks ballooned with all sorts of SaaS apps solving ALL the problems — often with little oversight or long-term integration planning, and yes frequently a lot of duplicated functionality ... But now the music's faded. The lights are on. Everyone from the CIO to the CFO is checking the bill. Welcome to the Great SaaS Hangover ...

Regardless of OpenShift being a scalable and flexible software, it can be a pain to monitor since complete visibility into the underlying operations is not guaranteed ... To effectively monitor an OpenShift environment, IT administrators should focus on these five key elements and their associated metrics ...

An overwhelming majority of IT leaders (95%) believe the upcoming wave of AI-powered digital transformation is set to be the most impactful and intensive seen thus far, according to The Science of Productivity: AI, Adoption, And Employee Experience, a new report from Nexthink ...

Overall outage frequency and the general level of reported severity continue to decline, according to the Outage Analysis 2025 from Uptime Institute. However, cyber security incidents are on the rise and often have severe, lasting impacts ...

Finding the Needle in the Haystack: How Machine Learning Will Revolutionize Root Cause Analysis

Ajay Singh
Zebrium

When a website or app fails or falters, the standard operating procedure is to assemble a sizable team to quickly "divide and conquer" to find a solution. The details of the problem can usually be found somewhere among millions of log events and metrics, leading to slow and painstaking searches that can take hours and often involve handoffs between experts in different areas of the software. The immediate goal in these situations is not to be comprehensive, but rather to troubleshoot until you find a solution that remedies the symptom, even if the underlying root cause is not addressed.

The entire troubleshooting process takes time — generally lots and lots of it — and experience. Development teams tend to be chronically short-staffed and overworked, so adding the burden to hunt for the cause of an app problem means a substantial opportunity cost among other things. To help with the task, most companies leverage multiple best-of-breed observability tools including application performance management (APM), tracing, monitoring and log management. These are used to detect and find a solution to the problem being experienced. Although each tool provides useful data, in total, it can be hard for a person to interpret what is important and what is less so.

Instead of a disruptive and often frenzied, big team approach, this kind of challenge is a perfect application for machine learning (ML) to sift through volumes of data and find meaningful patterns or anomalies that can explain the root cause.

AIOps — using AI for IT operations — has emerged as a possible solution for correlating data from multiple tools to reduce noise and translate events into something more meaningful for a user. On the plus side, AIOps solutions are designed to handle events from a wide range of tools, making them versatile. On the negative side, most AIOps solutions require very long training periods (typically many months) against labeled data sets. These solutions also fall short, because they are designed to correlate events against known problems rather than find the root cause of new or unknown failure modes. This is a particular weakness in fast changing cloud-native environments, where new failure modes crop up on a regular basis.

In order to find the root cause of new failure modes, a different type of AI approach is needed. Since logs often contain the source of truth when a software failure occurs, one approach is to use ML on logs. The concept is to identify just the anomalous patterns in the logs that explain the details of the problem. This can be challenging since logs are mostly unstructured and "noisy." On top of that, log volumes are typically huge with data coming from many different log streams, each with a large number of log lines. Historical approaches have focused on basic anomaly detection which not only produce verbose results that require human interpretation, but also don't explain correlations across micro-services, often entirely missing key details of the problem.

It turns out, the most effective way to perform ML on logs is to use a pipeline with multiple different ML strategies depending on stage of the process. Specialized ML starts by self-learning (i.e. unsupervised) how to structure and categorize the logs — this produces a solid foundation for the remaining ML stages. Next, the ML learns the patterns of each type of log event. Once this learning has occurred, the ML system can identify anomalous log events within each log stream (events that break pattern).

Finally, to pull out the signal from the noise, the system needs to find correlations between anomalies and errors across multiple log streams. This process provides an effective way of uncovering just the sequence of log lines that describe the problem and its root cause. In doing so, it allows for accurate detection of new types of failure modes as well as the information needed to identify root cause.

Such a methodology is similar to the approach taken by skilled engineers — understanding the logs, identifying rare and high-severity events and then finding correlations between clusters of these events across multiple log streams. But it requires considerable time for humans to do this. In practice, the task would be spread out across multiple people in a divide and conquer mode in attempt to accelerate the process and lessen the load for each person. While this inherently makes sense, it creates an additional challenge of requiring team members to communicate with each other in such a way that all are aware of all anomalies and errors, and the observations and learnings are all known and shared across the group. In essence, the team needs to function as a single entity.

A multi-staged ML approach works as a single automated entity, and it should not require any manual training, whether in reviewing correlations for tuning algorithms or massaging data sets. The system should free up DevOps teams, so that they only have to respond to actual findings of root cause. A system should only need a few hours of log data to achieve proper levels of accuracy.

While AIOps is useful for reducing the overall event "noise" from the many observability tools in use in an organization, applying multi-stage unsupervised ML to logs is a great way of both detecting new types of failure modes as well as their root cause. Rather than just triaging a problem and coming up with a quick fix or workaround, the system can determine the true root cause and likely avoid other such problems in the future.

Ajay Singh is Founder and CEO of Zebrium

Hot Topics

The Latest

AI is the catalyst for significant investment in data teams as enterprises require higher-quality data to power their AI applications, according to the State of Analytics Engineering Report from dbt Labs ...

Misaligned architecture can lead to business consequences, with 93% of respondents reporting negative outcomes such as service disruptions, high operational costs and security challenges ...

A Gartner analyst recently suggested that GenAI tools could create 25% time savings for network operational teams. Where might these time savings come from? How are GenAI tools helping NetOps teams today, and what other tasks might they take on in the future as models continue improving? In general, these savings come from automating or streamlining manual NetOps tasks ...

IT and line-of-business teams are increasingly aligned in their efforts to close the data gap and drive greater collaboration to alleviate IT bottlenecks and offload growing demands on IT teams, according to The 2025 Automation Benchmark Report: Insights from IT Leaders on Enterprise Automation & the Future of AI-Driven Businesses from Jitterbit ...

A large majority (86%) of data management and AI decision makers cite protecting data privacy as a top concern, with 76% of respondents citing ROI on data privacy and AI initiatives across their organization, according to a new Harris Poll from Collibra ...

According to Gartner, Inc. the following six trends will shape the future of cloud over the next four years, ultimately resulting in new ways of working that are digital in nature and transformative in impact ...

2020 was the equivalent of a wedding with a top-shelf open bar. As businesses scrambled to adjust to remote work, digital transformation accelerated at breakneck speed. New software categories emerged overnight. Tech stacks ballooned with all sorts of SaaS apps solving ALL the problems — often with little oversight or long-term integration planning, and yes frequently a lot of duplicated functionality ... But now the music's faded. The lights are on. Everyone from the CIO to the CFO is checking the bill. Welcome to the Great SaaS Hangover ...

Regardless of OpenShift being a scalable and flexible software, it can be a pain to monitor since complete visibility into the underlying operations is not guaranteed ... To effectively monitor an OpenShift environment, IT administrators should focus on these five key elements and their associated metrics ...

An overwhelming majority of IT leaders (95%) believe the upcoming wave of AI-powered digital transformation is set to be the most impactful and intensive seen thus far, according to The Science of Productivity: AI, Adoption, And Employee Experience, a new report from Nexthink ...

Overall outage frequency and the general level of reported severity continue to decline, according to the Outage Analysis 2025 from Uptime Institute. However, cyber security incidents are on the rise and often have severe, lasting impacts ...