Finding the Needle in the Haystack: How Machine Learning Will Revolutionize Root Cause Analysis
September 29, 2021

Ajay Singh
Zebrium

Share this

When a website or app fails or falters, the standard operating procedure is to assemble a sizable team to quickly "divide and conquer" to find a solution. The details of the problem can usually be found somewhere among millions of log events and metrics, leading to slow and painstaking searches that can take hours and often involve handoffs between experts in different areas of the software. The immediate goal in these situations is not to be comprehensive, but rather to troubleshoot until you find a solution that remedies the symptom, even if the underlying root cause is not addressed.

The entire troubleshooting process takes time — generally lots and lots of it — and experience. Development teams tend to be chronically short-staffed and overworked, so adding the burden to hunt for the cause of an app problem means a substantial opportunity cost among other things. To help with the task, most companies leverage multiple best-of-breed observability tools including application performance management (APM), tracing, monitoring and log management. These are used to detect and find a solution to the problem being experienced. Although each tool provides useful data, in total, it can be hard for a person to interpret what is important and what is less so.

Instead of a disruptive and often frenzied, big team approach, this kind of challenge is a perfect application for machine learning (ML) to sift through volumes of data and find meaningful patterns or anomalies that can explain the root cause.

AIOps — using AI for IT operations — has emerged as a possible solution for correlating data from multiple tools to reduce noise and translate events into something more meaningful for a user. On the plus side, AIOps solutions are designed to handle events from a wide range of tools, making them versatile. On the negative side, most AIOps solutions require very long training periods (typically many months) against labeled data sets. These solutions also fall short, because they are designed to correlate events against known problems rather than find the root cause of new or unknown failure modes. This is a particular weakness in fast changing cloud-native environments, where new failure modes crop up on a regular basis.

In order to find the root cause of new failure modes, a different type of AI approach is needed. Since logs often contain the source of truth when a software failure occurs, one approach is to use ML on logs. The concept is to identify just the anomalous patterns in the logs that explain the details of the problem. This can be challenging since logs are mostly unstructured and "noisy." On top of that, log volumes are typically huge with data coming from many different log streams, each with a large number of log lines. Historical approaches have focused on basic anomaly detection which not only produce verbose results that require human interpretation, but also don't explain correlations across micro-services, often entirely missing key details of the problem.

It turns out, the most effective way to perform ML on logs is to use a pipeline with multiple different ML strategies depending on stage of the process. Specialized ML starts by self-learning (i.e. unsupervised) how to structure and categorize the logs — this produces a solid foundation for the remaining ML stages. Next, the ML learns the patterns of each type of log event. Once this learning has occurred, the ML system can identify anomalous log events within each log stream (events that break pattern).

Finally, to pull out the signal from the noise, the system needs to find correlations between anomalies and errors across multiple log streams. This process provides an effective way of uncovering just the sequence of log lines that describe the problem and its root cause. In doing so, it allows for accurate detection of new types of failure modes as well as the information needed to identify root cause.

Such a methodology is similar to the approach taken by skilled engineers — understanding the logs, identifying rare and high-severity events and then finding correlations between clusters of these events across multiple log streams. But it requires considerable time for humans to do this. In practice, the task would be spread out across multiple people in a divide and conquer mode in attempt to accelerate the process and lessen the load for each person. While this inherently makes sense, it creates an additional challenge of requiring team members to communicate with each other in such a way that all are aware of all anomalies and errors, and the observations and learnings are all known and shared across the group. In essence, the team needs to function as a single entity.

A multi-staged ML approach works as a single automated entity, and it should not require any manual training, whether in reviewing correlations for tuning algorithms or massaging data sets. The system should free up DevOps teams, so that they only have to respond to actual findings of root cause. A system should only need a few hours of log data to achieve proper levels of accuracy.

While AIOps is useful for reducing the overall event "noise" from the many observability tools in use in an organization, applying multi-stage unsupervised ML to logs is a great way of both detecting new types of failure modes as well as their root cause. Rather than just triaging a problem and coming up with a quick fix or workaround, the system can determine the true root cause and likely avoid other such problems in the future.

Ajay Singh is Founder and CEO of Zebrium
Share this

The Latest

March 18, 2024

Gartner has highlighted the top trends that will impact technology providers in 2024: Generative AI (GenAI) is dominating the technical and product agenda of nearly every tech provider ...

March 15, 2024

In MEAN TIME TO INSIGHT Episode 4 - Part 1, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses artificial intelligence and network management ...

March 14, 2024

The integration and maintenance of AI-enabled Software as a Service (SaaS) applications have emerged as pivotal points in enterprise AI implementation strategies, offering both significant challenges and promising benefits. Despite the enthusiasm surrounding AI's potential impact, the reality of its implementation presents hurdles. Currently, over 90% of enterprises are grappling with limitations in integrating AI into their tech stack ...

March 13, 2024

In the intricate landscape of IT infrastructure, one critical component often relegated to the back burner is Active Directory (AD) forest recovery — an oversight with costly consequences ...

March 12, 2024

eBPF is a technology that allows users to run custom programs inside the Linux kernel, which changes the behavior of the kernel and makes execution up to 10x faster(link is external) and more efficient for key parts of what makes our computing lives work. That includes observability, networking and security ...

March 11, 2024

Data mesh, an increasingly important decentralized approach to data architecture and organizational design, focuses on treating data as a product, emphasizing domain-oriented data ownership, self-service tools and federated governance. The 2024 State of the Data Lakehouse report from Dremio presents evidence of the growing adoption of data mesh architectures in enterprises ... The report highlights that the drive towards data mesh is increasingly becoming a business strategy to enhance agility and speed in problem-solving and innovation ...

March 07, 2024
In this digital era, consumers prefer a seamless user experience, and here, the significance of performance testing cannot be overstated. Application performance testing is essential in ensuring that your software products, websites, or other related systems operate seamlessly under varying conditions. However, the cost of poor performance extends beyond technical glitches and slow load times; it can directly affect customer satisfaction and brand reputation. Understand the tangible and intangible consequences of poor application performance and how it can affect your business ...
March 06, 2024

Too much traffic can crash a website ... That stampede of traffic is even more horrifying when it's part of a malicious denial of service attack ... These attacks are becoming more common, more sophisticated and increasingly tied to ransomware-style demands. So it's no wonder that the threat of DDoS remains one of the many things that keep IT and marketing leaders up at night ...

March 05, 2024

Today, applications serve as the backbone of businesses, and therefore, ensuring optimal performance has never been more critical. This is where application performance monitoring (APM) emerges as an indispensable tool, empowering organizations to safeguard their applications proactively, match user expectations, and drive growth. But APM is not without its challenges. Choosing to implement APM is a path that's not easily realized, even if it offers great benefits. This blog deals with the potential hurdles that may manifest when you actualize your APM strategy in your IT application environment ...

March 04, 2024

This year's Super Bowl drew in viewership of nearly 124 million viewers and made history as the most-watched live broadcast event since the 1969 moon landing. To support this spike in viewership, streaming companies like YouTube TV, Hulu and Paramount+ began preparing their IT infrastructure months in advance to ensure an exceptional viewer experience without outages or major interruptions. New Relic conducted a survey to understand the importance of a seamless viewing experience and the impact of outages during major streaming events such as the Super Bowl ...