How Observability Helps Ingest and Normalize Data for DevOps Engineers
September 08, 2021

Richard Whitehead
Moogsoft

Share this

Humans naturally love structure. Just take books, for example. We've been ingesting and normalizing data through bookmaking since ancient times. In bookmaking, we transport, or ingest, data (in the form of text and images) from the spoken word or author's imagination to a physical structure. Covers denote the information's beginning and end, and a table of contents and chapters categorize, or normalize, the data.

The same logic applies to modern computer data. Humans prefer information that is easy to understand, and we make sense of unstructured data — whether it's text or time series data — by ingesting and normalizing it.

DevOps, SRE and other operations teams use observability solutions with AIOps to ingest and normalize data to get visibility into tech stacks from a centralized system, reduce noise and understand the data's context for quicker mean time to recovery (MTTR). With AI using these processes to produce actionable insights, teams are free to spend more time innovating and providing superior service assurance.

Let's explore AI's role in ingestion and normalization, and then dive into correlation and deduplication too:

How Is Data Ingested into an Observability Platform?

Solutions that provide observability with AIOps are flexible, incorporating data from a broad range of sources. These monitoring systems ingest event management data, like alerts, log events and time series data. Modern observability solutions also notify teams about system changes, which is critical considering an environmental change instigates most system failures. In the end, any data source is fair game, as long as the data tells you something about your real-time operational environment.

The data source dictates how your monitoring tool ingests the information. The first, more preferred method is a continuous data stream. The alternative is a pull mechanism, like a Prometheus pattern, which scrapes data at regular intervals. In older applications, you may have to use a creative plug-in or adapter that converts information into an accessible format and enables teams to query an application or system for data.

So why move all of this data into an observability platform? Transporting information from multiple sources and putting it into a centralized system can reveal the big picture behind the data.

How Is Data Normalized?

Once data is coming into your observability platform, it's helpful to normalize the information according to its common features. AI can extract information from unstructured data and elevate it to a feature, like a source or timestamp. These features allow you to sort or query the data or, in more sophisticated environments, apply AI-based techniques such as natural language processing (NLP).

As you normalize data, it helps to understand the incoming format and structure. If you're going to map fields and break down the message into component parts, understand what part of the message is variable and what part is static.

You can use enrichment techniques if data doesn't have a required field, appropriate feature or required information. Enrichment skirts the lack of information by finding a key to cross-reference with an external data source.

How Does Observability with AIOps Reduce Toil?

When you have normalized data, you can use AI to detect problems quickly through correlation and deduplication. Imagine if your system fails and you have to dig through hundreds of logs to see how the environment changed. That's time-consuming, not to mention boring.

Correlate, or group, data based on common characteristics like service, class or description field. Time is also handy operational information and serves as a practical classifier. Let's go back to our system failure. If you just made an environmental change, understanding the time the alerts came in helps pinpoint the problem.

Correlation can also mimic human behavior, which is a challenge for most computer systems. For example, online checkout processes are complex, with many integrated, interdependent parts. An intelligent observability tool with AIOps can correlate data alerts related to a checkout process using NLP. If that's an issue, your observability platform will group all of the alerts associated with the stem word "check," which accommodates derivations and variations like "checking," "Check," and "check out."

Let's move on to the benefits of deduplicating normalizing data. You're working and, suddenly, a "CPU overloaded" alert pops up. You start fixing the issue, but another "CPU overloaded" alert hits your inbox. And it's followed by 30 more similar alerts. That's distracting and not particularly useful.

Deduplication reduces noise and minimizes incident volumes by eliminating excessive copies of the data. Instead of the monitoring system telling you that the CPU is overloaded 32 separate times, AI compresses repeated messages into one stateful message. Deduplication can seem trivial, especially compared to techniques like NLP, but the devil is in the details. Understanding when a message indicates a new issue, rather than just a repeated message, must be considered.

Intelligent observability with AIOps centralizes data and makes it easier for teams to understand. And when these systems detect incidents, AI-enabled correlation and deduplication minimize the impact of this unplanned work. The downstream effects on DevOps practitioners and SRE teams are significant. These teams can spend less time putting out fires and more time focusing their time and attention on keeping up with the constant demand to innovate and delight customers.

Richard Whitehead is Chief Evangelist at Moogsoft
Share this

The Latest

January 25, 2022

Most enterprise IT organizations struggle with network and cloud infrastructure configuration management and worry that their networks will fail configuration compliance audits. This issue is amplified as enterprise network engineers continue to take the "if it's not broke, don't fix it" approach to network infrastructure management ...

January 24, 2022

The shift to containers and microservices is a key component of the digital transformation and shift to an all encompassing digital experience that modern customers have grown to expect. But these seismic shifts have also presented a nearly impossible task for IT teams: achieve ceaseless innovation whilst maintaining an ever more complex infrastructure environment, one that tends to produce vast volumes of data. Oh and can you also ensure that these systems are continuously available? ...

January 20, 2022

As part of 2022 APM Predictions list, APMdigest asked industry experts to predict how Digital Transformation will evolve and impact business in 2022 ...

January 19, 2022

As part of APMdigest's list of 2022 predictions, industry experts offer thoughtful and insightful predictions on how Cloud will evolve and impact business in 2022 ...

January 18, 2022

As part of APMdigest's list of 2022 predictions, industry experts offer thoughtful and insightful predictions on how Network Performance Management (NPM) and related technologies will evolve and impact business in 2022 ...

January 13, 2022

Gartner highlighted 6 trends that infrastructure and operations (I&O) leaders must start preparing for in the next 12-18 months ...

January 11, 2022

Technology is now foundational to financial companies' operations with many institutions relying on tech to deliver critical services. As a result, uptime is essential to customer satisfaction and company success, and systems must be subject to continuous monitoring. But modern IT architectures are disparate, complex and interconnected, and the data is too voluminous for the human mind to handle. Enter AIOps ...

January 11, 2022

Having a variety of tools to choose from creates challenges in telemetry data collection. Organizations find themselves managing multiple libraries for logging, metrics, and traces, with each vendor having its own APIs, SDKs, agents, and collectors. An open source, community-driven approach to observability will gain steam in 2022 to remove unnecessary complications by tapping into the latest advancements in observability practice ...

January 10, 2022

These are the trends that will set up your engineers and developers to deliver amazing software that powers amazing digital experiences that fuel your organization's growth in 2022 — and beyond ...

January 06, 2022

In a world where digital services have become a critical part of how we go about our daily lives, the risk of undergoing an outage has become even more significant. Outages can range in severity and impact companies of every size — while outages from larger companies in the social media space or a cloud provider tend to receive a lot of coverage, application downtime from even the most targeted companies can disrupt users' personal and business operations ...