Entering a Golden Age of Data Monitoring
June 13, 2018

Thomas Stocking
GroundWork Open Source

Share this

The importance of artificial intelligence and machine learning for customer insight, product support, operational efficiency, and capacity planning are well-established, however, the benefits of monitoring data in those use cases is still evolving. Three main factors obscuring the benefits of data monitoring are the infinite volume of data, its diversity, and inconsistency. However, it's these same factors that are fueling a Golden Age of systems monitoring.

1. Data Availability is Increasing

The trend over the last several years has been to collect more data – more than can ever be analyzed by humans. Data monitoring tools, by their very function, are in and of themselves a significant source of data. With the advent of NoSQL databases, optimize-on-read technologies, and the availability of very fast data consumers (influxdb, Opentsdb, Cloudera, etc.), the amount of data from monitoring systems is exploding.

2. Monitoring Data is Diverse

You would think more is better, as is often the case with data. That is what we learned in high school stats class, after all. However, more isn't always better, and in fact, most of the data we gather from monitoring is rather difficult to analyze programmatically. There are many reasons for this such as the complexity of modern IT infrastructures as well as the diversity of data.

Data diversity is an old IT problem. We collect data on network traffic, for example, using SNMP counters in router and switch MIBs. We also use netflow/sflow and do direct packet capture and decoding. So to even answer the question, "Why is the network slow?" we have at least three potential data sources, each with its own collection method, data types, indices, units and formats. It's not impossible to do analysis on the data we collect, but it is hard to gain insight when dealing with what my colleagues and I call "plumbing problems."

3. Monitoring Data is Inconsistent

You would think after all this time monitoring systems there would be a standard for the storage and indexing of metrics for analysis. Well, there is. In fact, there are several (Metrics 2.0, etc.). Yet, we are still dealing with inconsistency across tools in such basic areas as units, time scales, and even appropriate collection methods. With these inconsistencies, sampling data at five minutes vs. five seconds can yield vastly divergent results.

Benefits from Monitoring Data

Despite these issues, we are moving into a Golden Age of analysis. It's clear the most consistent parts of the monitoring data stream such as availability (as determined by health checks, for example) can be mined for very useful data, and used to create easily understood reports. If you combine this with endpoint testing, such as synthetic transactions from an end-user perspective, the picture of availability becomes much clearer and can be used to effectively manage SLAs.

Delving a level or two deeper, measurements of resource consumption over time can reveal trends that help with capacity planning and cost prediction. Time series analysis of sets of data that are consistent can reveal bottlenecks and even begin to point the way to root cause analysis, though we are still far away from automating this aspect.

The Future of Data Monitoring

There's a revolution in monitoring data with the advent of the cloud. We are suddenly able to gather a lot of data on the availability and performance of nearly every aspect of our systems that we run in the cloud.

In fact, as far as APIs go, there are even services that will consume all of your application traffic and analyze it for you, opening the possibility of dynamic tracing of transactions through your systems. If you are going cloud-native, you can take advantage of this area of unprecedented completeness and consistency of data, with minimal "plumbing" to worry about.

However, expect your job to get both easier and harder. Easier, since you will have more data, and sophisticated systems to analyze it. These systems and data it produces are becoming more homogeneous with cloud technologies and more consistent as the monitoring industry settles on standards. This will provide you better data for the systems you buy to analyze.

It will also be harder. When your systems fail, you won't easily find the data needed to fix things yourself. Similar to your cloud vendor, your monitoring system will be a complex and powerful toolset that will need time to learn, and you will absolutely be reliant on your providers for their expertise in its finer points.

Despite these challenges, the potential impact of effective data monitoring is significant. Effective data monitoring can help reduce outage and availability issues, support capacity planning, optimize capital investment, and help maintain productivity and profitability across an entire IT infrastructure. As IT systems become increasingly more complex, data monitoring becomes increasingly more vital.

Thomas Stocking is Co-Founder and VP of Product Strategy at GroundWork Open Source
Share this

The Latest

March 04, 2024

This year's Super Bowl drew in viewership of nearly 124 million viewers and made history as the most-watched live broadcast event since the 1969 moon landing. To support this spike in viewership, streaming companies like YouTube TV, Hulu and Paramount+ began preparing their IT infrastructure months in advance to ensure an exceptional viewer experience without outages or major interruptions. New Relic conducted a survey to understand the importance of a seamless viewing experience and the impact of outages during major streaming events such as the Super Bowl ...

March 01, 2024

As organizations continue to navigate the complexities of the digital era, which has been marked by exponential advancements in AI and technology, the strategic deployment of modern, practical applications has become indispensable for sustaining competitive advantage and realizing business goals. The Info-Tech Research Group report, Applications Priorities 2024, explores the following five initiatives for emerging and leading-edge technologies and practices that can enable IT and applications leaders to optimize their application portfolio and improve on capabilities needed to meet the ambitions of their organizations ...

February 29, 2024

Despite the growth in popularity of artificial intelligence (AI) and ML across a number of industries, there is still a huge amount of unrealized potential, with many businesses playing catch-up and still planning how ML solutions can best facilitate processes. Further progression could be limited without investment in specialized technical teams to drive development and integration ...

February 28, 2024

With over 200 streaming services to choose from, including multiple platforms featuring similar types of entertainment, users have little incentive to remain loyal to any given platform if it exhibits performance issues. Big names in streaming like Hulu, Amazon Prime and HBO Max invest thousands of hours into engineering observability and closed-loop monitoring to combat infrastructure and application issues, but smaller platforms struggle to remain competitive without access to the same resources ...

February 27, 2024

Generative AI has recently experienced unprecedented dramatic growth, making it one of the most exciting transformations the tech industry has seen in some time. However, this growth also poses a challenge for tech leaders who will be expected to deliver on the promise of new technology. In 2024, delivering tangible outcomes that meet the potential of AI, and setting up incubator projects for the future will be key tasks ...

February 26, 2024

SAP is a tool for automating business processes. Managing SAP solutions, especially with the shift to the cloud-based S/4HANA platform, can be intricate. To explore the concerns of SAP users during operational transformations and automation, a survey was conducted in mid-2023 by Digitate and Americas' SAP Users' Group ...

February 22, 2024

Some companies are just starting to dip their toes into developing AI capabilities, while (few) others can claim they have built a truly AI-first product. Regardless of where a company is on the AI journey, leaders must understand what it means to build every aspect of their product with AI in mind ...

February 21, 2024

Generative AI will usher in advantages within various industries. However, the technology is still nascent, and according to the recent Dynatrace survey there are many challenges and risks that organizations need to overcome to use this technology effectively ...

February 20, 2024

In today's digital era, monitoring and observability are indispensable in software and application development. Their efficacy lies in empowering developers to swiftly identify and address issues, enhance performance, and deliver flawless user experiences. Achieving these objectives requires meticulous planning, strategic implementation, and consistent ongoing maintenance. In this blog, we're sharing our five best practices to fortify your approach to application performance monitoring (APM) and observability ...

February 16, 2024

In MEAN TIME TO INSIGHT Episode 3, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses network security with Chris Steffen, VP of Research Covering Information Security, Risk, and Compliance Management at EMA ...