Entering a Golden Age of Data Monitoring
June 13, 2018

Thomas Stocking
GroundWork Open Source

Share this

The importance of artificial intelligence and machine learning for customer insight, product support, operational efficiency, and capacity planning are well-established, however, the benefits of monitoring data in those use cases is still evolving. Three main factors obscuring the benefits of data monitoring are the infinite volume of data, its diversity, and inconsistency. However, it's these same factors that are fueling a Golden Age of systems monitoring.

1. Data Availability is Increasing

The trend over the last several years has been to collect more data – more than can ever be analyzed by humans. Data monitoring tools, by their very function, are in and of themselves a significant source of data. With the advent of NoSQL databases, optimize-on-read technologies, and the availability of very fast data consumers (influxdb, Opentsdb, Cloudera, etc.), the amount of data from monitoring systems is exploding.

2. Monitoring Data is Diverse

You would think more is better, as is often the case with data. That is what we learned in high school stats class, after all. However, more isn't always better, and in fact, most of the data we gather from monitoring is rather difficult to analyze programmatically. There are many reasons for this such as the complexity of modern IT infrastructures as well as the diversity of data.

Data diversity is an old IT problem. We collect data on network traffic, for example, using SNMP counters in router and switch MIBs. We also use netflow/sflow and do direct packet capture and decoding. So to even answer the question, "Why is the network slow?" we have at least three potential data sources, each with its own collection method, data types, indices, units and formats. It's not impossible to do analysis on the data we collect, but it is hard to gain insight when dealing with what my colleagues and I call "plumbing problems."

3. Monitoring Data is Inconsistent

You would think after all this time monitoring systems there would be a standard for the storage and indexing of metrics for analysis. Well, there is. In fact, there are several (Metrics 2.0, etc.). Yet, we are still dealing with inconsistency across tools in such basic areas as units, time scales, and even appropriate collection methods. With these inconsistencies, sampling data at five minutes vs. five seconds can yield vastly divergent results.

Benefits from Monitoring Data

Despite these issues, we are moving into a Golden Age of analysis. It's clear the most consistent parts of the monitoring data stream such as availability (as determined by health checks, for example) can be mined for very useful data, and used to create easily understood reports. If you combine this with endpoint testing, such as synthetic transactions from an end-user perspective, the picture of availability becomes much clearer and can be used to effectively manage SLAs.

Delving a level or two deeper, measurements of resource consumption over time can reveal trends that help with capacity planning and cost prediction. Time series analysis of sets of data that are consistent can reveal bottlenecks and even begin to point the way to root cause analysis, though we are still far away from automating this aspect.

The Future of Data Monitoring

There's a revolution in monitoring data with the advent of the cloud. We are suddenly able to gather a lot of data on the availability and performance of nearly every aspect of our systems that we run in the cloud.

In fact, as far as APIs go, there are even services that will consume all of your application traffic and analyze it for you, opening the possibility of dynamic tracing of transactions through your systems. If you are going cloud-native, you can take advantage of this area of unprecedented completeness and consistency of data, with minimal "plumbing" to worry about.

However, expect your job to get both easier and harder. Easier, since you will have more data, and sophisticated systems to analyze it. These systems and data it produces are becoming more homogeneous with cloud technologies and more consistent as the monitoring industry settles on standards. This will provide you better data for the systems you buy to analyze.

It will also be harder. When your systems fail, you won't easily find the data needed to fix things yourself. Similar to your cloud vendor, your monitoring system will be a complex and powerful toolset that will need time to learn, and you will absolutely be reliant on your providers for their expertise in its finer points.

Despite these challenges, the potential impact of effective data monitoring is significant. Effective data monitoring can help reduce outage and availability issues, support capacity planning, optimize capital investment, and help maintain productivity and profitability across an entire IT infrastructure. As IT systems become increasingly more complex, data monitoring becomes increasingly more vital.

Thomas Stocking is Co-Founder and VP of Product Strategy at GroundWork Open Source
Share this

The Latest

October 20, 2020

Although cost control/expense management remains top of mind, organizations are realizing the necessity of technology solutions to enable them to steer the business during these turbulent times, according to IDG's CIO Pandemic Business Impact Study ...

October 19, 2020

The COVID-19 pandemic has compressed six years of modernization projects into 6 months. According to a recent report, IT leaders have accelerated projects aimed at increasing productivity and business agility, improving application performance and end-user experience, and driving additional revenue through existing channels ...

October 15, 2020

There is no doubt that automation has become the key aspect of modern IT management. The end-user computing market is no exception. With a large and complex technology stack and a huge number of applications, EUC specialists need to handle an ever-increasing number of changes at an ever-increasing rate. Many IT organizations are starting to realize that they can no longer control the flow of changes. It is time to think about how to facilitate change ...

October 14, 2020

Starting this September, the lifespan of an SSL/TLS certificate has been limited to 398 days, a reduction from the previous maximum certificate lifetime of 825 days. With this change, everyone needs to more carefully monitor SSL certificate expiration and server characteristics ...

October 13, 2020

Nearly 6 in 10 responding organizations have accelerated their digital transformations due to the COVID-19 pandemic, according to The IBM Institute for Business Value study COVID-19 and the Future of Business ...

October 08, 2020

Two-thirds (67%) of those surveyed expect the sheer quantity of data to grow nearly five times by 2025, according to a new report from Splunk: The Data Age Is Here. Are You Ready? ...

October 07, 2020

Gaming introduced the world to a whole new range of experiences through augmented reality (AR) and virtual reality (VR). And consumers are really catching on. To unlock the potential of these platforms, enterprises must ensure massive amounts of data can be transferred quickly and reliably to ensure an acceptable quality of experience. As such, this means that enterprises will need to turn to a 5G infrastructure powered by an adaptive network ...

October 06, 2020

A distributed, remote workforce is the new business reality. How can businesses keep operations going smoothly and quickly resolve issues when IT staff is in San Jose, employee A is working remotely in Denver at their home and employee B is a salesperson still doing some road traveling? The key is an IT architecture that promotes and supports "self-healing" at the endpoint to take care of issues before they impact employees. The essential element to achieve this is hyper-automation ...

October 05, 2020

In Episode 10, Prem Naraindas, CEO of Katonic.ai, joins the AI+ITOPS Podcast to discuss how emerging technologies can make life better for ITOps ...

October 02, 2020

Sean McDermott on the AI+ITOPS Podcast: "AIOps is really about the processing of vast amounts of data and the ability to move into a more analytical, prescriptive and automated methodology."