Entering a Golden Age of Data Monitoring
June 13, 2018

Thomas Stocking
GroundWork Open Source

Share this

The importance of artificial intelligence and machine learning for customer insight, product support, operational efficiency, and capacity planning are well-established, however, the benefits of monitoring data in those use cases is still evolving. Three main factors obscuring the benefits of data monitoring are the infinite volume of data, its diversity, and inconsistency. However, it's these same factors that are fueling a Golden Age of systems monitoring.

1. Data Availability is Increasing

The trend over the last several years has been to collect more data – more than can ever be analyzed by humans. Data monitoring tools, by their very function, are in and of themselves a significant source of data. With the advent of NoSQL databases, optimize-on-read technologies, and the availability of very fast data consumers (influxdb, Opentsdb, Cloudera, etc.), the amount of data from monitoring systems is exploding.

2. Monitoring Data is Diverse

You would think more is better, as is often the case with data. That is what we learned in high school stats class, after all. However, more isn't always better, and in fact, most of the data we gather from monitoring is rather difficult to analyze programmatically. There are many reasons for this such as the complexity of modern IT infrastructures as well as the diversity of data.

Data diversity is an old IT problem. We collect data on network traffic, for example, using SNMP counters in router and switch MIBs. We also use netflow/sflow and do direct packet capture and decoding. So to even answer the question, "Why is the network slow?" we have at least three potential data sources, each with its own collection method, data types, indices, units and formats. It's not impossible to do analysis on the data we collect, but it is hard to gain insight when dealing with what my colleagues and I call "plumbing problems."

3. Monitoring Data is Inconsistent

You would think after all this time monitoring systems there would be a standard for the storage and indexing of metrics for analysis. Well, there is. In fact, there are several (Metrics 2.0, etc.). Yet, we are still dealing with inconsistency across tools in such basic areas as units, time scales, and even appropriate collection methods. With these inconsistencies, sampling data at five minutes vs. five seconds can yield vastly divergent results.

Benefits from Monitoring Data

Despite these issues, we are moving into a Golden Age of analysis. It's clear the most consistent parts of the monitoring data stream such as availability (as determined by health checks, for example) can be mined for very useful data, and used to create easily understood reports. If you combine this with endpoint testing, such as synthetic transactions from an end-user perspective, the picture of availability becomes much clearer and can be used to effectively manage SLAs.

Delving a level or two deeper, measurements of resource consumption over time can reveal trends that help with capacity planning and cost prediction. Time series analysis of sets of data that are consistent can reveal bottlenecks and even begin to point the way to root cause analysis, though we are still far away from automating this aspect.

The Future of Data Monitoring

There's a revolution in monitoring data with the advent of the cloud. We are suddenly able to gather a lot of data on the availability and performance of nearly every aspect of our systems that we run in the cloud.

In fact, as far as APIs go, there are even services that will consume all of your application traffic and analyze it for you, opening the possibility of dynamic tracing of transactions through your systems. If you are going cloud-native, you can take advantage of this area of unprecedented completeness and consistency of data, with minimal "plumbing" to worry about.

However, expect your job to get both easier and harder. Easier, since you will have more data, and sophisticated systems to analyze it. These systems and data it produces are becoming more homogeneous with cloud technologies and more consistent as the monitoring industry settles on standards. This will provide you better data for the systems you buy to analyze.

It will also be harder. When your systems fail, you won't easily find the data needed to fix things yourself. Similar to your cloud vendor, your monitoring system will be a complex and powerful toolset that will need time to learn, and you will absolutely be reliant on your providers for their expertise in its finer points.

Despite these challenges, the potential impact of effective data monitoring is significant. Effective data monitoring can help reduce outage and availability issues, support capacity planning, optimize capital investment, and help maintain productivity and profitability across an entire IT infrastructure. As IT systems become increasingly more complex, data monitoring becomes increasingly more vital.

Thomas Stocking is Co-Founder and VP of Product Strategy at GroundWork Open Source
Share this

The Latest

July 28, 2021

Business leaders are in the unique position of having immediate access to huge amounts of data in today's smartphone and laptop-dominated world. They are also under pressure to make data-driven decisions and mobile business intelligence can one of the most valuable decision making tools in their arsenal ...

July 27, 2021

Unlike some AI initiatives, AIOps doesn't always necessitate the use of a data scientist, so don't let hiring expenses put your AIOps initiatives on hold. It is always nice to have IT team members with AI skills, but this becomes less critical as more intelligent solutions come into prominence that offer AIOps features out of the box, a readily deployable option for IT ...

July 26, 2021

AIOps is rapidly becoming a de-facto option for enterprises' IT strategies, with nearly immeasurable benefits to be provided. However, AIOps is still a relatively new discipline and misconceptions surrounding the technology's capabilities and uses have caused bottlenecks and roadblocks in its widespread adoption. So, what should organizations expect from AIOps? How can organizations that want to digitally transform their IT pursue AIOps for maximum benefit? ...

July 22, 2021

In response to the global pandemic, companies have given their workforce the tools they need to work remote. And research shows it has increased their engagement and productivity. But these gains are on the brink of being wiped out. According to a new study from Citrix Systems, Inc., employees feel they've been given too many tools and not enough efficient ways to execute. And it's hindering their ability to get things done ...

July 21, 2021

The third installment of Aptum's four-part Cloud Impact Study, A Bright Forecast on Cloud, presents data showing the benefits organizations gain from cloud computing, as well as mistakes to avoid during migration. As organizations migrate workloads to different cloud platforms, they often run into unexpected challenges due to a lack of proactive planning. Here are a few key findings from Part 3 of the Cloud Impact Study ...

July 20, 2021

Currently, (and most likely well into the future) the overwhelming majority of organizations still need to monitor and maintain enterprise applications. Moreover, where these are complex systems developed, debugged and refined over years, often decades, around a business's core processes, there can also be very strong practical arguments for viewing them as classics. They can offer a valuable legacy, one best left where it is, doing what it does, how it always has done ...

July 19, 2021

Anti-patterns involve realizing a problem and implementing a non-optimal solution that is broadly embraced as the go-to method for solving that problem. This solution sounds good in theory, but for one reason or another it is not the best means of solving the problem. Anti-patterns are common across IT as well, especially around application monitoring and observability. One that is particularly prevalent is in response to the increasing complexity of cloud-native infrastructure and applications ...

July 15, 2021

The introduction of the latest technology — such as AI and machine learning — can be seen as a way for organizations to accelerate growth, increase efficiency, and improve customer service. However, the truth is that the technology alone will do little to deliver on these business outcomes. AI for IT operations (AIOps) is one area where the application of technology, if not matched with organizational maturity readiness, will fail to deliver all the promised benefits ...

July 14, 2021

SREs that fail to deliver customer value run the risk of being stuck in an operational toil rut. Conversely, businesses failing to recognize the bi-modal nature and importance of SRE activities run the risk of losing talented employees and their competitive edge ...

July 13, 2021

As part of digital transformation initiatives, IT teams are quickly adopting AIOps solutions to accommodate a new multifaceted infrastructure. However, there are still several roadblocks IT leaders must overcome when adopting AIOps — namely, understanding how to showcase ROI and changing their team's cultural mindset around adopting a new strategy ...