Skip to main content

Analytics That Matter - For APM-Generated Big Data

“Big Data” is everywhere. What does it mean? Just as Cloud Computing bursted onto the scene a few years ago, it depends on whom you ask.

Traditionally, in the Business Intelligence (BI) world, Big Data included analyzing historical business data from large data warehouses with the purpose of identifying long-term trends that could be leveraged in consumer business strategies. In recent years, Big Data has been a term talked about in the IT industry as an application of technology to attack extremely large, unstructured data sets that can reside both within and outside of an organization. If you look at a recent definition of Big Data, it is a term applied to data sets whose size has grown beyond the capability of commonly used software tools to capture, manage and analyze within a tolerable period of times for different use cases.

Application Performance Management (APM) is an extremely relevant use case and has a developing “Big Data” problem. Several factors are contributing to the explosive growth and type of data that must be analyzed and/or correlated in application performance monitoring and business service management (BSM).

First, the number of components that make up today’s mission critical applications has exploded. Instead of hundreds of servers for an application, nowadays, because of virtualization, you can easily be talking about thousands of virtual servers and objects for web applications.

Secondly, the diversity of data that people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply understand traditional IT infrastructure performance based on server operating system, network traffic, and storage capacity. Application Performance analysis is now based on the relationships of IT infrastructure components, application performance metrics from applications and application servers, business activity monitors (BAM) data, customer experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the aggregated transactional data, there are new systems that capture transactions’ actual path encompassing the entire application stack.

Finally, the requirements for analysis speed and data granularity have also increased significantly. Mission critical application performance now requires real-time or near real-time data analysis. When we were doing server availability and performance monitoring 10 years ago, it was the norm to collect and analyze data every 15 minutes.  Today, this has evolved to data analysis every 5 minutes or less with sub-minute data collection where all transaction paths are collected for data analysis. When mapped out, it's easy to see the enormous growth particularly when you look at APM related storage requirements that are quickly growing from gigabytes to terabytes and tomorrow petabytes.

Where APM and Big Data Meet the Cloud

All this data requires extremely complex analysis and correlation in order to truly understand performance of critical applications.

One of Netuitive’s large enterprise customers reported that it monitors and correlates a billion infrastructure and application data points and business metrics daily as part of its global service delivery. This is what I am referring to as APM-generated Big Data.

In addition to the shear number of data points, IT operators are expected to provide real-time analysis to the business and long-term storage for post-mortem analysis, capacity planning and compliance.

So where does this lead us? This is where APM and Big Data meet the Cloud. The cloud can deliver cheaper and more flexible storage and computing power crucial to analytics for Big Data. It also has the capability to be much more elastic for your APM data storage and analytics needs. Organizations can actually think about storing years of collected and aggregated APM data for compliance and analysis purposes without the cost being prohibitive.

But what does this mean to vendors in the APM space? 

First of all, the analytics platform for APM data has to evolve to be able to process the growing number of different data sources across business, customer experience, applications and IT domains. Netuitives’ “Open” analytics platform is engineered to address virtually any data source in real-time.

Secondly, data storage and access time will be critical even as APM data volumes continue to explode, so not only does the technology need to be able to run in the cloud, but the traditional pull-based data collection architecture has to evolve into a push based model with an horizontally scalable computing and storage architecture in order to become virtually limitless in terms of scalability. This is critical for larger organizations as “real” time no longer means analysis every 5 to 15 minutes, but sub-minute analytics.

Lastly, because storage and computing costs should not significantly exceed the cost of analytics software for a solution to be viable, Netuitive is advancing its product architecture to leverage NoSQL columnar data store as a replacement to traditional database.

While our R&D challenges are complex, the goal is simple: provide APM Analytics that matter by enabling our enterprise customers to process billions of infrastructure, application, and business metrics from hundreds of thousands of managed elements at 10x less cost than existing infrastructures.

ABOUT Jean-François Huard

Jean-François Huard is Chief Technical Officer and Vice President of Research and Development at Netuitive, Inc. In this role he is responsible for leading the company’s vision and technology innovation effort.

Previously, Huard was Chief Network Architect and Vice President of Network Engineering at InvisibleHand Networks, a start-up company funded by Polaris Venture Partners. Earlier, he led the technology team at Xbind, Inc. Earlier in his career, Huard worked in network fault management at AT&T Bell Labs, and was a member of the research staff at the Center for Telecommunications Research.

Jean-François contributed to the definition of the international MPEG-4 standard, and was chair and technical editor of the IEEE P1520.2 working group. He has authored or co-authored many scientific papers published in technical journals and conferences, standard contributions, and has filed multiple patents.

Hot Topics

The Latest

E-commerce is set to skyrocket with a 9% rise over the next few years ... To thrive in this competitive environment, retailers must identify digital resilience as their top priority. In a world where savvy shoppers expect 24/7 access to online deals and experiences, any unexpected downtime to digital services can lead to significant financial losses, damage to brand reputation, abandoned carts with designer shoes, and additional issues ...

Efficiency is a highly-desirable objective in business ... We're seeing this scenario play out in enterprises around the world as they continue to struggle with infrastructures and remote work models with an eye toward operational efficiencies. In contrast to that goal, a recent Broadcom survey of global IT and network professionals found widespread adoption of these strategies is making the network more complex and hampering observability, leading to uptime, performance and security issues. Let's look more closely at these challenges ...

Image
Broadcom

The 2025 Catchpoint SRE Report dives into the forces transforming the SRE landscape, exploring both the challenges and opportunities ahead. Let's break down the key findings and what they mean for SRE professionals and the businesses relying on them ...

Image
Catchpoint

The pressure on IT teams has never been greater. As data environments grow increasingly complex, resource shortages are emerging as a major obstacle for IT leaders striving to meet the demands of modern infrastructure management ... According to DataStrike's newly released 2025 Data Infrastructure Survey Report, more than half (54%) of IT leaders cite resource limitations as a top challenge, highlighting a growing trend toward outsourcing as a solution ...

Image
Datastrike

Gartner revealed its top strategic predictions for 2025 and beyond. Gartner's top predictions explore how generative AI (GenAI) is affecting areas where most would assume only humans can have lasting impact ...

The adoption of artificial intelligence (AI) is accelerating across the telecoms industry, with 88% of fixed broadband service providers now investigating or trialing AI automation to enhance their fixed broadband services, according to new research from Incognito Software Systems and Omdia ...

 

AWS is a cloud-based computing platform known for its reliability, scalability, and flexibility. However, as helpful as its comprehensive infrastructure is, disparate elements and numerous siloed components make it difficult for admins to visualize the cloud performance in detail. It requires meticulous monitoring techniques and deep visibility to understand cloud performance and analyze operational efficiency in detail to ensure seamless cloud operations ...

Imagine a future where software, once a complex obstacle, becomes a natural extension of daily workflow — an intuitive, seamless experience that maximizes productivity and efficiency. This future is no longer a distant vision but a reality being crafted by the transformative power of Artificial Intelligence ...

Enterprise data sprawl already challenges companies' ability to protect and back up their data. Much of this information is never fully secured, leaving organizations vulnerable. Now, as GenAI platforms emerge as yet another environment where enterprise data is consumed, transformed, and created, this fragmentation is set to intensify ...

Image
Crashplan

OpenTelemetry (OTel) has revolutionized the way we approach observability by standardizing the collection of telemetry data ... Here are five myths — and truths — to help elevate your OTel integration by harnessing the untapped power of logs ...

Analytics That Matter - For APM-Generated Big Data

“Big Data” is everywhere. What does it mean? Just as Cloud Computing bursted onto the scene a few years ago, it depends on whom you ask.

Traditionally, in the Business Intelligence (BI) world, Big Data included analyzing historical business data from large data warehouses with the purpose of identifying long-term trends that could be leveraged in consumer business strategies. In recent years, Big Data has been a term talked about in the IT industry as an application of technology to attack extremely large, unstructured data sets that can reside both within and outside of an organization. If you look at a recent definition of Big Data, it is a term applied to data sets whose size has grown beyond the capability of commonly used software tools to capture, manage and analyze within a tolerable period of times for different use cases.

Application Performance Management (APM) is an extremely relevant use case and has a developing “Big Data” problem. Several factors are contributing to the explosive growth and type of data that must be analyzed and/or correlated in application performance monitoring and business service management (BSM).

First, the number of components that make up today’s mission critical applications has exploded. Instead of hundreds of servers for an application, nowadays, because of virtualization, you can easily be talking about thousands of virtual servers and objects for web applications.

Secondly, the diversity of data that people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply understand traditional IT infrastructure performance based on server operating system, network traffic, and storage capacity. Application Performance analysis is now based on the relationships of IT infrastructure components, application performance metrics from applications and application servers, business activity monitors (BAM) data, customer experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the aggregated transactional data, there are new systems that capture transactions’ actual path encompassing the entire application stack.

Finally, the requirements for analysis speed and data granularity have also increased significantly. Mission critical application performance now requires real-time or near real-time data analysis. When we were doing server availability and performance monitoring 10 years ago, it was the norm to collect and analyze data every 15 minutes.  Today, this has evolved to data analysis every 5 minutes or less with sub-minute data collection where all transaction paths are collected for data analysis. When mapped out, it's easy to see the enormous growth particularly when you look at APM related storage requirements that are quickly growing from gigabytes to terabytes and tomorrow petabytes.

Where APM and Big Data Meet the Cloud

All this data requires extremely complex analysis and correlation in order to truly understand performance of critical applications.

One of Netuitive’s large enterprise customers reported that it monitors and correlates a billion infrastructure and application data points and business metrics daily as part of its global service delivery. This is what I am referring to as APM-generated Big Data.

In addition to the shear number of data points, IT operators are expected to provide real-time analysis to the business and long-term storage for post-mortem analysis, capacity planning and compliance.

So where does this lead us? This is where APM and Big Data meet the Cloud. The cloud can deliver cheaper and more flexible storage and computing power crucial to analytics for Big Data. It also has the capability to be much more elastic for your APM data storage and analytics needs. Organizations can actually think about storing years of collected and aggregated APM data for compliance and analysis purposes without the cost being prohibitive.

But what does this mean to vendors in the APM space? 

First of all, the analytics platform for APM data has to evolve to be able to process the growing number of different data sources across business, customer experience, applications and IT domains. Netuitives’ “Open” analytics platform is engineered to address virtually any data source in real-time.

Secondly, data storage and access time will be critical even as APM data volumes continue to explode, so not only does the technology need to be able to run in the cloud, but the traditional pull-based data collection architecture has to evolve into a push based model with an horizontally scalable computing and storage architecture in order to become virtually limitless in terms of scalability. This is critical for larger organizations as “real” time no longer means analysis every 5 to 15 minutes, but sub-minute analytics.

Lastly, because storage and computing costs should not significantly exceed the cost of analytics software for a solution to be viable, Netuitive is advancing its product architecture to leverage NoSQL columnar data store as a replacement to traditional database.

While our R&D challenges are complex, the goal is simple: provide APM Analytics that matter by enabling our enterprise customers to process billions of infrastructure, application, and business metrics from hundreds of thousands of managed elements at 10x less cost than existing infrastructures.

ABOUT Jean-François Huard

Jean-François Huard is Chief Technical Officer and Vice President of Research and Development at Netuitive, Inc. In this role he is responsible for leading the company’s vision and technology innovation effort.

Previously, Huard was Chief Network Architect and Vice President of Network Engineering at InvisibleHand Networks, a start-up company funded by Polaris Venture Partners. Earlier, he led the technology team at Xbind, Inc. Earlier in his career, Huard worked in network fault management at AT&T Bell Labs, and was a member of the research staff at the Center for Telecommunications Research.

Jean-François contributed to the definition of the international MPEG-4 standard, and was chair and technical editor of the IEEE P1520.2 working group. He has authored or co-authored many scientific papers published in technical journals and conferences, standard contributions, and has filed multiple patents.

Hot Topics

The Latest

E-commerce is set to skyrocket with a 9% rise over the next few years ... To thrive in this competitive environment, retailers must identify digital resilience as their top priority. In a world where savvy shoppers expect 24/7 access to online deals and experiences, any unexpected downtime to digital services can lead to significant financial losses, damage to brand reputation, abandoned carts with designer shoes, and additional issues ...

Efficiency is a highly-desirable objective in business ... We're seeing this scenario play out in enterprises around the world as they continue to struggle with infrastructures and remote work models with an eye toward operational efficiencies. In contrast to that goal, a recent Broadcom survey of global IT and network professionals found widespread adoption of these strategies is making the network more complex and hampering observability, leading to uptime, performance and security issues. Let's look more closely at these challenges ...

Image
Broadcom

The 2025 Catchpoint SRE Report dives into the forces transforming the SRE landscape, exploring both the challenges and opportunities ahead. Let's break down the key findings and what they mean for SRE professionals and the businesses relying on them ...

Image
Catchpoint

The pressure on IT teams has never been greater. As data environments grow increasingly complex, resource shortages are emerging as a major obstacle for IT leaders striving to meet the demands of modern infrastructure management ... According to DataStrike's newly released 2025 Data Infrastructure Survey Report, more than half (54%) of IT leaders cite resource limitations as a top challenge, highlighting a growing trend toward outsourcing as a solution ...

Image
Datastrike

Gartner revealed its top strategic predictions for 2025 and beyond. Gartner's top predictions explore how generative AI (GenAI) is affecting areas where most would assume only humans can have lasting impact ...

The adoption of artificial intelligence (AI) is accelerating across the telecoms industry, with 88% of fixed broadband service providers now investigating or trialing AI automation to enhance their fixed broadband services, according to new research from Incognito Software Systems and Omdia ...

 

AWS is a cloud-based computing platform known for its reliability, scalability, and flexibility. However, as helpful as its comprehensive infrastructure is, disparate elements and numerous siloed components make it difficult for admins to visualize the cloud performance in detail. It requires meticulous monitoring techniques and deep visibility to understand cloud performance and analyze operational efficiency in detail to ensure seamless cloud operations ...

Imagine a future where software, once a complex obstacle, becomes a natural extension of daily workflow — an intuitive, seamless experience that maximizes productivity and efficiency. This future is no longer a distant vision but a reality being crafted by the transformative power of Artificial Intelligence ...

Enterprise data sprawl already challenges companies' ability to protect and back up their data. Much of this information is never fully secured, leaving organizations vulnerable. Now, as GenAI platforms emerge as yet another environment where enterprise data is consumed, transformed, and created, this fragmentation is set to intensify ...

Image
Crashplan

OpenTelemetry (OTel) has revolutionized the way we approach observability by standardizing the collection of telemetry data ... Here are five myths — and truths — to help elevate your OTel integration by harnessing the untapped power of logs ...