Analytics That Matter - For APM-Generated Big Data
September 20, 2012
Jean-François Huard, Ph.D.
Share this

“Big Data” is everywhere. What does it mean? Just as Cloud Computing bursted onto the scene a few years ago, it depends on whom you ask.

Traditionally, in the Business Intelligence (BI) world, Big Data included analyzing historical business data from large data warehouses with the purpose of identifying long-term trends that could be leveraged in consumer business strategies. In recent years, Big Data has been a term talked about in the IT industry as an application of technology to attack extremely large, unstructured data sets that can reside both within and outside of an organization. If you look at a recent definition of Big Data, it is a term applied to data sets whose size has grown beyond the capability of commonly used software tools to capture, manage and analyze within a tolerable period of times for different use cases.

Application Performance Management (APM) is an extremely relevant use case and has a developing “Big Data” problem. Several factors are contributing to the explosive growth and type of data that must be analyzed and/or correlated in application performance monitoring and business service management (BSM).

First, the number of components that make up today’s mission critical applications has exploded. Instead of hundreds of servers for an application, nowadays, because of virtualization, you can easily be talking about thousands of virtual servers and objects for web applications.

Secondly, the diversity of data that people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply understand traditional IT infrastructure performance based on server operating system, network traffic, and storage capacity. Application Performance analysis is now based on the relationships of IT infrastructure components, application performance metrics from applications and application servers, business activity monitors (BAM) data, customer experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the aggregated transactional data, there are new systems that capture transactions’ actual path encompassing the entire application stack.

Finally, the requirements for analysis speed and data granularity have also increased significantly. Mission critical application performance now requires real-time or near real-time data analysis. When we were doing server availability and performance monitoring 10 years ago, it was the norm to collect and analyze data every 15 minutes.  Today, this has evolved to data analysis every 5 minutes or less with sub-minute data collection where all transaction paths are collected for data analysis. When mapped out, it's easy to see the enormous growth particularly when you look at APM related storage requirements that are quickly growing from gigabytes to terabytes and tomorrow petabytes.

Where APM and Big Data Meet the Cloud

All this data requires extremely complex analysis and correlation in order to truly understand performance of critical applications.

One of Netuitive’s large enterprise customers reported that it monitors and correlates a billion infrastructure and application data points and business metrics daily as part of its global service delivery. This is what I am referring to as APM-generated Big Data.

In addition to the shear number of data points, IT operators are expected to provide real-time analysis to the business and long-term storage for post-mortem analysis, capacity planning and compliance.

So where does this lead us? This is where APM and Big Data meet the Cloud. The cloud can deliver cheaper and more flexible storage and computing power crucial to analytics for Big Data. It also has the capability to be much more elastic for your APM data storage and analytics needs. Organizations can actually think about storing years of collected and aggregated APM data for compliance and analysis purposes without the cost being prohibitive.

But what does this mean to vendors in the APM space? 

First of all, the analytics platform for APM data has to evolve to be able to process the growing number of different data sources across business, customer experience, applications and IT domains. Netuitives’ “Open” analytics platform is engineered to address virtually any data source in real-time.

Secondly, data storage and access time will be critical even as APM data volumes continue to explode, so not only does the technology need to be able to run in the cloud, but the traditional pull-based data collection architecture has to evolve into a push based model with an horizontally scalable computing and storage architecture in order to become virtually limitless in terms of scalability. This is critical for larger organizations as “real” time no longer means analysis every 5 to 15 minutes, but sub-minute analytics.

Lastly, because storage and computing costs should not significantly exceed the cost of analytics software for a solution to be viable, Netuitive is advancing its product architecture to leverage NoSQL columnar data store as a replacement to traditional database.

While our R&D challenges are complex, the goal is simple: provide APM Analytics that matter by enabling our enterprise customers to process billions of infrastructure, application, and business metrics from hundreds of thousands of managed elements at 10x less cost than existing infrastructures.

ABOUT Jean-François Huard

Jean-François Huard is Chief Technical Officer and Vice President of Research and Development at Netuitive, Inc. In this role he is responsible for leading the company’s vision and technology innovation effort.

Previously, Huard was Chief Network Architect and Vice President of Network Engineering at InvisibleHand Networks, a start-up company funded by Polaris Venture Partners. Earlier, he led the technology team at Xbind, Inc. Earlier in his career, Huard worked in network fault management at AT&T Bell Labs, and was a member of the research staff at the Center for Telecommunications Research.

Jean-François contributed to the definition of the international MPEG-4 standard, and was chair and technical editor of the IEEE P1520.2 working group. He has authored or co-authored many scientific papers published in technical journals and conferences, standard contributions, and has filed multiple patents.

Share this

The Latest

April 15, 2024

Organizations recognize the value of observability, but only 10% of them are actually practicing full observability of their applications and infrastructure. This is among the key findings from the recently completed Logz.io 2024 Observability Pulse Survey and Report ...

April 11, 2024

Businesses must adopt a comprehensive Internet Performance Monitoring (IPM) strategy, says Enterprise Management Associates (EMA), a leading IT analyst research firm. This strategy is crucial to bridge the significant observability gap within today's complex IT infrastructures. The recommendation is particularly timely, given that 99% of enterprises are expanding their use of the Internet as a primary connectivity conduit while facing challenges due to the inefficiency of multiple, disjointed monitoring tools, according to Modern Enterprises Must Boost Observability with Internet Performance Monitoring, a new report from EMA and Catchpoint ...

April 10, 2024

Choosing the right approach is critical with cloud monitoring in hybrid environments. Otherwise, you may drive up costs with features you don’t need and risk diminishing the visibility of your on-premises IT ...

April 09, 2024

Consumers ranked the marketing strategies and missteps that most significantly impact brand trust, which 73% say is their biggest motivator to share first-party data, according to The Rules of the Marketing Game, a 2023 report from Pantheon ...

April 08, 2024

Digital experience monitoring is the practice of monitoring and analyzing the complete digital user journey of your applications, websites, APIs, and other digital services. It involves tracking the performance of your web application from the perspective of the end user, providing detailed insights on user experience, app performance, and customer satisfaction ...

April 04, 2024
Modern organizations race to launch their high-quality cloud applications as soon as possible. On the other hand, time to market also plays an essential role in determining the application's success. However, without effective testing, it's hard to be confident in the final product ...
April 03, 2024

Enterprises are experiencing a 13% year-over-year increase in customer-facing incidents, reflecting rising levels of complexity and risk as businesses drive operational transformation at scale, according to the 2024 State of Digital Operations study from PagerDuty ...

April 02, 2024

According to Grafana Labs' 2024 Observability Survey, it doesn't matter what industry a company is in or the number of employees they have, the truth is: the more mature their observability practices are, the more time and money they save. From AI to OpenTelemetry — here are four key takeaways from this year's report ...

April 01, 2024

In an age where technology evolves at a breakneck pace, it's crucial to explore how AI assistants can revolutionize our work processes and daily lives, ultimately enhancing overall performance ...

March 28, 2024

Nearly all (99%) globa IT decision makers, regardless of region or industry, recognize generative AI's (GenAI) transformative potential to influence change within their organizations, according to The Elastic Generative AI Report ...