“Big Data” is everywhere. What does it mean? Just as Cloud Computing bursted onto the scene a few years ago, it depends on whom you ask.

Traditionally, in the Business Intelligence (BI) world, Big Data included analyzing historical business data from large data warehouses with the purpose of identifying long-term trends that could be leveraged in consumer business strategies. In recent years, Big Data has been a term talked about in the IT industry as an application of technology to attack extremely large, unstructured data sets that can reside both within and outside of an organization. If you look at a recent definition of Big Data, it is a term applied to data sets whose size has grown beyond the capability of commonly used software tools to capture, manage and analyze within a tolerable period of times for different use cases.

Application Performance Management (APM) is an extremely relevant use case and has a developing “Big Data” problem. Several factors are contributing to the explosive growth and type of data that must be analyzed and/or correlated in application performance monitoring and business service management (BSM).

First, the number of components that make up today’s mission critical applications has exploded. Instead of hundreds of servers for an application, nowadays, because of virtualization, you can easily be talking about thousands of virtual servers and objects for web applications.

Secondly, the diversity of data that people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply understand traditional IT infrastructure performance based on server operating system, network traffic, and storage capacity. Application Performance analysis is now based on the relationships of IT infrastructure components, application performance metrics from applications and application servers, business activity monitors (BAM) data, customer experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the aggregated transactional data, there are new systems that capture transactions’ actual path encompassing the entire application stack.

Finally, the requirements for analysis speed and data granularity have also increased significantly. Mission critical application performance now requires real-time or near real-time data analysis. When we were doing server availability and performance monitoring 10 years ago, it was the norm to collect and analyze data every 15 minutes.  Today, this has evolved to data analysis every 5 minutes or less with sub-minute data collection where all transaction paths are collected for data analysis. When mapped out, it's easy to see the enormous growth particularly when you look at APM related storage requirements that are quickly growing from gigabytes to terabytes and tomorrow petabytes.

Where APM and Big Data Meet the Cloud

All this data requires extremely complex analysis and correlation in order to truly understand performance of critical applications.

One of Netuitive’s large enterprise customers reported that it monitors and correlates a billion infrastructure and application data points and business metrics daily as part of its global service delivery. This is what I am referring to as APM-generated Big Data.

In addition to the shear number of data points, IT operators are expected to provide real-time analysis to the business and long-term storage for post-mortem analysis, capacity planning and compliance.

So where does this lead us? This is where APM and Big Data meet the Cloud. The cloud can deliver cheaper and more flexible storage and computing power crucial to analytics for Big Data. It also has the capability to be much more elastic for your APM data storage and analytics needs. Organizations can actually think about storing years of collected and aggregated APM data for compliance and analysis purposes without the cost being prohibitive.

But what does this mean to vendors in the APM space? 

First of all, the analytics platform for APM data has to evolve to be able to process the growing number of different data sources across business, customer experience, applications and IT domains. Netuitives’ “Open” analytics platform is engineered to address virtually any data source in real-time.

Secondly, data storage and access time will be critical even as APM data volumes continue to explode, so not only does the technology need to be able to run in the cloud, but the traditional pull-based data collection architecture has to evolve into a push based model with an horizontally scalable computing and storage architecture in order to become virtually limitless in terms of scalability. This is critical for larger organizations as “real” time no longer means analysis every 5 to 15 minutes, but sub-minute analytics.

Lastly, because storage and computing costs should not significantly exceed the cost of analytics software for a solution to be viable, Netuitive is advancing its product architecture to leverage NoSQL columnar data store as a replacement to traditional database.

While our R&D challenges are complex, the goal is simple: provide APM Analytics that matter by enabling our enterprise customers to process billions of infrastructure, application, and business metrics from hundreds of thousands of managed elements at 10x less cost than existing infrastructures.

ABOUT Jean-François Huard

Jean-François Huard is Chief Technical Officer and Vice President of Research and Development at Netuitive, Inc. In this role he is responsible for leading the company’s vision and technology innovation effort.

Previously, Huard was Chief Network Architect and Vice President of Network Engineering at InvisibleHand Networks, a start-up company funded by Polaris Venture Partners. Earlier, he led the technology team at Xbind, Inc. Earlier in his career, Huard worked in network fault management at AT&T Bell Labs, and was a member of the research staff at the Center for Telecommunications Research.

Jean-François contributed to the definition of the international MPEG-4 standard, and was chair and technical editor of the IEEE P1520.2 working group. He has authored or co-authored many scientific papers published in technical journals and conferences, standard contributions, and has filed multiple patents.