Big Data in Application and Cloud Performance - Why and How
May 01, 2013

Vikas Aggarwal

Share this

Always regarded as a non-critical part of day-to-day operations in the past, Big Data and its delayed analysis was relegated to batch processing tools and monthly meetings. Today, as the IT industry has snowballed into a fast moving avalanche of Cloud, virtualization, outsourcing and distributed computing, the science of extracting meaningful intelligent metrics from Big Data has become an important and real-time component of IT Operations.

Why Big Data in Cloud Performance Tools?

No longer do IT management systems work in vertical or horizontal isolation as just a few years ago. The inter-dependence between IT Services, applications, servers, cloud services and network infrastructure has a direct and measurable impact on Business Services.

The amount of data generated by these components is huge and the rate at which this data is generated is so fast that traditional tools cannot keep up with any kind of real time correlation. The combined volume of data generated by this hybrid infrastructure can be huge, but if it is correlated properly, it can give misson critical insight into:

- the response times and behavior of an IT service or application

- the cause of performance degradation of an IT service

- trend analysis and proactive capacity planning

- see if SLAs are being met for business services

This data has to be analyzed and processed in real-time in order to provide proactive responses and alerting for service degradation. The data that is being collected can be structured or unstructured, coming from a variety of systems which depend on each other to offer optimal performance, and has little to no obvious linkage or keys to one another (i.e. the data coming from an application is completely independent of the data coming from the network that it is running on).

Some examples of data sources that need to be correlated are application logs, netflow, JMX, XML, SNMP, WMI, security logs, packet analysis, business service response times, weather, news, etc.

Enterprises are moving to hybrid cloud environments at an alarming rate and all customer surveys indicate that the complexity of these platforms are their biggest concern. Enterprises must adopt monitoring systems that are flexible and can handle Big Data efficiently so that they can offer real-time responses to alarms and get meaningful business impact analysis from all of the different data sources.

Contextual analytics and presentation of data from multiple sources is invaluable to IT Operations in troubleshooting poor application performance and user satisfaction.

As a simple example, a user response time application could send an alert that the response time of an application is too high. Application Performance Monitoring (APM) data could indicate that a database is responding slowly to queries because the buffers are starved and the number of transactions is abnormally high. Integrating with network netflow or packet data would allow immediate drill down to isolate which client IP address is the source of the high number of queries.

How to Handle Big Data for Cloud Performance

Traditional monitoring or BI platforms are not designed to handle the volume and variety of data from this hybrid IT infrastructure. The management platforms need to be designed to correlate Big Data from the IT components in real-time and provide feedback to the operations team for proactive responses. As these monitoring systems evolve, their Big Data correlation components will become richer and more analytical and will position these enterprises for the IT environments of the future.

New generation enterprise monitoring solutions that are scalable, have predictive analytics, multi-tenant and a granular security model are now available from a small number of vendors. Single use systems that are designed for just network data or just application data are trapped within the same boundaries that makes Big Data meaningless - by its very nature, Big Data systems need to be able to handle a very wide variety of data sources to provide greater uptime from faster troubleshooting and lower OpEx from correlated analysis.

Vikas Aggarwal is CEO of Zyrion.

Share this

The Latest

June 13, 2019

Establishing a digital business is top-of-mind, even more so than last year, as 91% of organizations have adopted or have plans to adopt a digital-first strategy, according to IDG Communications Digital Business Research ...

June 12, 2019

If digital transformation is to succeed at the pace enterprises demand, IT teams, the CIOs who lead them, and the boardroom must forge a far greater alignment than presently exists. That is the over-arching sentiment expressed by IT professionals in a recent survey on the state of IT infrastructure and roadblocks to digital success ...

June 11, 2019

Given the incredible amount of traffic traversing corporate WANs, it's not surprising that businesses are seeing performance issues. If anything, it's amazing applications work as well as they do ...

June 10, 2019

Are your business applications sluggish? Choppy? Prone to getting hung up or crashing at the most inopportune times? If these symptoms sound familiar, you might be suffering from the heartache of … poor application performance. Stop me if any of this sounds familiar ...

June 06, 2019
AIOps Exchange, a not-for-profit private forum defining the future of AIOps, published <span style="font-style: italic;">The AIOps Manifesto</span> discussing the role of AI in supporting digital transformation ...
June 05, 2019

As network transformation initiatives like SD-WAN, edge computing and public/private clouds are adopted at increasing rates, hybrid networks are quickly becoming the new normal for IT and NetOps professionals.Without visibility into these hybrid network environments, NetOps are unable to troubleshoot the business-critical applications every organization relies on today. Here are four ways IT and NetOps teams can gain better visibility into complex, hybrid networks ...

June 04, 2019

A minimum Internet Performance Bar exists that, if met, should deliver top-tier website performance, regardless of industry, according to the 2019 Digital Experience Performance Benchmark Report, from ThousandEyes, a comparative analysis of web, infrastructure and network performance metrics from the top 20 US digital retail, travel and media websites ...

June 03, 2019

Since digital transformation is happening at such a rapid pace based on new, highly complex technologies like multi-cloud, containers and microservice architectures, customers are experiencing more challenges than ever in managing this complexity. However, with every challenge comes an opportunity. So, how can channel partners leverage these market disruptions to open the door to opportunity? The answer is simple ...

May 30, 2019

Executives from proactive organizations reported using performance management strategies to deliver innovation and meet broader business goals, and implementing application performance management (APM) tools with advanced monitoring features such as real-time user experience monitoring, and providing a composite view of log and performance data, according to Driving Business Performance Through Application Performance Management, a new report from GigaOm ...

May 29, 2019

Through our recent study, we wanted to better understand how service desk users are interacting with the service teams; how they connect for service; the manner in which most service desks receive user requests; and if organizations employ a knowledge base and how that information might be stored. Here’s what we’ve discovered ...