Always regarded as a non-critical part of day-to-day operations in the past, Big Data and its delayed analysis was relegated to batch processing tools and monthly meetings. Today, as the IT industry has snowballed into a fast moving avalanche of Cloud, virtualization, outsourcing and distributed computing, the science of extracting meaningful intelligent metrics from Big Data has become an important and real-time component of IT Operations.
Why Big Data in Cloud Performance Tools?
No longer do IT management systems work in vertical or horizontal isolation as just a few years ago. The inter-dependence between IT Services, applications, servers, cloud services and network infrastructure has a direct and measurable impact on Business Services.
The amount of data generated by these components is huge and the rate at which this data is generated is so fast that traditional tools cannot keep up with any kind of real time correlation. The combined volume of data generated by this hybrid infrastructure can be huge, but if it is correlated properly, it can give misson critical insight into:
- the response times and behavior of an IT service or application
- the cause of performance degradation of an IT service
- trend analysis and proactive capacity planning
- see if SLAs are being met for business services
This data has to be analyzed and processed in real-time in order to provide proactive responses and alerting for service degradation. The data that is being collected can be structured or unstructured, coming from a variety of systems which depend on each other to offer optimal performance, and has little to no obvious linkage or keys to one another (i.e. the data coming from an application is completely independent of the data coming from the network that it is running on).
Some examples of data sources that need to be correlated are application logs, netflow, JMX, XML, SNMP, WMI, security logs, packet analysis, business service response times, weather, news, etc.
Enterprises are moving to hybrid cloud environments at an alarming rate and all customer surveys indicate that the complexity of these platforms are their biggest concern. Enterprises must adopt monitoring systems that are flexible and can handle Big Data efficiently so that they can offer real-time responses to alarms and get meaningful business impact analysis from all of the different data sources.
Contextual analytics and presentation of data from multiple sources is invaluable to IT Operations in troubleshooting poor application performance and user satisfaction.
As a simple example, a user response time application could send an alert that the response time of an application is too high. Application Performance Monitoring (APM) data could indicate that a database is responding slowly to queries because the buffers are starved and the number of transactions is abnormally high. Integrating with network netflow or packet data would allow immediate drill down to isolate which client IP address is the source of the high number of queries.
How to Handle Big Data for Cloud Performance
Traditional monitoring or BI platforms are not designed to handle the volume and variety of data from this hybrid IT infrastructure. The management platforms need to be designed to correlate Big Data from the IT components in real-time and provide feedback to the operations team for proactive responses. As these monitoring systems evolve, their Big Data correlation components will become richer and more analytical and will position these enterprises for the IT environments of the future.
New generation enterprise monitoring solutions that are scalable, have predictive analytics, multi-tenant and a granular security model are now available from a small number of vendors. Single use systems that are designed for just network data or just application data are trapped within the same boundaries that makes Big Data meaningless - by its very nature, Big Data systems need to be able to handle a very wide variety of data sources to provide greater uptime from faster troubleshooting and lower OpEx from correlated analysis.
Vikas Aggarwal is CEO of Zyrion.
The Latest
A long-running study of DevOps practices ... suggests that any historical gains in MTTR reduction have now plateaued. For years now, the time it takes to restore services has stayed about the same: less than a day for high performers but up to a week for middle-tier teams and up to a month for laggards. The fact that progress is flat despite big investments in people, tools and automation is a cause for concern ...
Companies implementing observability benefit from increased operational efficiency, faster innovation, and better business outcomes overall, according to 2023 IT Trends Report: Lessons From Observability Leaders, a report from SolarWinds ...
IT leaders are driving an increasing number of automation initiatives as a way to stay competitive, reduce costs and scale as they navigate an unpredictable social and economic environment, according to the 2023 State of Automation in IT survey conducted by Jitterbit ...
Customer loyalty is changing as retailers get increasingly competitive. More than 75% of consumers say they would end business with a company after a single bad customer experience. This means that just one price discrepancy, inventory mishap or checkout issue in a physical or digital store, could have customers running out to the next store that can provide them with better service. Retailers must be able to predict business outages in advance, and act proactively before an incident occurs, impacting customer experience ...
Earlier this year, New Relic conducted a study on observability ... The 2023 Observability Forecast reveals observability's impact on the lives of technical professionals and businesses' bottom lines. Here are 10 key takeaways from the forecast ...
Only 33% of executives are "very confident" in their ability to operate in a public cloud environment, according to the 2023 State of CloudOps report from NetApp. This represents an increase from 2022 when only 21% reported feeling very confident ...
The majority of organizations across Australia and New Zealand (A/NZ) breached over the last year had personally identifiable information (PII) compromised, but most have not yet modified their data management policies, according to the Cybersecurity and PII Report from ManageEngine ...
A large majority of organizations employ more than one cloud automation solution, and this practice creates significant challenges that are resulting in delays and added costs for businesses, according to Why companies lose efficiency and compliance with cloud automation solutions from Broadcom ...
