Always regarded as a non-critical part of day-to-day operations in the past, Big Data and its delayed analysis was relegated to batch processing tools and monthly meetings. Today, as the IT industry has snowballed into a fast moving avalanche of Cloud, virtualization, outsourcing and distributed computing, the science of extracting meaningful intelligent metrics from Big Data has become an important and real-time component of IT Operations.
Why Big Data in Cloud Performance Tools?
No longer do IT management systems work in vertical or horizontal isolation as just a few years ago. The inter-dependence between IT Services, applications, servers, cloud services and network infrastructure has a direct and measurable impact on Business Services.
The amount of data generated by these components is huge and the rate at which this data is generated is so fast that traditional tools cannot keep up with any kind of real time correlation. The combined volume of data generated by this hybrid infrastructure can be huge, but if it is correlated properly, it can give misson critical insight into:
- the response times and behavior of an IT service or application
- the cause of performance degradation of an IT service
- trend analysis and proactive capacity planning
- see if SLAs are being met for business services
This data has to be analyzed and processed in real-time in order to provide proactive responses and alerting for service degradation. The data that is being collected can be structured or unstructured, coming from a variety of systems which depend on each other to offer optimal performance, and has little to no obvious linkage or keys to one another (i.e. the data coming from an application is completely independent of the data coming from the network that it is running on).
Some examples of data sources that need to be correlated are application logs, netflow, JMX, XML, SNMP, WMI, security logs, packet analysis, business service response times, weather, news, etc.
Enterprises are moving to hybrid cloud environments at an alarming rate and all customer surveys indicate that the complexity of these platforms are their biggest concern. Enterprises must adopt monitoring systems that are flexible and can handle Big Data efficiently so that they can offer real-time responses to alarms and get meaningful business impact analysis from all of the different data sources.
Contextual analytics and presentation of data from multiple sources is invaluable to IT Operations in troubleshooting poor application performance and user satisfaction.
As a simple example, a user response time application could send an alert that the response time of an application is too high. Application Performance Monitoring (APM) data could indicate that a database is responding slowly to queries because the buffers are starved and the number of transactions is abnormally high. Integrating with network netflow or packet data would allow immediate drill down to isolate which client IP address is the source of the high number of queries.
How to Handle Big Data for Cloud Performance
Traditional monitoring or BI platforms are not designed to handle the volume and variety of data from this hybrid IT infrastructure. The management platforms need to be designed to correlate Big Data from the IT components in real-time and provide feedback to the operations team for proactive responses. As these monitoring systems evolve, their Big Data correlation components will become richer and more analytical and will position these enterprises for the IT environments of the future.
New generation enterprise monitoring solutions that are scalable, have predictive analytics, multi-tenant and a granular security model are now available from a small number of vendors. Single use systems that are designed for just network data or just application data are trapped within the same boundaries that makes Big Data meaningless - by its very nature, Big Data systems need to be able to handle a very wide variety of data sources to provide greater uptime from faster troubleshooting and lower OpEx from correlated analysis.
Vikas Aggarwal is CEO of Zyrion.
The Latest
When employees encounter tech friction or feel frustrated with the tools they are asked to use, they will find a workaround. In fact, one in two office workers admit to using personal devices to log into work networks, with 32% of them revealing their employers are unaware of this practice, according to Securing the Digital Employee Experience ...
In today's high-stakes race to deliver innovative products without disruptions, the importance of feature management and experimentation has never been more clear. But what strategies are driving success, and which tools are truly moving the needle? ...
The demand for real-time AI capabilities is pushing data scientists to develop and manage infrastructure that can handle massive volumes of data in motion. This includes streaming data pipelines, edge computing, scalable cloud architecture, and data quality and governance. These new responsibilities require data scientists to expand their skill sets significantly ...
As the digital landscape constantly evolves, it's critical for businesses to stay ahead, especially when it comes to operating systems updates. A recent ControlUp study revealed that 82% of enterprise Windows endpoint devices have yet to migrate to Windows 11. With Microsoft's cutoff date on October 14, 2025, for Windows 10 support fast approaching, the urgency cannot be overstated ...
In Part 1 of this two-part series, I defined multi-CDN and explored how and why this approach is used by streaming services, e-commerce platforms, gaming companies and global enterprises for fast and reliable content delivery ... Now, in Part 2 of the series, I'll explore one of the biggest challenges of multi-CDN: observability.
CDNs consist of geographically distributed data centers with servers that cache and serve content close to end users to reduce latency and improve load times. Each data center is strategically placed so that digital signals can rapidly travel from one "point of presence" to the next, getting the digital signal to the viewer as fast as possible ... Multi-CDN refers to the strategy of utilizing multiple CDNs to deliver digital content across the internet ...
We surveyed IT professionals on their attitudes and practices regarding using Generative AI with databases. We asked how they are layering the technology in with their systems, where it's working the best for them, and what their concerns are ...
40% of generative AI (GenAI) solutions will be multimodal (text, image, audio and video) by 2027, up from 1% in 2023, according to Gartner ...
Today's digital business landscape evolves rapidly ... Among the areas primed for innovation, the long-standing ticket-based IT support model stands out as particularly outdated. Emerging as a game-changer, the concept of the "ticketless enterprise" promises to shift IT management from a reactive stance to a proactive approach ...
