Modern IT and security organizations often need to manage petabytes of observability (logs, metrics, traces) data in real time. The adoption of cloud, modern application architectures, Kubernetes, and edge is behind this massive growth in observability data volumes. And for some organizations, log data volumes are approaching the exabyte range.
IT teams face many obstacles when managing massive amounts of observability data, from siloed tooling to prolonged incident resolution and security risks such as accidental exposure of personal identifiable information (PII) or credential data.
To shed light on key trends, challenges, and approaches these teams take to resolve those challenges, in Feb 2022, we ran a survey of professionals across various industries and roles within IT organizations. We are excited to share today the results of our 2022 State of Observability and Log Management report.
Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. These professionals are responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data, and their companies have at least 100 employees.
The survey results show that IT teams have difficulty with the massive growth of log data and use various methods to manage data volumes and their associated costs. These include only storing the most critical data to prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.
Two-thirds of IT organizations require engineering time to manage their log management tools; larger organizations with more log data are more likely to have dedicated teams for tool management.
For the purposes of our survey, we defined observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.
Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. In addition, as organizations mature in implementing observability, the value of critical insights from their log data is more significant.
Participants also shared details about their current streaming data use. Streaming data connects, filters, processes, and routes log data between different observability tools (commercial or open source) or offline cold storage (S3, Google Cloud Services, etc.) and is sometimes called observability pipeline or observability data management. According to responses, streaming observability pipelines adoption is a work in progress, with 20% of organizations reporting full deployments while 36% are evaluating or considering options.
Report findings also reveal:
■ Observability log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. In addition, 68% say log data is necessary, but it's tough to work with.
■ IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data or troubleshooting, and security analyses are impacted.
■ Existing log management tools present challenges and risks related to scalability, 97% of respondents report.
■ Log data is key to observability, and innovation is needed. 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don't evolve.
■ Problems are beyond storing data. For example, 96% report the need also to use the data to solve business problems.
■ 90% report reducing engineering toil to scale tools helps IT focus on more important work.
■ Volumes of log data in organizations are exploding, according to 96% of IT professionals surveyed.
Survey Demographics: Roles include a third IT executives, a third enterprise (cloud or application) architects, and a third in DevOps/SRE/Ops roles. Companies are in the following regions: AMER (77%), followed by EMEA (20%) and APAC (3%), and include a variety of industry verticals, including financial, technology, healthcare, service, retail, manufacturing, etc.
The Latest
Site reliability engineering (SRE) is fast becoming an essential aspect of modern IT operations, particularly in highly scaled, big data environments. As businesses and industries shift to the digital and embrace new IT infrastructures and technologies to remain operational and competitive, the need for a new approach for IT teams to find and manage the balance between launching new systems and features and ensuring these are intuitive, reliable, and friendly for end users has intensified as well ...
The most sophisticated observability practitioners (leaders) are able to cut downtime costs by 90%, from an estimated $23.8 million annually to just $2.5 million, compared to observability beginners, according to the State of Observability 2022 from Splunk in collaboration with the Enterprise Strategy Group. What's more, leaders in observability are more innovative and more successful at achieving digital transformation outcomes and other initiatives ...
Programmatically tracked service level indicators (SLIs) are foundational to every site reliability engineering practice. When engineering teams have programmatic SLIs in place, they lessen the need to manually track performance and incident data. They're also able to reduce manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically — hence "programmatic" ...
Recently, a regional healthcare organization wanted to retire its legacy monitoring tools and adopt AIOps. The organization asked Windward Consulting to implement an AIOps strategy that would help streamline its outdated and unwieldy IT system management. Our team's AIOps implementation process helped this client and can help others in the industry too. Here's what my team did ...
You've likely heard it before: every business is a digital business. However, some businesses and sectors digitize more quickly than others. Healthcare has traditionally been on the slower side of digital transformation and technology adoption, but that's changing. As healthcare organizations roll out innovations at increasing velocity, they must build a long-term strategy for how they will maintain the uptime of their critical apps and services. And there's only one tool that can ensure this continuous availability in our modern IT ecosystems. AIOps can help IT Operations teams ensure the uptime of critical apps and services ...
Between 2012 to 2015 all of the hyperscalers attempted to use the legacy APM solutions to improve their own visibility. To no avail. The problem was that none of the previous generations of APM solutions could match the scaling demand, nor could they provide interoperability due to their proprietary and exclusive agentry ...
The DevOps journey begins by understanding a team's DevOps flow and identifying precisely what tasks deliver the best return on engineers' time when automated. The rest of this blog will help DevOps team managers by outlining what jobs can — and should be automated ...
A survey from Snow Software polled more than 500 IT leaders to determine the current state of cloud infrastructure. Nearly half of the IT leaders who responded agreed that cloud was critical to operations during the pandemic with the majority deploying a hybrid cloud strategy consisting of both public and private clouds. Unsurprisingly, over the last 12 months, the majority of respondents had increased overall cloud spend — a substantial increase over the 2020 findings ...
As we all know, the drastic changes in the world have caused the workforce to take a hybrid approach over the last two years. A lot of that time, being fully remote. With the back and forth between home and office, employees need ways to stay productive and access useful information necessary to complete their daily work. The ability to obtain a holistic view of data relevant to the user and get answers to topics, no matter the worker's location, is crucial for a successful and efficient hybrid working environment ...
For the past decade, Application Performance Management has been a capability provided by a very small and exclusive set of vendors. These vendors provided a bolt-on solution that provided monitoring capabilities without requiring developers to take ownership of instrumentation and monitoring. You may think of this as a benefit, but in reality, it was not ...
