90% Report Reducing Engineering Toil to Scale Tools Puts Focus on Business Bottom Line
March 17, 2022

Stela Udovicic
Era Software

Share this

Modern IT and security organizations often need to manage petabytes of observability (logs, metrics, traces) data in real time. The adoption of cloud, modern application architectures, Kubernetes, and edge is behind this massive growth in observability data volumes. And for some organizations, log data volumes are approaching the exabyte range.

IT teams face many obstacles when managing massive amounts of observability data, from siloed tooling to prolonged incident resolution and security risks such as accidental exposure of personal identifiable information (PII) or credential data.

To shed light on key trends, challenges, and approaches these teams take to resolve those challenges, in Feb 2022, we ran a survey of professionals across various industries and roles within IT organizations. We are excited to share today the results of our 2022 State of Observability and Log Management report.

Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. These professionals are responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data, and their companies have at least 100 employees.

The survey results show that IT teams have difficulty with the massive growth of log data and use various methods to manage data volumes and their associated costs. These include only storing the most critical data to prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.

Two-thirds of IT organizations require engineering time to manage their log management tools; larger organizations with more log data are more likely to have dedicated teams for tool management.
For the purposes of our survey, we defined observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.

Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. In addition, as organizations mature in implementing observability, the value of critical insights from their log data is more significant.

Participants also shared details about their current streaming data use. Streaming data connects, filters, processes, and routes log data between different observability tools (commercial or open source) or offline cold storage (S3, Google Cloud Services, etc.) and is sometimes called observability pipeline or observability data management. According to responses, streaming observability pipelines adoption is a work in progress, with 20% of organizations reporting full deployments while 36% are evaluating or considering options.

Report findings also reveal:

■ Observability log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. In addition, 68% say log data is necessary, but it's tough to work with.

■ IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data or troubleshooting, and security analyses are impacted.
■ Existing log management tools present challenges and risks related to scalability, 97% of respondents report.

■ Log data is key to observability, and innovation is needed. 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don't evolve.

■ Problems are beyond storing data. For example, 96% report the need also to use the data to solve business problems.

■ 90% report reducing engineering toil to scale tools helps IT focus on more important work.

■ Volumes of log data in organizations are exploding, according to 96% of IT professionals surveyed.

Survey Demographics: Roles include a third IT executives, a third enterprise (cloud or application) architects, and a third in DevOps/SRE/Ops roles. Companies are in the following regions: AMER (77%), followed by EMEA (20%) and APAC (3%), and include a variety of industry verticals, including financial, technology, healthcare, service, retail, manufacturing, etc.

Stela Udovicic is SVP, Marketing, at Era Software
Share this

The Latest

March 27, 2023

To achieve maximum availability, IT leaders must employ domain-agnostic solutions that identify and escalate issues across all telemetry points. These technologies, which we refer to as Artificial Intelligence for IT Operations, create convergence — in other words, they provide IT and DevOps teams with the full picture of event management and downtime ...

March 23, 2023

APMdigest and leading IT research firm Enterprise Management Associates (EMA) are partnering to bring you the EMA-APMdigest Podcast, a new podcast focused on the latest technologies impacting IT Operations. In Episode 2 - Part 1 Pete Goldin, Editor and Publisher of APMdigest, discusses Network Observability with Shamus McGillicuddy, Vice President of Research, Network Infrastructure and Operations, at EMA ...

March 22, 2023

CIOs have stepped into the role of digital leader and strategic advisor, according to the 2023 Global CIO Survey from Logicalis ...

March 21, 2023

Synthetic monitoring is crucial to deploy code with confidence as catching bugs with E2E tests on staging is becoming increasingly difficult. It isn't trivial to provide realistic staging systems, especially because today's apps are intertwined with many third-party APIs ...

March 20, 2023

Recent EMA field research found that ServiceOps is either an active effort or a formal initiative in 78% of the organizations represented by a global panel of 400+ IT leaders. It is relatively early but gaining momentum across industries and organizations of all sizes globally ...

March 16, 2023

Managing availability and performance within SAP environments has long been a challenge for IT teams. But as IT environments grow more complex and dynamic, and the speed of innovation in almost every industry continues to accelerate, this situation is becoming a whole lot worse ...

March 15, 2023

Harnessing the power of network-derived intelligence and insights is critical in detecting today's increasingly sophisticated security threats across hybrid and multi-cloud infrastructure, according to a new research study from IDC ...

March 14, 2023

Recent research suggests that many organizations are paying for more software than they need. If organizations are looking to reduce IT spend, leaders should take a closer look at the tools being offered to employees, as not all software is essential ...

March 13, 2023

Organizations are challenged by tool sprawl and data source overload, according to the Grafana Labs Observability Survey 2023, with 52% of respondents reporting that their companies use 6 or more observability tools, including 11% that use 16 or more.

March 09, 2023

An array of tools purport to maintain availability — the trick is sorting through the noise to find the right one. Let us discuss why availability is so important and then unpack the ROI of deploying Artificial Intelligence for IT Operations (AIOps) during an economic downturn ...