90% Report Reducing Engineering Toil to Scale Tools Puts Focus on Business Bottom Line
March 17, 2022

Stela Udovicic
Era Software

Share this

Modern IT and security organizations often need to manage petabytes of observability (logs, metrics, traces) data in real time. The adoption of cloud, modern application architectures, Kubernetes, and edge is behind this massive growth in observability data volumes. And for some organizations, log data volumes are approaching the exabyte range.

IT teams face many obstacles when managing massive amounts of observability data, from siloed tooling to prolonged incident resolution and security risks such as accidental exposure of personal identifiable information (PII) or credential data.

To shed light on key trends, challenges, and approaches these teams take to resolve those challenges, in Feb 2022, we ran a survey of professionals across various industries and roles within IT organizations. We are excited to share today the results of our 2022 State of Observability and Log Management report.

Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. These professionals are responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data, and their companies have at least 100 employees.

The survey results show that IT teams have difficulty with the massive growth of log data and use various methods to manage data volumes and their associated costs. These include only storing the most critical data to prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.

Two-thirds of IT organizations require engineering time to manage their log management tools; larger organizations with more log data are more likely to have dedicated teams for tool management.
For the purposes of our survey, we defined observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.

Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. In addition, as organizations mature in implementing observability, the value of critical insights from their log data is more significant.

Participants also shared details about their current streaming data use. Streaming data connects, filters, processes, and routes log data between different observability tools (commercial or open source) or offline cold storage (S3, Google Cloud Services, etc.) and is sometimes called observability pipeline or observability data management. According to responses, streaming observability pipelines adoption is a work in progress, with 20% of organizations reporting full deployments while 36% are evaluating or considering options.

Report findings also reveal:

■ Observability log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. In addition, 68% say log data is necessary, but it's tough to work with.

■ IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data or troubleshooting, and security analyses are impacted.
■ Existing log management tools present challenges and risks related to scalability, 97% of respondents report.

■ Log data is key to observability, and innovation is needed. 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don't evolve.

■ Problems are beyond storing data. For example, 96% report the need also to use the data to solve business problems.

■ 90% report reducing engineering toil to scale tools helps IT focus on more important work.

■ Volumes of log data in organizations are exploding, according to 96% of IT professionals surveyed.

Survey Demographics: Roles include a third IT executives, a third enterprise (cloud or application) architects, and a third in DevOps/SRE/Ops roles. Companies are in the following regions: AMER (77%), followed by EMEA (20%) and APAC (3%), and include a variety of industry verticals, including financial, technology, healthcare, service, retail, manufacturing, etc.

Stela Udovicic is SVP, Marketing, at Era Software
Share this

The Latest

March 04, 2024

This year's Super Bowl drew in viewership of nearly 124 million viewers and made history as the most-watched live broadcast event since the 1969 moon landing. To support this spike in viewership, streaming companies like YouTube TV, Hulu and Paramount+ began preparing their IT infrastructure months in advance to ensure an exceptional viewer experience without outages or major interruptions. New Relic conducted a survey to understand the importance of a seamless viewing experience and the impact of outages during major streaming events such as the Super Bowl ...

March 01, 2024

As organizations continue to navigate the complexities of the digital era, which has been marked by exponential advancements in AI and technology, the strategic deployment of modern, practical applications has become indispensable for sustaining competitive advantage and realizing business goals. The Info-Tech Research Group report, Applications Priorities 2024, explores the following five initiatives for emerging and leading-edge technologies and practices that can enable IT and applications leaders to optimize their application portfolio and improve on capabilities needed to meet the ambitions of their organizations ...

February 29, 2024

Despite the growth in popularity of artificial intelligence (AI) and ML across a number of industries, there is still a huge amount of unrealized potential, with many businesses playing catch-up and still planning how ML solutions can best facilitate processes. Further progression could be limited without investment in specialized technical teams to drive development and integration ...

February 28, 2024

With over 200 streaming services to choose from, including multiple platforms featuring similar types of entertainment, users have little incentive to remain loyal to any given platform if it exhibits performance issues. Big names in streaming like Hulu, Amazon Prime and HBO Max invest thousands of hours into engineering observability and closed-loop monitoring to combat infrastructure and application issues, but smaller platforms struggle to remain competitive without access to the same resources ...

February 27, 2024

Generative AI has recently experienced unprecedented dramatic growth, making it one of the most exciting transformations the tech industry has seen in some time. However, this growth also poses a challenge for tech leaders who will be expected to deliver on the promise of new technology. In 2024, delivering tangible outcomes that meet the potential of AI, and setting up incubator projects for the future will be key tasks ...

February 26, 2024

SAP is a tool for automating business processes. Managing SAP solutions, especially with the shift to the cloud-based S/4HANA platform, can be intricate. To explore the concerns of SAP users during operational transformations and automation, a survey was conducted in mid-2023 by Digitate and Americas' SAP Users' Group ...

February 22, 2024

Some companies are just starting to dip their toes into developing AI capabilities, while (few) others can claim they have built a truly AI-first product. Regardless of where a company is on the AI journey, leaders must understand what it means to build every aspect of their product with AI in mind ...

February 21, 2024

Generative AI will usher in advantages within various industries. However, the technology is still nascent, and according to the recent Dynatrace survey there are many challenges and risks that organizations need to overcome to use this technology effectively ...

February 20, 2024

In today's digital era, monitoring and observability are indispensable in software and application development. Their efficacy lies in empowering developers to swiftly identify and address issues, enhance performance, and deliver flawless user experiences. Achieving these objectives requires meticulous planning, strategic implementation, and consistent ongoing maintenance. In this blog, we're sharing our five best practices to fortify your approach to application performance monitoring (APM) and observability ...

February 16, 2024

In MEAN TIME TO INSIGHT Episode 3, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses network security with Chris Steffen, VP of Research Covering Information Security, Risk, and Compliance Management at EMA ...