Skip to main content

90% Report Reducing Engineering Toil to Scale Tools Puts Focus on Business Bottom Line

Stela Udovicic
Era Software

Modern IT and security organizations often need to manage petabytes of observability (logs, metrics, traces) data in real time. The adoption of cloud, modern application architectures, Kubernetes, and edge is behind this massive growth in observability data volumes. And for some organizations, log data volumes are approaching the exabyte range.

IT teams face many obstacles when managing massive amounts of observability data, from siloed tooling to prolonged incident resolution and security risks such as accidental exposure of personal identifiable information (PII) or credential data.

To shed light on key trends, challenges, and approaches these teams take to resolve those challenges, in Feb 2022, we ran a survey of professionals across various industries and roles within IT organizations. We are excited to share today the results of our 2022 State of Observability and Log Management report.

Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. These professionals are responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data, and their companies have at least 100 employees.

The survey results show that IT teams have difficulty with the massive growth of log data and use various methods to manage data volumes and their associated costs. These include only storing the most critical data to prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.

Two-thirds of IT organizations require engineering time to manage their log management tools; larger organizations with more log data are more likely to have dedicated teams for tool management.
For the purposes of our survey, we defined observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.

Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. In addition, as organizations mature in implementing observability, the value of critical insights from their log data is more significant.

Participants also shared details about their current streaming data use. Streaming data connects, filters, processes, and routes log data between different observability tools (commercial or open source) or offline cold storage (S3, Google Cloud Services, etc.) and is sometimes called observability pipeline or observability data management. According to responses, streaming observability pipelines adoption is a work in progress, with 20% of organizations reporting full deployments while 36% are evaluating or considering options.

Report findings also reveal:

■ Observability log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. In addition, 68% say log data is necessary, but it's tough to work with.

■ IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data or troubleshooting, and security analyses are impacted.
■ Existing log management tools present challenges and risks related to scalability, 97% of respondents report.

■ Log data is key to observability, and innovation is needed. 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don't evolve.

■ Problems are beyond storing data. For example, 96% report the need also to use the data to solve business problems.

■ 90% report reducing engineering toil to scale tools helps IT focus on more important work.

■ Volumes of log data in organizations are exploding, according to 96% of IT professionals surveyed.

Survey Demographics: Roles include a third IT executives, a third enterprise (cloud or application) architects, and a third in DevOps/SRE/Ops roles. Companies are in the following regions: AMER (77%), followed by EMEA (20%) and APAC (3%), and include a variety of industry verticals, including financial, technology, healthcare, service, retail, manufacturing, etc.

Stela Udovicic is SVP, Marketing, at Era Software

Hot Topics

The Latest

Application performance monitoring (APM) is a game of catching up — building dashboards, setting thresholds, tuning alerts, and manually correlating metrics to root causes. In the early days, this straightforward model worked as applications were simpler, stacks more predictable, and telemetry was manageable. Today, the landscape has shifted, and more assertive tools are needed ...

Cloud adoption has accelerated, but backup strategies haven't always kept pace. Many organizations continue to rely on backup strategies that were either lifted directly from on-prem environments or use cloud-native tools in limited, DR-focused ways ... Eon uncovered a handful of critical gaps regarding how organizations approach cloud backup. To capture these prevailing winds, we gathered insights from 150+ IT and cloud leaders at the recent Google Cloud Next conference, which we've compiled into the 2025 State of Cloud Data Backup ...

Private clouds are no longer playing catch-up, and public clouds are no longer the default as organizations recalibrate their cloud strategies, according to the Private Cloud Outlook 2025 report from Broadcom. More than half (53%) of survey respondents say private cloud is their top priority for deploying new workloads over the next three years, while 69% are considering workload repatriation from public to private cloud, with one-third having already done so ...

As organizations chase productivity gains from generative AI, teams are overwhelmingly focused on improving delivery speed (45%) over enhancing software quality (13%), according to the Quality Transformation Report from Tricentis ...

Back in March of this year ... MongoDB's stock price took a serious tumble ... In my opinion, it reflects a deeper structural issue in enterprise software economics altogether — vendor lock-in ...

In MEAN TIME TO INSIGHT Episode 15, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA discusses Do-It-Yourself Network Automation ... 

Zero-day vulnerabilities — security flaws that are exploited before developers even know they exist — pose one of the greatest risks to modern organizations. Recently, such vulnerabilities have been discovered in well-known VPN systems like Ivanti and Fortinet, highlighting just how outdated these legacy technologies have become in defending against fast-evolving cyber threats ... To protect digital assets and remote workers in today's environment, companies need more than patchwork solutions. They need architecture that is secure by design ...

Traditional observability requires users to leap across different platforms or tools for metrics, logs, or traces and related issues manually, which is very time-consuming, so as to reasonably ascertain the root cause. Observability 2.0 fixes this by unifying all telemetry data, logs, metrics, and traces into a single, context-rich pipeline that flows into one smart platform. But this is far from just having a bunch of additional data; this data is actionable, predictive, and tied to revenue realization ...

64% of enterprise networking teams use internally developed software or scripts for network automation, but 61% of those teams spend six or more hours per week debugging and maintaining them, according to From Scripts to Platforms: Why Homegrown Tools Dominate Network Automation and How Vendors Can Help, my latest EMA report ...

Cloud computing has transformed how we build and scale software, but it has also quietly introduced one of the most persistent challenges in modern IT: cost visibility and control ... So why, after more than a decade of cloud adoption, are cloud costs still spiraling out of control? The answer lies not in tooling but in culture ...

90% Report Reducing Engineering Toil to Scale Tools Puts Focus on Business Bottom Line

Stela Udovicic
Era Software

Modern IT and security organizations often need to manage petabytes of observability (logs, metrics, traces) data in real time. The adoption of cloud, modern application architectures, Kubernetes, and edge is behind this massive growth in observability data volumes. And for some organizations, log data volumes are approaching the exabyte range.

IT teams face many obstacles when managing massive amounts of observability data, from siloed tooling to prolonged incident resolution and security risks such as accidental exposure of personal identifiable information (PII) or credential data.

To shed light on key trends, challenges, and approaches these teams take to resolve those challenges, in Feb 2022, we ran a survey of professionals across various industries and roles within IT organizations. We are excited to share today the results of our 2022 State of Observability and Log Management report.

Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. These professionals are responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data, and their companies have at least 100 employees.

The survey results show that IT teams have difficulty with the massive growth of log data and use various methods to manage data volumes and their associated costs. These include only storing the most critical data to prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.

Two-thirds of IT organizations require engineering time to manage their log management tools; larger organizations with more log data are more likely to have dedicated teams for tool management.
For the purposes of our survey, we defined observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.

Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. In addition, as organizations mature in implementing observability, the value of critical insights from their log data is more significant.

Participants also shared details about their current streaming data use. Streaming data connects, filters, processes, and routes log data between different observability tools (commercial or open source) or offline cold storage (S3, Google Cloud Services, etc.) and is sometimes called observability pipeline or observability data management. According to responses, streaming observability pipelines adoption is a work in progress, with 20% of organizations reporting full deployments while 36% are evaluating or considering options.

Report findings also reveal:

■ Observability log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. In addition, 68% say log data is necessary, but it's tough to work with.

■ IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data or troubleshooting, and security analyses are impacted.
■ Existing log management tools present challenges and risks related to scalability, 97% of respondents report.

■ Log data is key to observability, and innovation is needed. 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don't evolve.

■ Problems are beyond storing data. For example, 96% report the need also to use the data to solve business problems.

■ 90% report reducing engineering toil to scale tools helps IT focus on more important work.

■ Volumes of log data in organizations are exploding, according to 96% of IT professionals surveyed.

Survey Demographics: Roles include a third IT executives, a third enterprise (cloud or application) architects, and a third in DevOps/SRE/Ops roles. Companies are in the following regions: AMER (77%), followed by EMEA (20%) and APAC (3%), and include a variety of industry verticals, including financial, technology, healthcare, service, retail, manufacturing, etc.

Stela Udovicic is SVP, Marketing, at Era Software

Hot Topics

The Latest

Application performance monitoring (APM) is a game of catching up — building dashboards, setting thresholds, tuning alerts, and manually correlating metrics to root causes. In the early days, this straightforward model worked as applications were simpler, stacks more predictable, and telemetry was manageable. Today, the landscape has shifted, and more assertive tools are needed ...

Cloud adoption has accelerated, but backup strategies haven't always kept pace. Many organizations continue to rely on backup strategies that were either lifted directly from on-prem environments or use cloud-native tools in limited, DR-focused ways ... Eon uncovered a handful of critical gaps regarding how organizations approach cloud backup. To capture these prevailing winds, we gathered insights from 150+ IT and cloud leaders at the recent Google Cloud Next conference, which we've compiled into the 2025 State of Cloud Data Backup ...

Private clouds are no longer playing catch-up, and public clouds are no longer the default as organizations recalibrate their cloud strategies, according to the Private Cloud Outlook 2025 report from Broadcom. More than half (53%) of survey respondents say private cloud is their top priority for deploying new workloads over the next three years, while 69% are considering workload repatriation from public to private cloud, with one-third having already done so ...

As organizations chase productivity gains from generative AI, teams are overwhelmingly focused on improving delivery speed (45%) over enhancing software quality (13%), according to the Quality Transformation Report from Tricentis ...

Back in March of this year ... MongoDB's stock price took a serious tumble ... In my opinion, it reflects a deeper structural issue in enterprise software economics altogether — vendor lock-in ...

In MEAN TIME TO INSIGHT Episode 15, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA discusses Do-It-Yourself Network Automation ... 

Zero-day vulnerabilities — security flaws that are exploited before developers even know they exist — pose one of the greatest risks to modern organizations. Recently, such vulnerabilities have been discovered in well-known VPN systems like Ivanti and Fortinet, highlighting just how outdated these legacy technologies have become in defending against fast-evolving cyber threats ... To protect digital assets and remote workers in today's environment, companies need more than patchwork solutions. They need architecture that is secure by design ...

Traditional observability requires users to leap across different platforms or tools for metrics, logs, or traces and related issues manually, which is very time-consuming, so as to reasonably ascertain the root cause. Observability 2.0 fixes this by unifying all telemetry data, logs, metrics, and traces into a single, context-rich pipeline that flows into one smart platform. But this is far from just having a bunch of additional data; this data is actionable, predictive, and tied to revenue realization ...

64% of enterprise networking teams use internally developed software or scripts for network automation, but 61% of those teams spend six or more hours per week debugging and maintaining them, according to From Scripts to Platforms: Why Homegrown Tools Dominate Network Automation and How Vendors Can Help, my latest EMA report ...

Cloud computing has transformed how we build and scale software, but it has also quietly introduced one of the most persistent challenges in modern IT: cost visibility and control ... So why, after more than a decade of cloud adoption, are cloud costs still spiraling out of control? The answer lies not in tooling but in culture ...