Modern IT and security organizations often need to manage petabytes of observability (logs, metrics, traces) data in real time. The adoption of cloud, modern application architectures, Kubernetes, and edge is behind this massive growth in observability data volumes. And for some organizations, log data volumes are approaching the exabyte range.
IT teams face many obstacles when managing massive amounts of observability data, from siloed tooling to prolonged incident resolution and security risks such as accidental exposure of personal identifiable information (PII) or credential data.
To shed light on key trends, challenges, and approaches these teams take to resolve those challenges, in Feb 2022, we ran a survey of professionals across various industries and roles within IT organizations. We are excited to share today the results of our 2022 State of Observability and Log Management report.
Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. These professionals are responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data, and their companies have at least 100 employees.
The survey results show that IT teams have difficulty with the massive growth of log data and use various methods to manage data volumes and their associated costs. These include only storing the most critical data to prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.
Two-thirds of IT organizations require engineering time to manage their log management tools; larger organizations with more log data are more likely to have dedicated teams for tool management.
For the purposes of our survey, we defined observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.
Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. In addition, as organizations mature in implementing observability, the value of critical insights from their log data is more significant.
Participants also shared details about their current streaming data use. Streaming data connects, filters, processes, and routes log data between different observability tools (commercial or open source) or offline cold storage (S3, Google Cloud Services, etc.) and is sometimes called observability pipeline or observability data management. According to responses, streaming observability pipelines adoption is a work in progress, with 20% of organizations reporting full deployments while 36% are evaluating or considering options.
Report findings also reveal:
■ Observability log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. In addition, 68% say log data is necessary, but it's tough to work with.
■ IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data or troubleshooting, and security analyses are impacted.
■ Existing log management tools present challenges and risks related to scalability, 97% of respondents report.
■ Log data is key to observability, and innovation is needed. 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don't evolve.
■ Problems are beyond storing data. For example, 96% report the need also to use the data to solve business problems.
■ 90% report reducing engineering toil to scale tools helps IT focus on more important work.
■ Volumes of log data in organizations are exploding, according to 96% of IT professionals surveyed.
Survey Demographics: Roles include a third IT executives, a third enterprise (cloud or application) architects, and a third in DevOps/SRE/Ops roles. Companies are in the following regions: AMER (77%), followed by EMEA (20%) and APAC (3%), and include a variety of industry verticals, including financial, technology, healthcare, service, retail, manufacturing, etc.
The Latest
Users have high expectations around applications — quick loading times, look and feel visually advanced, with feature-rich content, video streaming, and multimedia capabilities — all of these devour network bandwidth. With millions of users accessing applications and mobile apps from multiple devices, most companies today generate seemingly unmanageable volumes of data and traffic on their networks ...
In Italy, it is customary to treat wine as part of the meal ... Too often, testing is treated with the same reverence as the post-meal task of loading the dishwasher, when it should be treated like an elegant wine pairing ...
In order to properly sort through all monitoring noise and identify true problems, their causes, and to prioritize them for response by the IT team, they have created and built a revolutionary new system using a meta-cognitive model ...
As we shift further into a digital-first world, where having a reliable online experience becomes more essential, Site Reliability Engineers remain in-demand among organizations of all sizes ... This diverse set of skills and values can be difficult to interview for. In this blog, we'll get you started with some example questions and processes to find your ideal SRE ...
US government agencies are bringing more of their employees back into the office and implementing hybrid work schedules, but federal workers are worried that their agencies' IT architectures aren't built to handle the "new normal." They fear that the reactive, manual methods used by the current systems in dealing with user, IT architecture and application problems will degrade the user experience and negatively affect productivity. In fact, according to a recent survey, many federal employees are concerned that they won't work as effectively back in the office as they did at home ...
Users today expect a seamless, uninterrupted experience when interacting with their web and mobile apps. Their expectations have continued to grow in tandem with their appetite for new features and consistent updates. Mobile apps have responded by increasing their release cadence by up to 40%, releasing a new full version of their app every 4-5 days, as determined in this year's SmartBear State of Software Quality | Application Stability Index report ...
In this second part of the blog series, we look at how adopting AIOps capabilities can drive business value for an organization ...
ITOPS and DevOps is in the midst of a surge of innovation. New devices and new systems are appearing at an unprecedented rate. There are many drivers of this phenomenon, from virtualization and containerization of applications and services to the need for improved security and the proliferation of 5G and IOT devices. The interconnectedness and the interdependencies of these technologies also greatly increase systems complexity and therefore increase the sheer volume of things that need to be integrated, monitored, and maintained ...
IT talent acquisition challenges are now heavily influencing technology investment decisions, according to new research from Salesforce's MuleSoft. The 2022 IT Leaders Pulse Report reveals that almost three quarters (73%) of senior IT leaders agree that acquiring IT talent has never been harder, and nearly all (98%) respondents say attracting IT talent influences their organization's technology investment choices ...
The findings of the 2022 Observability Forecast offer a detailed view of how this practice is shaping engineering and the technologies of the future. Here are 10 key takeaways from the forecast ...
