Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2024. Part 3 covers even more on Observability.
Observability Recognized as a Data Problem
Despite pouring $17 billion into observability and monitoring tools each year, enterprises are seeing a negligible impact on mean-time-to-resolution (MTTR) — in fact it is increasing. Why? Modern distributed applications are complex, they change multiple times a day which leads to DevOps teams seeing "unknown" problems in production every day. When troubleshooting an "unknown" problem, DevOps teams must triangulate on data points to determine where the problem may be occurring. That's where the problems start. Some data points are in a logging tool, a monitoring tool, or an APM tool. The best practice is often to screenshot what each tool is showing and post in a Slack channel so the final decision maker can correlate. This is not sustainable. For Observability to deliver on its promise, the observability data must be in one place, not in several silos. If the data is in one place it's easier to navigate, find relevant context for the incident being investigated, and for the DevOps team to collaborate in one consistent interface (that's not Slack!).
TELEMETRY DATA AS AN ENTERPRISE ASSET
As organizations scale, telemetry data complexity, volume, and cost of data management will continue to increase. Observability, security, and business teams will demand access to a wider variety of data. More functional groups will consider incorporating AI/ML into their observability workflows, but data quality will constrain business outcomes. As organizations realize that it's not the ML models but underlying data that will separate them from their competitors, they will have to take a newer enterprise approach to manage their telemetry data. In the coming year, we will see organizations taking steps to manage telemetry data as an enterprise asset and putting mechanisms and governance in place, first to understand the data and then provision the right data in suitable formats, with appropriate quality and context to observability and security teams.
DATA TIERING FOR OBSERVABILITY
Data Tiering will be widely adopted with buy-in from Engineer to Executive. It's not sustainable to store all your observability data in a premium storage tier, like a log data index. This is evidenced by the fact that 82% of DevOps and SRE teams limit log ingestion as a mechanism to reduce costs often or all the time. Moving forward, teams will leverage a combination of best-of-breed tools to move their data to the optimal location based on their use case (real-time analytics, ad hoc queries, compliance, etc.) and desired cost efficiencies (hot and searchable index, object storage, etc.).
CEO, Edge Delta
OBSERVABILITY CONVERGES WITH SECURITY
An emerging trend in 2023 — one that is sure to grow in 2024 — is the need for shared visibility into key enabling apps and IT infrastructure technologies from both an operational and security standpoint. Kubernetes is a primary example. No matter what model or teams you support internally, there's a shared interest in the performance and security of technologies like Kubernetes that are so fundamental to modern apps and infrastructure. Regardless of whether security and observability will converge in terms of organizational structure and roles and responsibilities of teams, there does seem to be a big advantage to adding security capabilities to the traditional three pillars of logs, metrics and traces for observability.
Co-Founder and CTO, Logz.io
OBSERVABILTY TRAINING TO COMBAT STAFF SHORTAGE
In the past few years, IT has gotten increasingly more complex and interconnected. The array of tools required to manage the sprawl of Digital Experience Management tools has also expanded, as have the alerts that require the attention of short-staffed IT teams. The complexity of managing the sprawl is biting into productivity and profitability. We have heard IT leaders say slow-running systems and applications, as well as outdated technology, are directly impacting the growth and performance of their organization. One of the key challenges for IT leaders who are driving digital transformation and improving DEM is being understaffed or having employees who lack key skills. To address this challenge, we will see more IT leaders set aside budget to retrain IT staff and focus on Unified Observability technology with greater automation to help close the skills gap.
VP Product Marketing, Riverbed
OBSERVABILITY FOR FINOPS
Observability will move up the organization to support the sustainability and FinOps drive. The combined pressure of needing to adopt more sustainable practices and tackle rising cloud costs will catapult observability from an IT priority to a business requirement in 2024. On the one hand, organizations will face the dualling trend of seeing their use of AI increasing cloud costs and adding to scope 3 emissions, while also — when applied to observability data — providing insights into how carbon footprint and expenditure can be reduced to offset this impact. By applying AI in this way, organizations will mature their emerging FinOps practices, by identifying hotspots of inefficiency in their cloud environment to enable more effective cost optimization. As an added advantage, this increased use of observability will make it easier for organizations to not just report on their ESG goals but gain the granular insights they need to continuously identify and eliminate hotspots where resources are being inefficiently consumed. This in turn enables them to orchestrate systems for optimal resource utilization, reducing emissions as well as the cost of running their cloud environments. We will therefore see growing interest in use cases of observability beyond the IT department as the wider business begins to take note.
CTO and Founder, Dynatrace
COST OF OBSERVABILITY INCREASES
The costs of observability will swell with data explosion and bloat, so find observability solutions that offer innovative ways to reduce the costs of data storage. One problem with running cloud-native applications on multi-cloud, multi-region architectures is that they generate enormous amounts of observability data — and increasingly so as organizations scale. With the volume of observability data exploding, most organizations aren't equipped to handle the budget-breaking costs that ensue. We can beat this observability bloat by investing in smart data collection and data hygiene. Many organizations just collect everything they can get their hands on. This costs a lot of money, creates noisy clutter in the environment, and hampers the ability to search and gain insights. Instead, observability solutions should help organizations filter out unneeded data to reduce costs and also help customers move their less valued data into far less expensive storage repositories without losing the ability to actively query it when needed.
Co-Founder and CTO, Logz.io
FOCUS ON COST OPTIMIZATION FOR OBSERVABILITY
Buyers of observability tools will continue to become more cost conscious. Tools that adjust the amount of observability data collected based on the health of the underlying services will become powerful solutions for organizations who want to control their observability spend without increasing risk.
VP of Engineering, Circonus
The year 2024 will continue to proudly carry the banner of efficiency and cost optimization. The main thing organizations focused in 2023 were the cloud providers, which in most cases are the top bill for any cloud-based company. Next in line are observability solutions which in some cases add up to 25% on top of the cloud provider expenses. Teams will turn to more cost effective solutions for log management, metrics and full tracing which are continuing to grow in data volumes. We'll see SMBs taking wiser decisions into which observability solution to adopt based on future projected costs and adopting multi-year deals that guardrail expenses, understanding the magnitude of their potential switching costs.
CEO and Co-Founder, groundcover
BOLSTERING OBSERVABILITY ROI: TYING OPERATIONAL STATUS TO BUSINESS OUTCOMES
Enterprise IT leaders struggle with rising observability costs. As custom metadata grows to provide increased context, enterprises are often saddled with surprise billing overages. IT organizations have been trained to reduce these costs by reducing the observability footprint, but that often leaves blind spots, increasing downtime and reducing ROI. To bolster ROI, enterprise IT leaders and business observability teams need to tie operational status to business outcomes. This approach helps teams reduce the need for increased data and spending time on dashboards. Instead, they can focus on what matters for customer experience.
VP, Product Marketing, Elastic
NEW OBSERVABILITY PRICING MODELS
We'll see observability vendors offering more diverse pricing models that include paying by the queried or used data only, paying by seats or paying by infrastructure size alone, trying to offer customer alternatives to the controversial volume-based models.
CEO and Co-Founder, groundcover
TOOL SPRAWL BECOMES A THING OF THE PAST
In 2024, tool sprawl will start to become a thing of the past. Companies often fall victim to employing an unnecessarily high number of IT tools, causing bottlenecks, inefficiencies, and negative ROI. Tech professionals are spending more time learning the intricacies of these tools than they are completing their actual work. Leaders need to mitigate this issue by consolidating their implementations — allowing for better manageability, efficiency, and security. With the constantly changing digital transformation, companies need to opt for a "best of suite" to ensure their employees can get to the real work that needs to be done.
Go to: 2024 Application Performance Management Predictions - Part 4, covering OpenTelemetry.
Generative AI will usher in advantages within various industries. However, the technology is still nascent, and according to the recent Dynatrace survey there are many challenges and risks that organizations need to overcome to use this technology effectively ...
In today's digital era, monitoring and observability are indispensable in software and application development. Their efficacy lies in empowering developers to swiftly identify and address issues, enhance performance, and deliver flawless user experiences. Achieving these objectives requires meticulous planning, strategic implementation, and consistent ongoing maintenance. In this blog, we're sharing our five best practices to fortify your approach to application performance monitoring (APM) and observability ...
In MEAN TIME TO INSIGHT Episode 3, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses network security with Chris Steffen, VP of Research Covering Information Security, Risk, and Compliance Management at EMA ...
In a time where we're constantly bombarded with new buzzwords and technological advancements, it can be challenging for businesses to determine what is real, what is useful, and what they truly need. Over the years, we've witnessed the rise and fall of various tech trends, such as the promises (and fears) of AI becoming sentient and replacing humans to the declaration that data is the new oil. At the end of the day, one fundamental question remains: How can companies navigate through the tech buzz and make informed decisions for their future? ...
We increasingly see companies using their observability data to support security use cases. It's not entirely surprising given the challenges that organizations have with legacy SIEMs. We wanted to dig into this evolving intersection of security and observability, so we surveyed 500 security professionals — 40% of whom were either CISOs or CSOs — for our inaugural State of Security Observability report ...