Are We Approaching an Observability Tipping Point?
July 26, 2022

James Mountifield
Humio

Share this

Observability is a trending topic in the DevOps landscape today. As with most trending topics, there are many declarations that the concept is important but fewer discussions about what the trend actually is or why it's important.

In the DevOps world, observability is trumpeted and lauded in many corners. However, in reading much of the coverage, there seemed to be some more fundamental issues at play. It's time to demystify the idea of observability, shedding light on what it means in a broader context. And once we break down the concept and its true value to an organization, let's answer a more important question: Are we approaching an observability tipping point?

What is Observability?

Fundamentally, observability is the determination of the state of a system or application based on the data it publishes. That data could be metrics, logs or more abstract data like traces. The ability to fully observe a system or application is critical for knowing if it has a problem or, even worse, if it has been compromised.

It's important to make a distinction between observability and monitoring. These are related concepts, but they differ in function. Monitoring uses observability data to make predictions on things that could become a problem and elevate trends related to key system or application indicators over time. In a sense, effective monitoring is built on a solid foundation of observability.

If we move our viewing lens to a higher, market-level view, observability is seen as the next frontier in day-to-day business continuity and as a driver of a necessary cultural shift. There is a steady increase in search engine hits on "observability" over the past five years. Along with an increasing interest in observability, more companies are marketing observability and monitoring solutions.

Lastly, we've seen acquisitions of observability-centric companies, which is another good signal of market viability. At the same time, Gartner pegged the observability market in the billions of dollars with a projected year-over-year increase topping 15%.

So, the observability trend is valid. The market has spoken, and it is here to stay. Beyond the hype, let's explore what organizations should be doing today to turn observability into a strength and a competitive differentiator. To get there, it's important to understand and appreciate the trends in observability.

What Are the Trends in Observability?

Trend #1: Observability is approaching "mission critical" status
Observability is seeing a shift that we saw previously with the move to public clouds. For businesses working with cloud vendors, it became mission critical to have flexible workloads that could deploy anywhere.

Ironically, the amount of configurations, cloud options and deployment options has led to the idea that observability is also mission critical for a business. Without comprehensive observability across on-premises, cloud and other deployment options, the likelihood of suffering reputation damage increases when an issue can't be rectified.

To put some metrics behind this, recent publications estimate that there are around 2.5 quintillion bytes of data created every day. A quintillion is a billion billion, so it's an almost incomprehensible amount of data. This volume will only increase as the number of active IoT devices in the world increases. Some major search engines process hundreds of petabytes of data each day. 

For the average mid-sized business, which may generate a few hundred gigabytes of data per day, that still means that you'll need search capabilities for your log data that can handle at least a terabyte or more data if you want to look back more than a few days. Modern log management platforms can exceed 1 Petabyte per day, which makes it easier to achieve the scale needed for mission-critical data retention to support observability.

Trend #2: Unification is increasing
Both in tooling and in culture, we will see a continued increase in unity as observability increases in importance. A primary driver for observability platforms will be an increased need for collaboration between engineers, DevOps/SREs and business stakeholders. 

Collaboration is easiest when there are fewer observability tools in place, allowing groups to more effectively work in the same environment instead of separate, independent tools. Forward-thinking organizations will opt for an observability approach using enterprise capabilities like RBAC and high availability, coupled with live dashboards that provide the perfect environment for sharing knowledge within and across teams

Trend #3: Observability is happening earlier in the process
Making applications and systems observable from the start is quickly becoming as important as a solid security posture or continuous integration and testing. It's becoming less acceptable to allow black box applications and systems. Observability must be a first-class citizen and first-step consideration before deploying to any environment.

Moreover, observability needs a place in the development environment just as it is for production. Discovering, developing and fine tuning the pieces of observability that make up the whole picture of your application can be a part of each stage of software and system development.

What Do These Trends Mean for You?

As observability reaches its tipping point — and adapts to new dynamics and drivers — how will your organization respond?

We have three recommendations for next steps.

Assess your position
First, aggressively assess your current position. Some important questions to ask yourself at this step are:

■ Have you adopted a "log everything" approach so that you're prepared for unexpected events?

■ Are all systems and applications in your environment observable?

■ How quickly can you perform a root cause analysis based on the observability information you have now?

If your answers are not "yes", "yes", and "within minutes," you may have some gaps in your observability infrastructure.

Close the gaps
Once you have identified gaps, you'll need to work quickly to close them. Why? Observability is all about data, and the longer critical data is missing, the more likely it is that you'll find yourself unprepared or unable to solve a problem that could cripple your business. If you want to have full observability, be able to log everything and solve a problem within minutes, then you must have a comprehensive observability solution in place.

Make the shift
Third, you'll need to start getting ahead of the trends. Observability is no longer a "nice to have," it's integral to your future success. As a result, more businesses are adopting this concept. Now is the time to make it part of your standard operating model. To do this, make it a requirement that all applications and systems pass a rigorous review that ensures they're observable in the right ways.

Red Team VS Blue Team exercises are great for finding gaps not just in security, but also in observability. In these exercises, internal teams intentionally try to find where observability can be improved for an application or system.

Beyond the cultural and technical shift of making observability paramount, you also need to adopt an approach and a platform to handle all of your observability needs and remove the silos. As observability achieves mission critical status in your business, make it a high priority to shift observability to a more prominent and earlier part of your process.

Conclusion

If these trends continue as they are, the new push in 2023 might just be DevSecObsOps. Maybe that won't happen — it's a mouthful. But, observability will continue to increase in focus as more businesses realize the value of observability and how critical it is to their businesses.

James Mountifield is Director of Product Management at Humio, a CrowdStrike company
Share this

The Latest

December 08, 2022

Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2023. Part 4 covers monitoring, site reliability engineering and ITSM ...

December 07, 2022

Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2023. Part 3 covers OpenTelemetry ...

December 06, 2022

Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2023. Part 2 covers more on observability ...

December 05, 2022

The Holiday Season means it is time for APMdigest's annual list of Application Performance Management (APM) predictions, covering IT performance topics. Industry experts — from analysts and consultants to the top vendors — offer thoughtful, insightful, and often controversial predictions on how APM, observability, AIOps and related technologies will evolve and impact business in 2023. Part 1 covers APM and Observability ...

December 01, 2022

You could argue that, until the pandemic, and the resulting shift to hybrid working, delivering flawless customer experiences and improving employee productivity were mutually exclusive activities. Evidence from Catchpoint's recently published Site Reliability Engineering (SRE) industry report suggests this is changing ...

November 30, 2022

There are many issues that can contribute to developer dissatisfaction on the job — inadequate pay and work-life imbalance, for example. But increasingly there's also a troubling and growing sense of lacking ownership and feeling out of control ... One key way to increase job satisfaction is to ameliorate this sense of ownership and control whenever possible, and approaches to observability offer several ways to do this ...

November 29, 2022

The need for real-time, reliable data is increasing, and that data is a necessity to remain competitive in today's business landscape. At the same time, observability has become even more critical with the complexity of a hybrid multi-cloud environment. To add to the challenges and complexity, the term "observability" has not been clearly defined ...

November 28, 2022

Many have assumed that the mainframe is a dying entity, but instead, a mainframe renaissance is underway. Despite this notion, we are ushering in a future of more strategic investments, increased capacity, and leading innovations ...

November 22, 2022

Most (85%) consumers shop online or via a mobile app, with 59% using these digital channels as their primary holiday shopping channel, according to the Black Friday Consumer Report from Perforce Software. As brands head into a highly profitable time of year, starting with Black Friday and Cyber Monday, it's imperative development teams prepare for peak traffic, optimal channel performance, and seamless user experiences to retain and attract shoppers ...

November 21, 2022

From staffing issues to ineffective cloud strategies, NetOps teams are looking at how to streamline processes, consolidate tools, and improve network monitoring. What are some best practices that can help achieve this? Let's dive into five ...