Are We Approaching an Observability Tipping Point?
July 26, 2022

James Mountifield
Humio

Share this

Observability is a trending topic in the DevOps landscape today. As with most trending topics, there are many declarations that the concept is important but fewer discussions about what the trend actually is or why it's important.

In the DevOps world, observability is trumpeted and lauded in many corners. However, in reading much of the coverage, there seemed to be some more fundamental issues at play. It's time to demystify the idea of observability, shedding light on what it means in a broader context. And once we break down the concept and its true value to an organization, let's answer a more important question: Are we approaching an observability tipping point?

What is Observability?

Fundamentally, observability is the determination of the state of a system or application based on the data it publishes. That data could be metrics, logs or more abstract data like traces. The ability to fully observe a system or application is critical for knowing if it has a problem or, even worse, if it has been compromised.

It's important to make a distinction between observability and monitoring. These are related concepts, but they differ in function. Monitoring uses observability data to make predictions on things that could become a problem and elevate trends related to key system or application indicators over time. In a sense, effective monitoring is built on a solid foundation of observability.

If we move our viewing lens to a higher, market-level view, observability is seen as the next frontier in day-to-day business continuity and as a driver of a necessary cultural shift. There is a steady increase in search engine hits on "observability" over the past five years. Along with an increasing interest in observability, more companies are marketing observability and monitoring solutions.

Lastly, we've seen acquisitions of observability-centric companies, which is another good signal of market viability. At the same time, Gartner pegged the observability market in the billions of dollars with a projected year-over-year increase topping 15%.

So, the observability trend is valid. The market has spoken, and it is here to stay. Beyond the hype, let's explore what organizations should be doing today to turn observability into a strength and a competitive differentiator. To get there, it's important to understand and appreciate the trends in observability.

What Are the Trends in Observability?

Trend #1: Observability is approaching "mission critical" status
Observability is seeing a shift that we saw previously with the move to public clouds. For businesses working with cloud vendors, it became mission critical to have flexible workloads that could deploy anywhere.

Ironically, the amount of configurations, cloud options and deployment options has led to the idea that observability is also mission critical for a business. Without comprehensive observability across on-premises, cloud and other deployment options, the likelihood of suffering reputation damage increases when an issue can't be rectified.

To put some metrics behind this, recent publications estimate that there are around 2.5 quintillion bytes of data created every day. A quintillion is a billion billion, so it's an almost incomprehensible amount of data. This volume will only increase as the number of active IoT devices in the world increases. Some major search engines process hundreds of petabytes of data each day. 

For the average mid-sized business, which may generate a few hundred gigabytes of data per day, that still means that you'll need search capabilities for your log data that can handle at least a terabyte or more data if you want to look back more than a few days. Modern log management platforms can exceed 1 Petabyte per day, which makes it easier to achieve the scale needed for mission-critical data retention to support observability.

Trend #2: Unification is increasing
Both in tooling and in culture, we will see a continued increase in unity as observability increases in importance. A primary driver for observability platforms will be an increased need for collaboration between engineers, DevOps/SREs and business stakeholders. 

Collaboration is easiest when there are fewer observability tools in place, allowing groups to more effectively work in the same environment instead of separate, independent tools. Forward-thinking organizations will opt for an observability approach using enterprise capabilities like RBAC and high availability, coupled with live dashboards that provide the perfect environment for sharing knowledge within and across teams

Trend #3: Observability is happening earlier in the process
Making applications and systems observable from the start is quickly becoming as important as a solid security posture or continuous integration and testing. It's becoming less acceptable to allow black box applications and systems. Observability must be a first-class citizen and first-step consideration before deploying to any environment.

Moreover, observability needs a place in the development environment just as it is for production. Discovering, developing and fine tuning the pieces of observability that make up the whole picture of your application can be a part of each stage of software and system development.

What Do These Trends Mean for You?

As observability reaches its tipping point — and adapts to new dynamics and drivers — how will your organization respond?

We have three recommendations for next steps.

Assess your position
First, aggressively assess your current position. Some important questions to ask yourself at this step are:

■ Have you adopted a "log everything" approach so that you're prepared for unexpected events?

■ Are all systems and applications in your environment observable?

■ How quickly can you perform a root cause analysis based on the observability information you have now?

If your answers are not "yes", "yes", and "within minutes," you may have some gaps in your observability infrastructure.

Close the gaps
Once you have identified gaps, you'll need to work quickly to close them. Why? Observability is all about data, and the longer critical data is missing, the more likely it is that you'll find yourself unprepared or unable to solve a problem that could cripple your business. If you want to have full observability, be able to log everything and solve a problem within minutes, then you must have a comprehensive observability solution in place.

Make the shift
Third, you'll need to start getting ahead of the trends. Observability is no longer a "nice to have," it's integral to your future success. As a result, more businesses are adopting this concept. Now is the time to make it part of your standard operating model. To do this, make it a requirement that all applications and systems pass a rigorous review that ensures they're observable in the right ways.

Red Team VS Blue Team exercises are great for finding gaps not just in security, but also in observability. In these exercises, internal teams intentionally try to find where observability can be improved for an application or system.

Beyond the cultural and technical shift of making observability paramount, you also need to adopt an approach and a platform to handle all of your observability needs and remove the silos. As observability achieves mission critical status in your business, make it a high priority to shift observability to a more prominent and earlier part of your process.

Conclusion

If these trends continue as they are, the new push in 2023 might just be DevSecObsOps. Maybe that won't happen — it's a mouthful. But, observability will continue to increase in focus as more businesses realize the value of observability and how critical it is to their businesses.

James Mountifield is Director of Product Management at Humio, a CrowdStrike company
Share this

The Latest

March 01, 2024

As organizations continue to navigate the complexities of the digital era, which has been marked by exponential advancements in AI and technology, the strategic deployment of modern, practical applications has become indispensable for sustaining competitive advantage and realizing business goals. The Info-Tech Research Group report, Applications Priorities 2024, explores the following five initiatives for emerging and leading-edge technologies and practices that can enable IT and applications leaders to optimize their application portfolio and improve on capabilities needed to meet the ambitions of their organizations ...

February 29, 2024

Despite the growth in popularity of artificial intelligence (AI) and ML across a number of industries, there is still a huge amount of unrealized potential, with many businesses playing catch-up and still planning how ML solutions can best facilitate processes. Further progression could be limited without investment in specialized technical teams to drive development and integration ...

February 28, 2024

With over 200 streaming services to choose from, including multiple platforms featuring similar types of entertainment, users have little incentive to remain loyal to any given platform if it exhibits performance issues. Big names in streaming like Hulu, Amazon Prime and HBO Max invest thousands of hours into engineering observability and closed-loop monitoring to combat infrastructure and application issues, but smaller platforms struggle to remain competitive without access to the same resources ...

February 27, 2024

Generative AI has recently experienced unprecedented dramatic growth, making it one of the most exciting transformations the tech industry has seen in some time. However, this growth also poses a challenge for tech leaders who will be expected to deliver on the promise of new technology. In 2024, delivering tangible outcomes that meet the potential of AI, and setting up incubator projects for the future will be key tasks ...

February 26, 2024

SAP is a tool for automating business processes. Managing SAP solutions, especially with the shift to the cloud-based S/4HANA platform, can be intricate. To explore the concerns of SAP users during operational transformations and automation, a survey was conducted in mid-2023 by Digitate and Americas' SAP Users' Group ...

February 22, 2024

Some companies are just starting to dip their toes into developing AI capabilities, while (few) others can claim they have built a truly AI-first product. Regardless of where a company is on the AI journey, leaders must understand what it means to build every aspect of their product with AI in mind ...

February 21, 2024

Generative AI will usher in advantages within various industries. However, the technology is still nascent, and according to the recent Dynatrace survey there are many challenges and risks that organizations need to overcome to use this technology effectively ...

February 20, 2024

In today's digital era, monitoring and observability are indispensable in software and application development. Their efficacy lies in empowering developers to swiftly identify and address issues, enhance performance, and deliver flawless user experiences. Achieving these objectives requires meticulous planning, strategic implementation, and consistent ongoing maintenance. In this blog, we're sharing our five best practices to fortify your approach to application performance monitoring (APM) and observability ...

February 16, 2024

In MEAN TIME TO INSIGHT Episode 3, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses network security with Chris Steffen, VP of Research Covering Information Security, Risk, and Compliance Management at EMA ...

February 15, 2024

In a time where we're constantly bombarded with new buzzwords and technological advancements, it can be challenging for businesses to determine what is real, what is useful, and what they truly need. Over the years, we've witnessed the rise and fall of various tech trends, such as the promises (and fears) of AI becoming sentient and replacing humans to the declaration that data is the new oil. At the end of the day, one fundamental question remains: How can companies navigate through the tech buzz and make informed decisions for their future? ...