Observability and security — are they a match made in IT heaven, or a culture clash from IT hell? Sorry to be so dramatic, but it's actually a serious question that has gravity. The convergence of observability and security could change IT operations as we know it. And many IT authorities see this as a good thing.

With input from industry experts — both analysts and vendors — this 8-part blog series to be posted over the next two weeks will explore what is driving this convergence, the challenges and advantages, and how it may transform the IT landscape.

Security and observability are really made for each other

"Security and observability are really made for each other," says Mike Loukides, VP of Emerging Tech Content at O'Reilly Media. "Security has always suffered from a lack of information. Logs and metrics just don't give you that much to work with. Add the trace data that a good observability platform can give you, and there's much more to work with. Which means a much greater chance of catching an intruder early, before they've had a chance to do a lot of damage."

Chaim Mazal, Chief Security Officer at Gigamon cites a recent study that found observability delivers a mix of tactical (resolution, continuity, tracking) and strategic (experience, governance, innovation) benefits, with security ranking as the highest benefit — 34% of surveyed IT leaders agreed.

Growing Complexity Makes Convergence a Necessity

The emergence of new technologies — including cloud computing, microservices and containerization — has led to more complex, connected systems, notes Roger Floren, Principal Product Manager at Red Hat. This complexity makes it harder to monitor and secure applications efficiently. So a holistic approach that combines both security and observability is the next natural step.

"Complexity is driving this convergence," Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) agrees. "When troubleshooting performance problems, IT operations teams often find that root cause is actually a security incident. This points to the need for better partnerships between IT/Network operations and security. Much of this complexity is driven by hybrid and multi-cloud architectures, which are causing both IT ops and security teams headaches."

Use the player or download the MP3 below to listen to EMA-APMdigest Podcast Episode 2 — Shamus McGillicuddy talks about Network Observability, the convergence of observability and security, and more.

Click here for a direct MP3 download of Episode 2 - Part 1

Because many organizations are expanding across a mix of cloud, fast development cycles, low-code and no-code platforms, this has significantly expanded the attack surface, according to Gregg Ostrowski, CTO Adviser at Cisco AppDynamics. He says, to identify and address higher volumes of security alerts, organizations must prioritize full visibility across complex IT environments, which can be achieved with observability.

"In today's complex, fast-paced environment, modern organizations are often perpetually overwhelmed and find themselves trapped in a cycle of reactivity," Spiros Xanthos, SVP and General Manager of Observability, Splunk, elaborates. "They're constantly dealing with cybersecurity threats, IT system stressors, and other adverse events; all while trying to keep their systems secure and reliable. To overcome such challenges, these organizations need to be able to detect, investigate and respond more quickly; pivot when the macro-environment demands it; and adapt, so they can respond to future events better."

"Taking a unified approach to security and observability helps address these challenges," he continues, "because it enables SecOps, ITOps and DevOps to work in tandem — not in silos — to proactively maintain business resilience and keep these adverse events at bay without slowing down innovation."

"The convergence is being driven by the general realization that observability and security are really two sides of the same coin," adds Glenn Gray, Director of Product Marketing at Auvik. "Simply put, you cannot properly secure IT infrastructure that you do not fully understand or regularly monitor. If one accepts that notion, then observability becomes a key component of any good IT infrastructure security strategy."

Combining observability and security is no longer an option — it is a necessity, warns Amit Shah, Director of Product Marketing at Dynatrace. "Providing observability context to security data can help organizations find issues that have escaped into runtime and enable teams to focus on what really matters. Additionally, observability-driven security can provide an additional layer of protection to catch threats that perimeter security solutions miss."

Cloud Drives Convergence

More specifically, some experts focus on cloud migration as the driving factor behind the convergence between security and observability.

Amit Shah of Dynatrace says, "Increased digital transformation is happening in hybrid and multicloud environments, which are dynamic, complex, and create an explosion of data. Using traditional approaches, it is difficult for organizations to react quickly to changing cloud environments and evolving security threats."

Shah cites the 2023 Global CISO Report from Dynatrace, which shows that more than two-thirds (68%) of CISOs say vulnerability management is more difficult because the complexity of their software supply chain and cloud ecosystem has increased.

"To address these challenges, leading organizations are turning to AI-driven solutions that converge observability and security capabilities," he continues. "These tools enable increased visibility across complex cloud environments and provide precise information so that organizations can automatically identify and reveal the impact of security vulnerabilities in real-time, freeing them up to focus on delivering faster, more secure innovation."

Chaim Mazal of Gigamon adds, "I believe the two key drivers of this overlap are the swift shift to the cloud coupled with the increasing levels of sophistication of the threat actors across today's continuously evolving threat landscape. It is becoming vitally important that NetOps, SecOps, and even DevOps teams work together to ensure cloud security. And this, in turn, requires increasing levels of visibility across hybrid and multi-cloud infrastructure. Technology organizations will be well served to bring network context to their observability tools to detect threats in real-time and mitigate exposure to risk."

All About the Data

Most experts agree that the observability data is what makes convergence compelling, from the security point of view.

Kirsten Newcomer, Director, Cloud and DevSecOps Strategy at Red Hat says, "The convergence is driven by the reality that both solutions need similar data sets and need to answer similar questions about running systems and are using similar technologies for cloud-native, Kubernetes environments."

"The single biggest driver of this convergence is that the IT teams involved with observability have the data, and they must share it with security teams so they can investigate critical threats," adds Adam Hert, Director of Product at Riverbed. "IT teams are collecting extremely large data volumes while, at the same time, gathering additional data from the APM and network sectors. It does not make sense for organizations to do that twice. Observability teams are winning the race when it comes to data gathering, but they need to share that with security teams to boost efficiencies and combat worsening threats and breaches."

An interesting trend is the need for shared visibility into key enabling apps and IT infrastructure technologies from both an operational and security standpoint, and Kubernetes is a primary example, according to Asaf Yigal, CTO of Logz.io. "No matter what model or teams you support internally, there's a shared interest in the performance and security of technologies like Kubernetes that are so fundamental to modern apps and infrastructure. In some cases this is driving greater convergence from a monitoring and observability standpoint, as in shared responsibility for analysis, investigation and response workflows."

Prashant Prahlad, VP of Cloud Security Products at Datadog says, "The added context from the observability data helps customers detect attacks and identify issues sooner than before. Further, the same observability data helps users identify and remediate security issues more quickly than before. Finally, the individuals responsible for observability (SRE/devops) are the ones most familiar with the applications and can resolve security issues sooner than a centralized security team that operates more broadly."

The Big Data Dilemma

Experts also say that convergence of observability and security efforts can help SecOps teams deal with the deluge of data collected across the enterprise.

"With so many tools, vendors, data sources, and technologies, security teams are flooded with mounds of data to sift through," says Esteban Gutierrez, CISO & VP, Information Security at New Relic.

Buddy Brewer, Chief Product Officer at Mezmo explains, "Organizations have been dealing with the challenge of handling an ever-increasing amount of data moving through their systems for a long time. The explosion of log data from cloud environments, stemming from more applications than ever, has overwhelmed many teams — especially security teams."

Brewer goes on to say that organizations realize they need a unified approach to manage telemetry data, both for security and observability. "Challenges such as too much data, data in the wrong format, and data not available to the right teams and applications are common for development, SRE, and security teams. Organizations must have a unified approach to manage the data and make it actionable to reduce MTTD/MTTR. This approach allows security teams to find attacks early and have the data needed to implement fixes before it becomes unmanageable."

Ajit Sancheti, GM, Falcon LogScale at CrowdStrike, agrees, "With the speed of business becoming increasingly faster and adversaries becoming more sophisticated, combining security and observability tools will allow organizations to efficiently operationalize the massive amounts of data currently being generated to better understand the activity inside their IT environments."

Why Now?

After all this discussion, we start to get an answer to the question: Why is the convergence of observability and security heating up now?

"Why now?" Mike Loukides of O'Reilly Media responds. "I don't think that's the right question. Why not three years ago? Giving the security team more data to work with can only be a good thing, and it's surprising it's taken that long to catch on."

Go to: Exploring the Convergence of Observability and Security - Part 2: Logs, Metrics and Traces