If you talk to any security or observability teams right now, they're all fighting the same fire: their tooling was built to ingest X, but their sources are pumping Y and soon to be doing Z. The knee-jerk reaction is always the same: we need more platform. However, this reaction is wrong. Let me explain why, because the solution to this problem is foundational, not financial.
Instead of hurling yet more money at the problem, make sure you've done what's needed upstream.
Background: When Telemetry Growth Surpassed Monitoring Capacity
This isn't the first time telemetry growth has outpaced traditional monitoring architectures. Early cloud-native adoption was the first growth spike we saw. Dependency on microservices, distributed systems, and just more instances stretched these monitoring systems to their limits, and teams reacted mostly by scaling up. Bigger Elasticsearch clusters. Bigger databases. Bigger warehouses.
AI is the second spike that's going to tear those tactics apart. AIOps tooling, security analytics, compliance monitoring, and now AI workloads themselves — each either generates new telemetry or demands more of what you already have. That database you scaled to support five years of growth is now being fed several multiples of that. AI workloads emit telemetry at volumes and in formats nothing upstream was designed to handle. And the strain surfaces last at the monitoring platform, right before it becomes the bottleneck.
A Real Example: When Scaling the SIEM Stopped Working
One of our customers runs security operations at meaningful scale. For years, telemetry growth stayed inside what their SIEM could absorb. Then that growth outpaced the SIEM, and the default move was to evaluate a bigger license.
The actual fix had nothing to do with the SIEM. It was the lack of a pipeline layer in front of it. Most web server logs, for instance, are 2xx-status entries confirming normal operation — data with essentially zero detection value. Condense that at the pipeline layer, surface only what's actionable, and the ingestion problem shrinks immediately. Apply the same filtering logic to Windows events, application logs, and infrastructure telemetry, and total volume drops 40 percent or more with no loss of meaningful signal.
The catch: this only works upstream. Filtering has to happen before data hits the SIEM or observability backend, not as a downstream cleanup step.
Why Platform-Bundled Collection No Longer Holds Up
For years, running the agents your observability or SIEM vendor bundled was good enough. It wasn't especially elegant, but reasonably functional.
Those bundled agents are now a liability when it comes to your ability to adapt when AI starts throwing new sources and use cases at you. If your collection tier is glued to one vendor, bringing in a new tool to handle a specific use case is often a six-month re-instrumentation slog.
Break that coupling by inserting your own pipeline in front of everything, and that completely changes. The process for routing telemetry to a new tool goes from weeks of database and ingestion setup to a config change. Teams can evaluate tools by feeding the same pipeline into an observability platform alongside their SIEM and swap them around in days.
What a Real Pipeline Needs to Do
OpenTelemetry has effectively settled the instrumentation question. It's the vendor-neutral standard most teams are converging on. But the standard itself isn't the pipeline. What sits on top of OTel is what determines whether you have a real control layer: real-time visibility into telemetry in motion, the ability to re-route without touching instrumentation, and the operational reliability to run this at production scale without becoming its own outage risk.
The Tradeoff Is Real, and Worth Naming
Of course, setting up a pipeline isn't free. Defining a schema and getting application and infrastructure teams to adopt it takes coordination. This is a genuine cost, not a footnote.
But the payoff compounds. Standardize your pipeline and watch it propagate org-wide without re-litigating "how we ingest" every other month. Teams that used to spend a significant percentage of their time just trying to get log payloads someone else designed into the SIEM can now actually start working on hard problems.
Ask most observability or security teams how many platforms they're feeding data into, and the honest answer from about half the room is "all of them" — a dozen vendors, a dozen ingestion paths, zero shared schema. A pipeline layer is what actually resolves that, turning fragmented telemetry into a single normalized asset instead of a standing liability.
AI Makes This a Compliance Issue, Not Just an Efficiency One
AI is adding massive new categories of telemetry most monitoring platforms weren't designed to handle: model prompts and responses, lineage and evaluation data, audit trails of what data reached which model under what terms. Add in more restrictive data residency laws coming out of the EU, and you've got a compliance nightmare.
Building your pipeline with OpenTelemetry at its core solves this. One cohesive pipeline means enforcing data residency rules in one place, across your AI, security, and observability telemetry. Without it, you're retrofitting a separate compliance layer on top of each ingestion point. One at a time.
Your stack isn't going to get any less fragmented just because regulators want their data guarantees. If that data can't leave country X, something needs to enforce that at scale, and a telemetry pipeline is the only layer built for that problem natively.