In APMdigest's new series, Exploring AI Observability, experts across observability and AI provide insight and opinions on the need for AI Observability, the challenges, the benefits, the current state, and predictions for the future. Part 3 reports on the current state of AI observability.
Most experts agree: enterprises are moving faster with AI than they are with the operational discipline needed to run it safely.
Early Days of AI Observability
Download a complimentary copy: 2026 Gartner® Magic Quadrant™ for Observability Platforms
Most enterprises are still early in their AI Observability journey, says Bernd Greifeneder, CTO and Founder at Dynatrace. AI adoption is advancing faster than the operational maturity needed to run Agentic AI reliably in production. As organizations move beyond experimentation, they need observability foundations that show how AI systems behave in their real application and business context. Without that visibility, it becomes difficult to operate AI at scale with confidence.
"Honestly, most enterprises are still playing catch up, often further behind than they'd like to admit," Krishna Sai, CTO of SolarWinds reveals. "When it comes to LLM-based systems, observability is too often treated as an afterthought. Teams wire logs and prompts into a model and hope their legacy monitoring will catch whatever goes wrong. That approach might survive a demo, but it falls apart the moment you hit real production traffic and the pressure of actual usage and cost."
Find Out What's New with SolarWinds
Sai continues, "The uncomfortable truth is that companies are adopting LLMs faster than they're building the visibility and guardrails needed to run them safely. Most haven't rethought what healthy even looks like for a probabilistic system, so they lack the baseline tracking, drift detection, and response-level visibility required to catch problems before they become incidents. Until that foundation is solid, enterprises simply aren't ready to take on the added complexity of agentic systems, where the questions only get harder."
"It's very early. Most organizations evaluating this space don't have a client or use case in production yet, as they are still comparing vendors, sitting in early demos, and trying to figure out which platform fits," adds Corey Beck, VP of Service Delivery at DataStrike. "Adoption right now is limited to a small number of early movers, and the rest of the market is watching and waiting for the tooling and the practice to mature."
"Most enterprises have mature observability for infrastructure, networks, and applications," Cathy Vuong, Chief Product Officer at TrustScale, concedes, "but AI has introduced an entirely new layer that many organizations are only beginning to monitor. Companies can measure latency, availability, and system health, but few have visibility into AI-specific behaviors such as hallucinations, model drift, agent actions, or response quality. AI observability is where application observability was a decade ago. It's rapidly evolving from a nice-to-have capability into an operational necessity."
Shams Chauthani, CTO at Tempo Software, adds: "It's still early days. Many organizations have basic visibility into prompts, responses, token usage, latency, and errors. That is useful, but it does not provide production-grade control over AI behavior."
"What convinces me we are early is the questions I hear from experienced observability engineers, people who have built large platforms, because their questions sound basic on the surface," says Juraci Paixão Kröhling, CEO of Ollygarden. "What is good telemetry for these systems? How do I observe the quality of a response rather than the shape of it? How do I detect model degradation, and how do I trust that detection? Nobody has settled answers. There is no best practice to follow yet, and anybody who tells you they know the answer is lying. The noise makes that harder to see, because almost every tool in this market grew an AI story in the past year and very few of them grew any new telemetry to go with it. Teams feel the need for this clearly. They are not yet sure what it looks like."
"Industry surveys consistently show that organizations are accelerating AI adoption faster than they are developing operational governance. The result is an operational visibility gap. Organizations know AI is running. They often don't know whether it's consistently delivering high-quality outcomes," notes Jeremy Rossbach, Chief Technical Evangelist | Network Observability by Broadcom.
Organizations lack the operational maturity to run AI reliably, according to Kellyn Gorman, Advocate and Engineer at Redgate Software. IDC research found that 88% of AI pilots fail to reach production, with failures clustering on governance, data-readiness, and observability gaps rather than model quality. On the agent side specifically, while 88% of surveyed organizations used AI in 2025, agent deployment remained in the single digits across most functions. This was often due to reliability concerns, not capability limits, which resulted in holding back scale. Separately, Publicis Sapient's 2026 Global Enterprise AI Report found that AI supports 53% of enterprise work, yet only 18% of organizations have fully integrated AI, and 42% of respondents say AI is capable, but their organization isn't set up to capture its value.
Omer Bhutta, Head of AI and ML at Secure.com, cited a similar data point: "Most enterprises lack what they need today. The typical organization has instrumented the infrastructure layer around its LLM applications, so it can see latency, token counts, and API errors, but it has little visibility into the semantic layer: whether responses are accurate, grounded, safe, and on-task. Evaluation is often manual, sporadic, and disconnected from production telemetry. The numbers bear this out: Gartner estimates that LLM observability investments cover only 15% of GenAI deployments today, though it predicts that share will reach 50% by 2028. Agentic observability is even less mature."
Tool Limitations
Most enterprises have adopted AI faster than they have adopted the ability to observe it, so yes, there is a real gap. AI observability itself is evolving fast, tracing prompts and responses, running evals, tracking token cost, that layer has matured quickly and enterprises are getting real value from it, says Moshe Sambol, VP of Customer Solutions, Lightrun. Where most tools fall short is the layer underneath all of that: runtime context, meaning visibility into what the code an agent generates or triggers is actually doing while it runs. Most tools marketed as AI observability stop at the prompt and the response. They can tell you what an agent decided to do. They can't show you what happened once that decision became running code, what state the system was in, or why a specific outcome occurred. That's the layer that matters most, because that's where the actual damage or the actual value gets created. Without it, every incident an agent causes takes longer to close, and every decision a team can't explain erodes a little more confidence in letting agents run unsupervised at all.
Yechezkel Rabinovich, CTO and Co-Founder at groundcover points out that many legacy observability architectures still rely on sampling, telemetry reduction, or pricing models that encourage teams to collect less data as environments grow; not to mention the assumption that instrumentation can keep up with the pace of change. That may have been acceptable when observability was primarily a troubleshooting tool for humans, but AI agents are only as reliable as the operational data they're given.
"Most enterprises already have what they need," counters Cameron Daniel, CTO, Megaport. "The gap isn't tooling, it's utilization. Teams stand up OTel, wire up the basics, and stop there rather than pushing into the AI-specific metrics, prompt-level tracing, and routing visibility that actually make it useful for these workloads. It's less a technology problem than a maturity and prioritization one. The tools exist, but few teams have invested the time to configure them properly or get full value out of what they're already collecting. That's actually the encouraging part — it means the fix is mostly in your own hands rather than something you're waiting on a vendor to solve."
Current State: Messy
"The current state is messy," Steve Giguere, Principal AI Security Advocate at Check Point, reports. Many organizations can collect basic LLM metrics such as latency, token consumption, error rates, and model usage if they are embracing modern AI gateway architectures. Fewer can consistently connect those metrics to application quality, security policy, user identity, retrieved data, or business outcomes.
The gap is greater for agentic systems, says Giguere. An enterprise may have logs from the model provider, application, API gateway, identity service, vector database, tool, and target system, but no coherent record of why the agent acted, what context influenced the decision, or how the complete sequence unfolded.
"Most organizations have fragments of AI observability rather than a complete operational picture," he continues. "The individual components often produce telemetry, but the enterprise lacks a common schema, shared identifiers, consistent retention, and a way to correlate events across the entire agentic execution path."
Discovery is another major weakness, says Giguere. Observability commonly begins after an application has been formally deployed and instrumented. Enterprises increasingly need visibility earlier: when models, agents, tools, MCP servers, skills, and data connections are created or introduced. Otherwise, the systems presenting the greatest risk may be precisely those absent from the observability platform.
"The gap isn't really about tooling availability — plenty of instrumentation options exist now," said David Girvin, Lead Technical Advocate at Sumo Logic. "The gap is maturity of practice. Production use of dedicated LLM and agent observability is still a minority behavior across enterprises, even though awareness and pilot activity have grown quickly over the past year. What's common today is fragmented visibility: teams bolt together a tracing tool, a manual spreadsheet of evaluation scores, and a cost report from the model provider, with no single view connecting quality, cost, and behavior. Enterprises that have gotten this right tend to be the ones already running mature observability practices elsewhere and extending that discipline to AI, rather than starting from scratch."
Aaron Winston, Head of Content, Communications, and Analyst Relations at Arize AI, says: "Most enterprises have pieces of AI observability rather than a complete operating loop. LLM teams may capture prompts, responses, token usage, latency, cost, or traces while lacking calibrated quality measurement, prompt and model lineage, production-to-dataset workflows, and reliable links to business outcomes. The result is visibility without a repeatable path from a production failure to a tested improvement. Agentic systems widen that maturity gap because their control flow is assembled at runtime and is not fully specified in source code."
Hong Wang, co-founder and CEO of Akuity, outlines what is mature and what isn't: "Model-layer tooling is real and maturing fast. Traces, token accounting, and evaluation have credible products, and OpenTelemetry gave them a shared vocabulary. The action layer is mostly absent. Most enterprises can answer 'what did the model say.' Few can answer 'what did the agent change, under whose authority, and can we undo it.'"
Most enterprises operate in deliberate ignorance, Lior Matkovich, Senior Director, ITOM, ServiceNow, asserts. They have APM monitoring infrastructure, LLM monitoring tracking tokens, workflow logs recording decisions, but none of these systems talk to each other. When a customer service agent fails, operations sees healthy infrastructure, the AI team sees normal token usage, and the business team watches customer issues pile up. Nobody has the full picture. The fragmentation isn't accidental, it's structural. Enterprise observability is fragmented by department ownership, and that was already a mess before agents. If organizations today struggle with silos while knowledge is spread across individuals and departments, what happens when that multiplies by 10x agents running across the enterprise. The control and visibility problems can become catastrophic. The real question becomes: what's the minimum viable observability you need to catch agent failure before it reaches customers? That's where the focus should be.
"The current state of AI observability is panic and anxiety," Jackson Stakeman, GM at Sparq, concludes. "The workloads of already-overburdened teams are changing out from under their feet. In our conversations with enterprises adopting AI, observability is one more thing on the plate, attached to business cases that often aren't fully fleshed out, handed to teams that didn't ask for the assignment. Most of the AI observability conversations we're having are driven by fear rather than by business value or clear goals."
Flying Blind
Making the problem worse, many enterprises believe they're covered when they aren't — mistaking infrastructure health for AI health.
"Most enterprises are flying blind and don't know it," said Kunal Agarwal, Co-founder and CEO of Unravel Data. "They instrumented latency and token counts, put it on a dashboard, and declared victory. Very few are continuously measuring the thing that actually matters, which is whether the outputs are any good in production. That's why so many AI projects sail through the pilot and die in production. The pilot had humans looking at every output. Production doesn't."
"From what I'm seeing in the field, most teams running LLM or agentic systems in production still have little-to-no instrumentation specifically for quality decay." says Anil Inamdar, Head of Data Services at NetApp Instaclustr. "They monitor latency, cost, and error rates, and assume that if those look fine, the system is fine. Then when answer quality drops, it pops up as a vague complaint that the product feels 'off,' and engineers end up debugging code that hasn't changed because the actual cause is drift in the data the model retrieves from."
Gou Rao, Co-Founder and CEO, NeuBird AI says most enterprises are still instrumenting for basic LLM-level metrics, such as token counts, latency and cost, and calling that coverage. As teams move from simple wrappers to multi-agent production systems, many hit a "DIY disillusionment" phase where custom scripts and basic dashboards fail to maintain speed, accuracy and cost control. What teams need is visibility into the agentic layer: the reasoning steps, the context an agent selected and where it diverged from a sound decision path, and how well agents operate within existing enterprise workflows.
"Ask most leaders whether they can trace a bad AI answer back to the document-and-retrieval decision that led to it, and the honest answer is usually no," says Nick Burling, Chief Product Officer, Nasuni. "Not because teams stopped watching their AI systems, but because they are watching the wrong layer. Uptime, latency, and failure rates are well covered industry-wide at this point, but visibility into which data was retrieved is missing. That decision trail is largely invisible in most enterprise AI deployments right now."
Aneesh Nathani, VP – Data and AI Service Line at Cybage Software, used a single word to assess the current state of AI observability: "Uneven. Most enterprises have mature infrastructure and APM practices but little systematic visibility into AI behavior. They know a request returned 200 in 800 milliseconds, not whether the answer was correct. The harder problem is organizational. Most teams haven't defined what 'healthy' looks like for their AI systems, and you can't monitor a standard you haven't set."
Developing Quickly
Despite this often bleak assessment of the current state of AI observability, this is only the beginning — as many experts mentioned — and the current state is changing rapidly.
Download the e-book: The ultimate APM playbook - Master challenges and implement best practices
AI observability is developing quickly, and the IT teams adopting it early are already seeing benefits, reports Gowrisankar Chinnayan, Director of Product Management at ManageEngine. Monitoring data has always existed, but companies are now learning to use it more effectively. Unified telemetry pipelines, real-time topology maps, and governance frameworks are becoming the essential tools for AI observability. IT organizations that invested early in deduplicated, correlated data are finding that agentic AI fits in more smoothly than they expected. The gap is closing, and it's clear that being prepared for AI observability leads to better results.
"Tracing and evaluating agents has come a long way fast," Rohit Choudhary, CEO of Acceldata, adds. "A year ago it was rare, but now most teams can instrument an agent, see its traces and score its outputs. So the problem isn't visibility into the model, it's that when an output is wrong they can't follow it back to the cause. Because the cause is almost always upstream in the data, such as in an unclear prompt, a stale record, a broken pipeline or a schema change. So the state of observability is that enterprises can see how an agent behaved, but not why."
"The good news is that organizations already possess much of the foundation needed," Pretam Mondal, Lead Product Manager, Observability and Service Management, OpenText counters. "The challenge is extending those existing observability investments to support AI workloads rather than creating separate monitoring silos. As AI adoption accelerates, enterprises will increasingly look for platforms that provide end-to-end visibility across traditional applications and AI systems."
"The industry has made substantial progress in collecting AI telemetry," Sekhar Sarukkai, Co-founder and CEO, ChatSee.ai, agrees. "For LLM applications, enterprises can increasingly capture prompts, responses, retrieval, model calls, evaluations, latency, token consumption, and traces. Agent frameworks are making execution paths and tool calls increasingly visible as well."
However, he adds: "Even an organization with extraordinary telemetry, infrastructure, AI expertise, and compute resources can incur enormous costs reconstructing what autonomous agents did and why. Most enterprises will not have those resources."