Skip to main content

Exploring AI Observability – Part 7: Tools

Pete Goldin
APMdigest

By now, this series has made the case that AI observability is its own discipline, with its own data, its own metrics, and arguably its own set of stakeholders. That naturally raises a very practical question for any enterprise: does any of this mean starting over? Or can the observability, APM, and AIOps platforms already running in production actually do the job? Part 7 covers the tools needed for AI observability.

It's not an abstract question. Most enterprises have spent years, and often significant budget, building out mature observability and APM practices for their traditional applications. The idea of standing up a completely parallel stack just for AI — with its own agents, its own dashboards, its own on-call rotation — is unappealing on cost grounds alone, and it risks creating siloed, uncorrelated telemetry that earlier parts of this series identified as one of the biggest challenges enterprises already face.

APM Already Does More Than Most Teams Realize

Download the e-book: The ultimate APM playbook - Master challenges and implement best practices

"When it comes to LLM observability, most teams don't realize how much existing APM platforms can already do," says Srinivasa Raghavan Santhanam, Director of Product Management at ManageEngine. "They can trace API calls to LLM providers, monitor response times, and log prompt-response pairs, all of which are natural extensions of what mature APM tools offer. While the instrumentation layer may look different, the analysis is similar." He drew a sharper line for agentic systems, though: "For agentic AI, current platforms are good at monitoring infrastructure and logging actions. The main area that needs new features is autonomous investigation and remediation, where observability tools must not only report but also connect information, make decisions, and take action."

"Transport, storage, correlation, alerting, and incident workflow all carry over, and every major APM vendor now maps OpenTelemetry GenAI conventions into its own UI," adds Hong Wang, Co-Founder and CEO of Akuity.

"Traditional APM tools are more capable here than people give them credit for," Cameron Daniel, CTO of Megaport, agrees. "They can model agentic workflows reasonably well, though quality varies a lot between vendors. For an organization still working out its AI strategy, that’s actually the sensible place to start: You don’t need to rip out your existing stack and bet on a purpose-built AI observability platform before you even know what you need to measure. Start with the tools you’ve already got, then augment via APIs as your usage matures; it’s a lower-risk path than jumping straight to a new and rapidly evolving category."

APM vendors have added LLM tracing, evaluations and cost dashboards, so the agent and application layers are served. What they do not do is connect an AI failure to the data underneath it, Rohit Choudhary, CEO of Acceldata, observes.

Joseph Voyles, Principal AI & Emerging Technology at PwC US, explains, "A lot of the platforms that already exist plug right into the stack you'd use for AI systems. You don't need to throw any of that out. AIOps platforms still have a role to play here, just like they do for traditional applications." His advice to enterprises evaluating what to buy next was to resist the instinct to rebuild from scratch: "Most of what you need already exists, and organizations shouldn't be starting from scratch. They should figure out where the actual gaps are in managing AI system quality and bring in new tools or standards to close those specific gaps. The observability playbook doesn't need a rewrite; you build on it."

"They're a necessary foundation, not a sufficient one," Douglas James, VP, Skylar AI at ScienceLogic, counters. "APM and AIOps still do real work at the infrastructure layer, and that layer doesn't vanish because the workload is AI. What many of them were not built to do is judge the semantic layer: whether an answer is grounded, whether an agent chose the right action, or whether an output is safe. Extend the observability foundation you trust with a purpose-built evaluation layer. Don't assume the old stack covers the new risk."

Full Stack Visibility

Download a complimentary copy: 2026 Gartner® Magic Quadrant™ for Observability Platforms

"Current observability, APM, and AIOps products can support AI Observability when they provide explicit capabilities for the AI layer," explains Bernd Greifeneder, CTO and Founder at Dynatrace. "That means observing AI infrastructure, models, conversations, and the inputs and outputs that shape system behavior. LLM-based and AI systems introduce models, orchestration, and agents into the application environment, so monitoring the AI layer in isolation is not enough. Full-stack visibility is essential to understand how AI behavior relates to the application, infrastructure, and end-user experience."

Camden Swita, Head of AI at New Relic, concurs: "Organizations already can get much of the operational foundation they need from current products. Telemetry collection, distributed tracing, incident response, and application observability remain essential because AI doesn’t replace traditional software. It becomes part of it. In practice, that means full-stack observability with AI monitoring built in is the best path forward, so teams can understand AI behavior in the context of the broader application and infrastructure environment. The real opportunity is to build on proven observability practices and expand them so they can also explain how AI systems behave in production, whether that's an LLM generating responses or an AI agent orchestrating a multi-step workflow."

The Big Difference

Even among the more optimistic answers, there was broad agreement on where traditional tools hit a wall: they can tell you a system ran, but not whether it was right.

"Current observability, APM, and AIOps products provide essential infrastructure context, but they cannot deliver a complete AI observability practice on their own," Aaron Winston, Head of Content, Communications, and Analyst Relations at Arize AI, asserts. "They were built primarily to help people diagnose failures in software and infrastructure by analyzing latency, errors, resource saturation, service dependencies, logs, and distributed traces. Those signals explain whether the system ran correctly. They do not determine whether an answer was accurate, an agent followed a valid path, a tool call was appropriate, or a task produced the intended outcome." His conclusion was that AI observability needs "a distinct, future-facing layer for prompt and model lineage, retrieval and tool visibility, session-level trajectories, evaluations, datasets, full-system experimentation, cost per successful outcome, and programmatic improvement workflows" — a list that has essentially no equivalent in a pre-AI observability stack.

According to Sekhar Sarukkai, Co-Founder and CEO, ChatSee.ai, enterprises ultimately need the following questions answered for agentic AI: What was the user or business trying to accomplish? What behavioral failure occurred? Is this a known failure? Where else is it happening? How important is it? What business outcome did it affect? What did we learn the last time it occurred? What treatment should be applied? And did that treatment work? Traditional observability can provide much of the evidence required to answer those questions. It should therefore be viewed as an important input into AI observability rather than as the complete solution.

However, he adds, "The architectural mistake would be to conclude that because we can trace every model call and tool invocation, we therefore understand whether the agent behaved correctly."

Layering AI Evaluation on Top

If LLM applications stretch the limits of existing tools, agentic systems break them in a different way, because the factor that needs observing isn't a request-response pair anymore — it's a chain of decisions and actions.

Omer Bhutta, Head of AI and ML at Secure.com pinpoints what existing observability and APM platforms cannot do out of the box: evaluate semantic quality. Scoring outputs for hallucination, groundedness, or task success requires evaluation infrastructure, including datasets, graders, and human-review workflows, that traditional APM was never designed for. For agentic systems, most existing tools also lack native concepts for trajectories, tool-call semantics, and multi-agent coordination. His recommended pattern echoes what most other experts described: "keep the existing platform as the system of record for operational telemetry and layer specialized AI evaluation capabilities on top, rather than standing up an entirely parallel stack."

The Retrieval Layer

One specific gap came up that's easy to overlook amid the broader conversation about models and agents: retrieval. As more enterprise AI applications lean on retrieval-augmented generation, the search and vector infrastructure feeding the model becomes just as critical to observe as the model itself — and it's an area most legacy tools have essentially no visibility into.

"Legacy APM and observability frameworks were designed for deterministic systems where a specific input predictably produces a specific output. Generative AI applications and agentic workflows invert that model," explains Bianca Lewis, Executive Director of the OpenSearch Software Foundation. "Traditional observability tools capture top-level latency or HTTP errors, and they often lack the retrieval-aware depth needed to inspect how context is fetched, how prompts are structured, or how non-deterministic decision loops execute." Her conclusion pointed to a layer of infrastructure that rarely shows up in observability conversations at all: "As organizations deploy retrieval-augmented generation pipelines and multi-step autonomous agents, the underlying search and analytics engines that index, retrieve, and monitor data become just as foundational to success as the models themselves. To achieve true AI observability, enterprises must move beyond generic metric collection toward a unified search and analytics data layer that seamlessly joins structured telemetry with unstructured vector context and execution state." It's a reminder that "the AI layer" isn't just the model call — it's everything upstream of it that decided what the model would even see.

Don't Just Rename the Dashboard

The most useful caution from the experts wasn't about whether existing tools have a role — everyone agreed they do. It was about the temptation to declare victory too early, simply by relabeling what's already there.

"Conventional tools require additional instrumentation and semantics to understand LLM and agent behavior. A generic API trace may show that a model was called, but not which system instruction applied, what documents were retrieved, which tools were available, why one was selected, or whether the resulting action violated policy," Steve Giguere, Principal AI Security Advocate with the Office of the CTO at Check Point, advises. "Existing platforms can therefore be part of the answer, but simply renaming an APM dashboard 'AI observability' is insufficient. The important question is whether the platform can represent and correlate the model, context, identity, policy, tool, action, and outcome, and not merely the API call."

Extend, Not Rip-and-Replace

"Organizations should think of AI observability as an extension of existing observability practices rather than an entirely separate discipline," advises Jeremy Rossbach, Chief Technical Evangelist, Network Observability, Broadcom.

"I'd start there rather than rip anything out, because a lot of the foundation carries straight over, in that OpenTelemetry tracing, metrics pipelines, dashboards, and alerting all apply perfectly well to agents, and the vendors are moving quickly to add GenAI-specific views for prompts, tokens, and tool calls," says Ian Livingstone, Co-Founder and CEO, Keycard.

"Enterprises should extend existing observability practices rather than discard them. But AI-specific telemetry and governance-aware context will be required," says Abhijit Kakhandiki, SVP and GM, Digital Business Automation, BMC.

You can reuse the existing telemetry backbone and then layer on dedicated AI capabilities to enable evaluation of metrics such as model performance and drift, token and cost analytics, and prompt/response clustering and evaluation, according to Dan Albright, Global AdvisoryX Leader, DXC Technology.

Modern observability platforms already provide much of the foundation required for AI observability, delivering the visibility, context, and analytical capabilities needed to monitor complex AI-powered systems, says Pretam Mondal, Lead Product Manager, Observability and Service Management, OpenText. Rather than deploying separate AI monitoring solutions, many organizations are extending existing observability platforms with AI-specific telemetry, evaluation frameworks and workflow visibility. This approach allows enterprises to leverage existing investments while maintaining a unified operational view across applications, infrastructure, and AI services.

Juraci Paixão Kröhling, CEO of Ollygarden, explains: "The generative AI semantic conventions are the industry's attempt at a shared vocabulary for model calls, token usage, and agent steps. They are still moving, and I would not describe them as settled, but they are the only credible path to telemetry from different services and different model providers lining up well enough to be compared. The same vendor-neutral pipeline that unifies traditional signals carries AI signals across heterogeneous providers without lock-in, which matters when the model in use changes every few months. That matters because the tools meant to reason about system behavior, increasingly AI tools themselves, depend on consistent and well-structured data to do so. A platform observability team at a large consumer-electronics company told me that standardizing on semantic conventions became urgent for them because their AI tooling could not reason about the system without it. Standardization is a prerequisite here, not a refinement."

However, Gou Rao, Co-Founder and CEO of NeuBird AI, warns: "Building a DIY wrapper or bolting an LLM dashboard onto a legacy platform gets you partial metrics, but it fails to deliver production-grade visibility. Real AI observability requires specialized platforms, often leveraging specialized observability agents capable of understanding agentic workflows, reasoning chains and complex multi-model interactions."

As a final thought, Aneesh Nathani, VP – Data and AI Service Line, Cybage Software, says you could go either way: "There’s a legitimate case for turnkey AI observability platforms that ship with LLM-aware tracing, evaluation frameworks, and prompt analytics out of the box. There’s an equally legitimate case for extending your existing observability platform with AI telemetry, keeping one pane of glass and one data pipeline. Which path is right depends on how central AI is to your stack and how fast your current platform is adding AI-aware capabilities. What doesn’t work is pretending standard APM covers it. You cannot diagnose a hallucination from a flame graph."

Go to: Exploring AI Observability – Part 8: Features

Pete Goldin is Editor and Publisher of APMdigest

Hot Topics

Exploring AI Observability – Part 7: Tools

Pete Goldin
APMdigest

By now, this series has made the case that AI observability is its own discipline, with its own data, its own metrics, and arguably its own set of stakeholders. That naturally raises a very practical question for any enterprise: does any of this mean starting over? Or can the observability, APM, and AIOps platforms already running in production actually do the job? Part 7 covers the tools needed for AI observability.

It's not an abstract question. Most enterprises have spent years, and often significant budget, building out mature observability and APM practices for their traditional applications. The idea of standing up a completely parallel stack just for AI — with its own agents, its own dashboards, its own on-call rotation — is unappealing on cost grounds alone, and it risks creating siloed, uncorrelated telemetry that earlier parts of this series identified as one of the biggest challenges enterprises already face.

APM Already Does More Than Most Teams Realize

Download the e-book: The ultimate APM playbook - Master challenges and implement best practices

"When it comes to LLM observability, most teams don't realize how much existing APM platforms can already do," says Srinivasa Raghavan Santhanam, Director of Product Management at ManageEngine. "They can trace API calls to LLM providers, monitor response times, and log prompt-response pairs, all of which are natural extensions of what mature APM tools offer. While the instrumentation layer may look different, the analysis is similar." He drew a sharper line for agentic systems, though: "For agentic AI, current platforms are good at monitoring infrastructure and logging actions. The main area that needs new features is autonomous investigation and remediation, where observability tools must not only report but also connect information, make decisions, and take action."

"Transport, storage, correlation, alerting, and incident workflow all carry over, and every major APM vendor now maps OpenTelemetry GenAI conventions into its own UI," adds Hong Wang, Co-Founder and CEO of Akuity.

"Traditional APM tools are more capable here than people give them credit for," Cameron Daniel, CTO of Megaport, agrees. "They can model agentic workflows reasonably well, though quality varies a lot between vendors. For an organization still working out its AI strategy, that’s actually the sensible place to start: You don’t need to rip out your existing stack and bet on a purpose-built AI observability platform before you even know what you need to measure. Start with the tools you’ve already got, then augment via APIs as your usage matures; it’s a lower-risk path than jumping straight to a new and rapidly evolving category."

APM vendors have added LLM tracing, evaluations and cost dashboards, so the agent and application layers are served. What they do not do is connect an AI failure to the data underneath it, Rohit Choudhary, CEO of Acceldata, observes.

Joseph Voyles, Principal AI & Emerging Technology at PwC US, explains, "A lot of the platforms that already exist plug right into the stack you'd use for AI systems. You don't need to throw any of that out. AIOps platforms still have a role to play here, just like they do for traditional applications." His advice to enterprises evaluating what to buy next was to resist the instinct to rebuild from scratch: "Most of what you need already exists, and organizations shouldn't be starting from scratch. They should figure out where the actual gaps are in managing AI system quality and bring in new tools or standards to close those specific gaps. The observability playbook doesn't need a rewrite; you build on it."

"They're a necessary foundation, not a sufficient one," Douglas James, VP, Skylar AI at ScienceLogic, counters. "APM and AIOps still do real work at the infrastructure layer, and that layer doesn't vanish because the workload is AI. What many of them were not built to do is judge the semantic layer: whether an answer is grounded, whether an agent chose the right action, or whether an output is safe. Extend the observability foundation you trust with a purpose-built evaluation layer. Don't assume the old stack covers the new risk."

Full Stack Visibility

Download a complimentary copy: 2026 Gartner® Magic Quadrant™ for Observability Platforms

"Current observability, APM, and AIOps products can support AI Observability when they provide explicit capabilities for the AI layer," explains Bernd Greifeneder, CTO and Founder at Dynatrace. "That means observing AI infrastructure, models, conversations, and the inputs and outputs that shape system behavior. LLM-based and AI systems introduce models, orchestration, and agents into the application environment, so monitoring the AI layer in isolation is not enough. Full-stack visibility is essential to understand how AI behavior relates to the application, infrastructure, and end-user experience."

Camden Swita, Head of AI at New Relic, concurs: "Organizations already can get much of the operational foundation they need from current products. Telemetry collection, distributed tracing, incident response, and application observability remain essential because AI doesn’t replace traditional software. It becomes part of it. In practice, that means full-stack observability with AI monitoring built in is the best path forward, so teams can understand AI behavior in the context of the broader application and infrastructure environment. The real opportunity is to build on proven observability practices and expand them so they can also explain how AI systems behave in production, whether that's an LLM generating responses or an AI agent orchestrating a multi-step workflow."

The Big Difference

Even among the more optimistic answers, there was broad agreement on where traditional tools hit a wall: they can tell you a system ran, but not whether it was right.

"Current observability, APM, and AIOps products provide essential infrastructure context, but they cannot deliver a complete AI observability practice on their own," Aaron Winston, Head of Content, Communications, and Analyst Relations at Arize AI, asserts. "They were built primarily to help people diagnose failures in software and infrastructure by analyzing latency, errors, resource saturation, service dependencies, logs, and distributed traces. Those signals explain whether the system ran correctly. They do not determine whether an answer was accurate, an agent followed a valid path, a tool call was appropriate, or a task produced the intended outcome." His conclusion was that AI observability needs "a distinct, future-facing layer for prompt and model lineage, retrieval and tool visibility, session-level trajectories, evaluations, datasets, full-system experimentation, cost per successful outcome, and programmatic improvement workflows" — a list that has essentially no equivalent in a pre-AI observability stack.

According to Sekhar Sarukkai, Co-Founder and CEO, ChatSee.ai, enterprises ultimately need the following questions answered for agentic AI: What was the user or business trying to accomplish? What behavioral failure occurred? Is this a known failure? Where else is it happening? How important is it? What business outcome did it affect? What did we learn the last time it occurred? What treatment should be applied? And did that treatment work? Traditional observability can provide much of the evidence required to answer those questions. It should therefore be viewed as an important input into AI observability rather than as the complete solution.

However, he adds, "The architectural mistake would be to conclude that because we can trace every model call and tool invocation, we therefore understand whether the agent behaved correctly."

Layering AI Evaluation on Top

If LLM applications stretch the limits of existing tools, agentic systems break them in a different way, because the factor that needs observing isn't a request-response pair anymore — it's a chain of decisions and actions.

Omer Bhutta, Head of AI and ML at Secure.com pinpoints what existing observability and APM platforms cannot do out of the box: evaluate semantic quality. Scoring outputs for hallucination, groundedness, or task success requires evaluation infrastructure, including datasets, graders, and human-review workflows, that traditional APM was never designed for. For agentic systems, most existing tools also lack native concepts for trajectories, tool-call semantics, and multi-agent coordination. His recommended pattern echoes what most other experts described: "keep the existing platform as the system of record for operational telemetry and layer specialized AI evaluation capabilities on top, rather than standing up an entirely parallel stack."

The Retrieval Layer

One specific gap came up that's easy to overlook amid the broader conversation about models and agents: retrieval. As more enterprise AI applications lean on retrieval-augmented generation, the search and vector infrastructure feeding the model becomes just as critical to observe as the model itself — and it's an area most legacy tools have essentially no visibility into.

"Legacy APM and observability frameworks were designed for deterministic systems where a specific input predictably produces a specific output. Generative AI applications and agentic workflows invert that model," explains Bianca Lewis, Executive Director of the OpenSearch Software Foundation. "Traditional observability tools capture top-level latency or HTTP errors, and they often lack the retrieval-aware depth needed to inspect how context is fetched, how prompts are structured, or how non-deterministic decision loops execute." Her conclusion pointed to a layer of infrastructure that rarely shows up in observability conversations at all: "As organizations deploy retrieval-augmented generation pipelines and multi-step autonomous agents, the underlying search and analytics engines that index, retrieve, and monitor data become just as foundational to success as the models themselves. To achieve true AI observability, enterprises must move beyond generic metric collection toward a unified search and analytics data layer that seamlessly joins structured telemetry with unstructured vector context and execution state." It's a reminder that "the AI layer" isn't just the model call — it's everything upstream of it that decided what the model would even see.

Don't Just Rename the Dashboard

The most useful caution from the experts wasn't about whether existing tools have a role — everyone agreed they do. It was about the temptation to declare victory too early, simply by relabeling what's already there.

"Conventional tools require additional instrumentation and semantics to understand LLM and agent behavior. A generic API trace may show that a model was called, but not which system instruction applied, what documents were retrieved, which tools were available, why one was selected, or whether the resulting action violated policy," Steve Giguere, Principal AI Security Advocate with the Office of the CTO at Check Point, advises. "Existing platforms can therefore be part of the answer, but simply renaming an APM dashboard 'AI observability' is insufficient. The important question is whether the platform can represent and correlate the model, context, identity, policy, tool, action, and outcome, and not merely the API call."

Extend, Not Rip-and-Replace

"Organizations should think of AI observability as an extension of existing observability practices rather than an entirely separate discipline," advises Jeremy Rossbach, Chief Technical Evangelist, Network Observability, Broadcom.

"I'd start there rather than rip anything out, because a lot of the foundation carries straight over, in that OpenTelemetry tracing, metrics pipelines, dashboards, and alerting all apply perfectly well to agents, and the vendors are moving quickly to add GenAI-specific views for prompts, tokens, and tool calls," says Ian Livingstone, Co-Founder and CEO, Keycard.

"Enterprises should extend existing observability practices rather than discard them. But AI-specific telemetry and governance-aware context will be required," says Abhijit Kakhandiki, SVP and GM, Digital Business Automation, BMC.

You can reuse the existing telemetry backbone and then layer on dedicated AI capabilities to enable evaluation of metrics such as model performance and drift, token and cost analytics, and prompt/response clustering and evaluation, according to Dan Albright, Global AdvisoryX Leader, DXC Technology.

Modern observability platforms already provide much of the foundation required for AI observability, delivering the visibility, context, and analytical capabilities needed to monitor complex AI-powered systems, says Pretam Mondal, Lead Product Manager, Observability and Service Management, OpenText. Rather than deploying separate AI monitoring solutions, many organizations are extending existing observability platforms with AI-specific telemetry, evaluation frameworks and workflow visibility. This approach allows enterprises to leverage existing investments while maintaining a unified operational view across applications, infrastructure, and AI services.

Juraci Paixão Kröhling, CEO of Ollygarden, explains: "The generative AI semantic conventions are the industry's attempt at a shared vocabulary for model calls, token usage, and agent steps. They are still moving, and I would not describe them as settled, but they are the only credible path to telemetry from different services and different model providers lining up well enough to be compared. The same vendor-neutral pipeline that unifies traditional signals carries AI signals across heterogeneous providers without lock-in, which matters when the model in use changes every few months. That matters because the tools meant to reason about system behavior, increasingly AI tools themselves, depend on consistent and well-structured data to do so. A platform observability team at a large consumer-electronics company told me that standardizing on semantic conventions became urgent for them because their AI tooling could not reason about the system without it. Standardization is a prerequisite here, not a refinement."

However, Gou Rao, Co-Founder and CEO of NeuBird AI, warns: "Building a DIY wrapper or bolting an LLM dashboard onto a legacy platform gets you partial metrics, but it fails to deliver production-grade visibility. Real AI observability requires specialized platforms, often leveraging specialized observability agents capable of understanding agentic workflows, reasoning chains and complex multi-model interactions."

As a final thought, Aneesh Nathani, VP – Data and AI Service Line, Cybage Software, says you could go either way: "There’s a legitimate case for turnkey AI observability platforms that ship with LLM-aware tracing, evaluation frameworks, and prompt analytics out of the box. There’s an equally legitimate case for extending your existing observability platform with AI telemetry, keeping one pane of glass and one data pipeline. Which path is right depends on how central AI is to your stack and how fast your current platform is adding AI-aware capabilities. What doesn’t work is pretending standard APM covers it. You cannot diagnose a hallucination from a flame graph."

Go to: Exploring AI Observability – Part 8: Features

Pete Goldin is Editor and Publisher of APMdigest

Hot Topics