Skip to main content

Watching an AI Agent Think: Instrumenting a Slack Assistant with OpenTelemetry

Nitish Mane
Philo

We run an internal AI assistant that answers engineering questions in Slack. Someone mentions it in a thread, and a few seconds or a few minutes later it replies with an answer it worked out by running commands, reading files and calling internal tools.

Each question kicks off what we call a turn. A turn is a small state machine of LLM steps. One step summarizes the thread. Another decides which tools and capabilities to load. Then the agent itself runs inside a sandbox with shell access and a handful of tool servers. A final reviewer step reads the draft answer and either ships it, rewrites it, or sends it back for another pass.

We could already audit a single turn after the fact. Every reply carries a short lookup token in its footer, and a slash command pulls up the full receipt for that turn. What we could not do was answer questions about the whole fleet. How long do turns take? What do they cost? Which tools does the agent reach for? Where does the time go inside a single answer?

Those are APM questions. So we treated the agent like any other service and instrumented it with OpenTelemetry, end to end. This post covers what we built, the design choices that turned out to matter, and the things that surprised us.

One Trace per Turn

The core idea is simple. Every turn becomes one trace, and the shape of that trace mirrors the state machine:

  • A root span for the whole turn, including time spent waiting for a free slot. Cost, token counts and (optionally) short excerpts of the question and answer live here.
  •  One child span per step call. If the reviewer kicks the answer back, the re-run shows up as another step span, so retries are visible instead of hidden inside one long bar.
  • Under each step, a span for every model call. Under the agent step, a span for every tool call. When the agent spawns a subagent, that subagent’s tool calls nest under the call that started it.

We followed the OpenTelemetry GenAI semantic conventions for span names and attributes (chat, invoke_agent, execute_tool, gen_ai.*). That cost us nothing and means any backend that understands those conventions can make sense of the data.

Alongside traces we emit a small set of metrics: turn count, turn duration and turn cost by status and model; step duration and step errors; token usage; tool calls by tool name; kick-backs by kind; and counters for the PII scrubber. A few gauges cover concurrency, queue depth and the health of the collector.
The export path has two hops. The engine sends to a collector sidecar on the same host. That sidecar forwards over HTTPS to a gateway collector in our Kubernetes cluster, which writes traces to an object-storage backed trace store and metrics to our existing time series database. Dashboards sit on top of both.

Everything ships switched off. Telemetry and excerpts each have their own flag, so merging the code changed nothing until we chose to turn it on, one engine at a time.

Designing for an Agent That Can Be Talked Into Things

Most services don’t take instructions from their users. An LLM agent does, and anything it reads can try to steer it. That changed how we designed the telemetry.

The engine never holds the credential that matters. There are two tokens. The engine uses an ingest token to talk to the local sidecar. Only the sidecar holds the gateway token for the cluster. The engine process is the part exposed to prompt injection, so it simply never sees the key to the outside. The operator CLI also refuses to start the collector if a literal gateway token shows up in the committed config file.

No network port between engine and collector. They talk over a Unix socket, and the socket path is a constant in the code rather than an environment variable. A config edit, or an agent that has been talked into editing config, can’t redirect exports somewhere else. Engines mount the socket directory read-only.

The agent’s own event stream is untrusted input. During review we realized that code the agent runs inside its sandbox could plausibly write fake events into the same stream the engine parses to build spans. We didn’t reproduce it, but we hardened against it anyway. Tool names are accepted only from the first initialization event. Tool call IDs have to match the expected format. Unknown tools are recorded as other. Tool descriptions on spans come from fixed text, never from model output. The broader sandbox issue was filed and handled separately.

Privacy is tiered, and excerpts fail closed. A turn counts as public only when the chat platform reports a public channel and nothing flags it private. Private turns drop channel, user and request identifiers and never get excerpts, but they still report timing, tokens, cost and tool names. For public turns, a second opt-in adds the question and answer as span attributes. Each one goes through a PII scrubber first, over the length cap plus a 64 character margin so an email address split at the boundary still gets caught, and is then cut to 512 characters. A scrub failure, a two second timeout, a disabled scrubber or an open circuit breaker all mean no excerpt. Never an unscrubbed one. Telemetry scrubbing has its own circuit breaker, so it can never starve the scrubber that protects actual Slack replies.

Metric labels go through an allowlist. A trace can carry a thread timestamp or a lookup token because every trace stands alone. A metric label can’t, because every distinct value creates a new time series. We learned to be strict about this early rather than clean up a cardinality explosion later.

Sampling stays at 100%. The lookup token in each reply footer is also stored on the root span, so you can paste it into the dashboard and land on the trace. Any sampling ratio below one would make some of those links dead. The trade-off is that the trace store becomes a fairly detailed activity record per developer, which is why we asked for sign-off on 30 day retention instead of assuming it.

What One Turn Actually Looks Like

The first real surprise came from a single trace. Someone asked the assistant which deployments were unhealthy. The agent ran seven shell commands and answered. The token breakdown:

Token type               Count
Uncached input        9,130
Cache read               690,343
Cache write              165,050
Total input                 864,523
Output                       4,151

That turn cost just under a dollar. The lesson is that agent turns are dominated by cached input. A dashboard that only shows uncached input tokens understates real volume by about 100 times. In one trace, the agent step’s uncached input was 8 tokens, which tells you nothing on its own.

Cost is computed per turn from token counts and a pricing table, with separate rates for cache reads and writes. A turn uses several models across its steps, and for the metric label we attribute the whole cost to the model with the most uncached tokens. That’s a known distortion. We already caught one turn labeled with the smaller model when the expensive agent step ran on the larger one. Per model cost is on the list.

Things That Broke Along the Way

Every turn looked like a PII failure. The excerpt function returned either a string or undefined, and undefined meant four different things: excerpts off, empty input, zero length cap, or a real scrub failure. The caller treated all of them as a drop, so every public turn was flagged as a dropped excerpt and the panel was useless. We replaced it with an explicit result type (excerpt, not attempted, dropped) so the caller can’t mix them up. Using undefined as an error channel throws away information you'll want later.

A DNS record that existed but didn't resolve. We created the record, and for about 15 minutes the host still couldn’t resolve it. The first lookup had happened before the record existed, and the resolver cached that negative answer for the zone’s 900 second minimum. Nothing was wrong except our patience.

Docker creates missing bind mount sources as root. A rebuild ran before the socket directory existed, so Docker created it owned by root. The collector’s start script correctly refused a socket directory with the wrong owner. Correct behavior, confusing afternoon.

The PII scrubber has limits, and we said so. In live traffic it tagged the word “weather” as a person’s name, and it missed a two letter city abbreviation while catching the full city name. Our docs now state plainly that PII the scrubber doesn’t detect stays in the excerpt. That’s the honest contract, and it’s why excerpts are a separate opt-in.

The Dashboard Got Better When It Got Smaller

Our first dashboard had 44 panels running 51 queries. At internal traffic levels, 24 of those queries came back empty. It looked thorough and told us very little.

The version we kept has 16 panels, and every query returns live data. A row of headline numbers (turns, p95 latency, error rate, cost, tokens, cost per turn). Traffic and latency over time. Tokens and cost by model. Step latency and tool calls by tool. And a recent turns table showing each question, answer, tools used, token breakdown, cost and duration, with a link straight to the trace.

One small lesson for low volume services: short window rates draw nothing when you get a few requests an hour. We switched the over time panels to 15 minute increase bars, and suddenly the charts had shape.

What We Haven't Solved Yet

We'd rather list these than pretend they don’t exist:

  • The GenAI convention histograms use the SDK’s default buckets, which are sized for milliseconds. Their quantiles mean nothing for LLM calls that take seconds. Only our custom turn and step histograms (1 second to 20 minutes) are useful.
  • We count tool calls but don’t yet measure tool errors or tool latency.
  • Cache tokens appear on spans but not in the token metric.
  • A brand new counter series starts at 1, so rate calculations miss its first increment and undercount on a quiet engine.
  • Export failures are silent in the engine log, and the collector health check stays green even when the gateway rejects the token. For now, the collector logs are the source of truth.

If Youre About to Do This Yourself

A few things we'd tell any team instrumenting an LLM agent:

  1. Model the trace on the agent’s control flow. If your agent has steps, retries and tool calls, those should be spans you can see, not log lines you have to grep.
  2. Treat the agent as a hostile neighbor to your telemetry pipeline. Keep credentials out of its process and don’t trust anything it can write.
  3. Track cached tokens. For agents they are most of the volume and a large share of the cost.
  4. Make privacy the default and content an explicit, fail closed opt-in.
  5. Link traces to whatever audit trail you already have. Being able to paste a token from a chat reply and land on the exact trace changed how people debugged.
  6. Start with a small dashboard that returns real data. Add panels when someone asks a question the dashboard can’t answer.

None of this needed a new category of tooling. OpenTelemetry, a collector and the backends we already ran were enough. What changed was treating the agent as a production service, with the same questions about latency, errors and cost, plus a few new ones about trust.

Nitish Mane is a Staff Infrastructure Engineer at Philo

Hot Topics

The Latest

"AI readiness" has quickly become one of the defining priorities of the modern enterprise. Business decision makers across virtually every industry are working to build roadmaps, solidify strategies, and set expectations around technological investment and adoption. However, at the very same time these leaders are laying their plans, the state of AI continues to change and shift beneath their feet. This raises a fundamental question about the notion of AI readiness: what is it we're actually getting ready for? ...

Genpact and HFS Research recently found that 92% of senior executives at Global 2000 companies believe agentic AI will fundamentally change how work gets done. Yet 85% of them also recognize enterprise debt as a factor limiting their AI value. To move beyond pilots and deliver tangible value, CIOs need to address enterprise debt — the accumulated operational drag from outdated technology, fragmented data, inefficient processes, and talent gaps ...

Proactive is going to become the default expectation, not a nice-to-have. Predicting problems before they happen and fixing them ahead of time will become table stakes, especially for smaller teams and MSPs who cannot afford a big NOC staff ...

The final two installments of the series, Exploring AI Observability, cover predictions from the experts about where AI observability is heading ... In the next few years, tracking your AI won't just be a nice-to-have tool, it will be a survival requirement for businesses ...

Cloud absolutely changed IT for the better. But after more than a decade of using it, companies have also learned something else. Where an application runs matters, and there isn't one environment that makes sense for everything ...

Automation often starts with the people closest to an immediate need. A developer may schedule a script with cron or Amazon EventBridge Scheduler; a business team may adopt an RPA bot, low-code platform, or agent. Many tools stay useful within one domain. Some become consequential across organizational boundaries because other systems and teams depend on them or business outcomes rely on their timing and recovery. Those cases may call for broader coordination ...

The conventional wisdom in our industry says that AI demands a new operating model and specific data tailored for it. I disagree. AI is changing the rules, but not in the way people think ...

Most organizations invest heavily in security tools, policies, and controls. Yet despite these efforts, one reality persists: employees often find ways around them. This isn't always malicious. In many cases, it’s a response to friction, urgency, or misalignment between how work actually happens and how systems are designed to protect it. When employees bypass security, it creates blind spots. Sensitive data may move through unmonitored channels, access controls may be weakened, and risks that security teams intended to mitigate reappear in new forms ...

Enterprise AI projects are often justified with a familiar metric: hours saved. A team automates a manual process, employees spend fewer hours reviewing documents or entering data, and the productivity gain becomes the headline. Time savings matter, but they capture only part of the financial impact AI can have inside an organization ... A more useful way to evaluate enterprise AI is to examine the financial decisions and risks surrounding the workflows it supports ...

We run an internal AI assistant that answers engineering questions in Slack ... Each question kicks off what we call a turn ... We could already audit a single turn after the fact. Every reply carries a short lookup token in its footer, and a slash command pulls up the full receipt for that turn. What we could not do was answer questions about the whole fleet. How long do turns take? What do they cost? Which tools does the agent reach for? Where does the time go inside a single answer? Those are APM questions. So we treated the agent like any other service and instrumented it with OpenTelemetry, end to end. This post covers what we built, the design choices that turned out to matter, and the things that surprised us ...

Watching an AI Agent Think: Instrumenting a Slack Assistant with OpenTelemetry

Nitish Mane
Philo

We run an internal AI assistant that answers engineering questions in Slack. Someone mentions it in a thread, and a few seconds or a few minutes later it replies with an answer it worked out by running commands, reading files and calling internal tools.

Each question kicks off what we call a turn. A turn is a small state machine of LLM steps. One step summarizes the thread. Another decides which tools and capabilities to load. Then the agent itself runs inside a sandbox with shell access and a handful of tool servers. A final reviewer step reads the draft answer and either ships it, rewrites it, or sends it back for another pass.

We could already audit a single turn after the fact. Every reply carries a short lookup token in its footer, and a slash command pulls up the full receipt for that turn. What we could not do was answer questions about the whole fleet. How long do turns take? What do they cost? Which tools does the agent reach for? Where does the time go inside a single answer?

Those are APM questions. So we treated the agent like any other service and instrumented it with OpenTelemetry, end to end. This post covers what we built, the design choices that turned out to matter, and the things that surprised us.

One Trace per Turn

The core idea is simple. Every turn becomes one trace, and the shape of that trace mirrors the state machine:

  • A root span for the whole turn, including time spent waiting for a free slot. Cost, token counts and (optionally) short excerpts of the question and answer live here.
  •  One child span per step call. If the reviewer kicks the answer back, the re-run shows up as another step span, so retries are visible instead of hidden inside one long bar.
  • Under each step, a span for every model call. Under the agent step, a span for every tool call. When the agent spawns a subagent, that subagent’s tool calls nest under the call that started it.

We followed the OpenTelemetry GenAI semantic conventions for span names and attributes (chat, invoke_agent, execute_tool, gen_ai.*). That cost us nothing and means any backend that understands those conventions can make sense of the data.

Alongside traces we emit a small set of metrics: turn count, turn duration and turn cost by status and model; step duration and step errors; token usage; tool calls by tool name; kick-backs by kind; and counters for the PII scrubber. A few gauges cover concurrency, queue depth and the health of the collector.
The export path has two hops. The engine sends to a collector sidecar on the same host. That sidecar forwards over HTTPS to a gateway collector in our Kubernetes cluster, which writes traces to an object-storage backed trace store and metrics to our existing time series database. Dashboards sit on top of both.

Everything ships switched off. Telemetry and excerpts each have their own flag, so merging the code changed nothing until we chose to turn it on, one engine at a time.

Designing for an Agent That Can Be Talked Into Things

Most services don’t take instructions from their users. An LLM agent does, and anything it reads can try to steer it. That changed how we designed the telemetry.

The engine never holds the credential that matters. There are two tokens. The engine uses an ingest token to talk to the local sidecar. Only the sidecar holds the gateway token for the cluster. The engine process is the part exposed to prompt injection, so it simply never sees the key to the outside. The operator CLI also refuses to start the collector if a literal gateway token shows up in the committed config file.

No network port between engine and collector. They talk over a Unix socket, and the socket path is a constant in the code rather than an environment variable. A config edit, or an agent that has been talked into editing config, can’t redirect exports somewhere else. Engines mount the socket directory read-only.

The agent’s own event stream is untrusted input. During review we realized that code the agent runs inside its sandbox could plausibly write fake events into the same stream the engine parses to build spans. We didn’t reproduce it, but we hardened against it anyway. Tool names are accepted only from the first initialization event. Tool call IDs have to match the expected format. Unknown tools are recorded as other. Tool descriptions on spans come from fixed text, never from model output. The broader sandbox issue was filed and handled separately.

Privacy is tiered, and excerpts fail closed. A turn counts as public only when the chat platform reports a public channel and nothing flags it private. Private turns drop channel, user and request identifiers and never get excerpts, but they still report timing, tokens, cost and tool names. For public turns, a second opt-in adds the question and answer as span attributes. Each one goes through a PII scrubber first, over the length cap plus a 64 character margin so an email address split at the boundary still gets caught, and is then cut to 512 characters. A scrub failure, a two second timeout, a disabled scrubber or an open circuit breaker all mean no excerpt. Never an unscrubbed one. Telemetry scrubbing has its own circuit breaker, so it can never starve the scrubber that protects actual Slack replies.

Metric labels go through an allowlist. A trace can carry a thread timestamp or a lookup token because every trace stands alone. A metric label can’t, because every distinct value creates a new time series. We learned to be strict about this early rather than clean up a cardinality explosion later.

Sampling stays at 100%. The lookup token in each reply footer is also stored on the root span, so you can paste it into the dashboard and land on the trace. Any sampling ratio below one would make some of those links dead. The trade-off is that the trace store becomes a fairly detailed activity record per developer, which is why we asked for sign-off on 30 day retention instead of assuming it.

What One Turn Actually Looks Like

The first real surprise came from a single trace. Someone asked the assistant which deployments were unhealthy. The agent ran seven shell commands and answered. The token breakdown:

Token type               Count
Uncached input        9,130
Cache read               690,343
Cache write              165,050
Total input                 864,523
Output                       4,151

That turn cost just under a dollar. The lesson is that agent turns are dominated by cached input. A dashboard that only shows uncached input tokens understates real volume by about 100 times. In one trace, the agent step’s uncached input was 8 tokens, which tells you nothing on its own.

Cost is computed per turn from token counts and a pricing table, with separate rates for cache reads and writes. A turn uses several models across its steps, and for the metric label we attribute the whole cost to the model with the most uncached tokens. That’s a known distortion. We already caught one turn labeled with the smaller model when the expensive agent step ran on the larger one. Per model cost is on the list.

Things That Broke Along the Way

Every turn looked like a PII failure. The excerpt function returned either a string or undefined, and undefined meant four different things: excerpts off, empty input, zero length cap, or a real scrub failure. The caller treated all of them as a drop, so every public turn was flagged as a dropped excerpt and the panel was useless. We replaced it with an explicit result type (excerpt, not attempted, dropped) so the caller can’t mix them up. Using undefined as an error channel throws away information you'll want later.

A DNS record that existed but didn't resolve. We created the record, and for about 15 minutes the host still couldn’t resolve it. The first lookup had happened before the record existed, and the resolver cached that negative answer for the zone’s 900 second minimum. Nothing was wrong except our patience.

Docker creates missing bind mount sources as root. A rebuild ran before the socket directory existed, so Docker created it owned by root. The collector’s start script correctly refused a socket directory with the wrong owner. Correct behavior, confusing afternoon.

The PII scrubber has limits, and we said so. In live traffic it tagged the word “weather” as a person’s name, and it missed a two letter city abbreviation while catching the full city name. Our docs now state plainly that PII the scrubber doesn’t detect stays in the excerpt. That’s the honest contract, and it’s why excerpts are a separate opt-in.

The Dashboard Got Better When It Got Smaller

Our first dashboard had 44 panels running 51 queries. At internal traffic levels, 24 of those queries came back empty. It looked thorough and told us very little.

The version we kept has 16 panels, and every query returns live data. A row of headline numbers (turns, p95 latency, error rate, cost, tokens, cost per turn). Traffic and latency over time. Tokens and cost by model. Step latency and tool calls by tool. And a recent turns table showing each question, answer, tools used, token breakdown, cost and duration, with a link straight to the trace.

One small lesson for low volume services: short window rates draw nothing when you get a few requests an hour. We switched the over time panels to 15 minute increase bars, and suddenly the charts had shape.

What We Haven't Solved Yet

We'd rather list these than pretend they don’t exist:

  • The GenAI convention histograms use the SDK’s default buckets, which are sized for milliseconds. Their quantiles mean nothing for LLM calls that take seconds. Only our custom turn and step histograms (1 second to 20 minutes) are useful.
  • We count tool calls but don’t yet measure tool errors or tool latency.
  • Cache tokens appear on spans but not in the token metric.
  • A brand new counter series starts at 1, so rate calculations miss its first increment and undercount on a quiet engine.
  • Export failures are silent in the engine log, and the collector health check stays green even when the gateway rejects the token. For now, the collector logs are the source of truth.

If Youre About to Do This Yourself

A few things we'd tell any team instrumenting an LLM agent:

  1. Model the trace on the agent’s control flow. If your agent has steps, retries and tool calls, those should be spans you can see, not log lines you have to grep.
  2. Treat the agent as a hostile neighbor to your telemetry pipeline. Keep credentials out of its process and don’t trust anything it can write.
  3. Track cached tokens. For agents they are most of the volume and a large share of the cost.
  4. Make privacy the default and content an explicit, fail closed opt-in.
  5. Link traces to whatever audit trail you already have. Being able to paste a token from a chat reply and land on the exact trace changed how people debugged.
  6. Start with a small dashboard that returns real data. Add panels when someone asks a question the dashboard can’t answer.

None of this needed a new category of tooling. OpenTelemetry, a collector and the backends we already ran were enough. What changed was treating the agent as a production service, with the same questions about latency, errors and cost, plus a few new ones about trust.

Nitish Mane is a Staff Infrastructure Engineer at Philo

Hot Topics

The Latest

"AI readiness" has quickly become one of the defining priorities of the modern enterprise. Business decision makers across virtually every industry are working to build roadmaps, solidify strategies, and set expectations around technological investment and adoption. However, at the very same time these leaders are laying their plans, the state of AI continues to change and shift beneath their feet. This raises a fundamental question about the notion of AI readiness: what is it we're actually getting ready for? ...

Genpact and HFS Research recently found that 92% of senior executives at Global 2000 companies believe agentic AI will fundamentally change how work gets done. Yet 85% of them also recognize enterprise debt as a factor limiting their AI value. To move beyond pilots and deliver tangible value, CIOs need to address enterprise debt — the accumulated operational drag from outdated technology, fragmented data, inefficient processes, and talent gaps ...

Proactive is going to become the default expectation, not a nice-to-have. Predicting problems before they happen and fixing them ahead of time will become table stakes, especially for smaller teams and MSPs who cannot afford a big NOC staff ...

The final two installments of the series, Exploring AI Observability, cover predictions from the experts about where AI observability is heading ... In the next few years, tracking your AI won't just be a nice-to-have tool, it will be a survival requirement for businesses ...

Cloud absolutely changed IT for the better. But after more than a decade of using it, companies have also learned something else. Where an application runs matters, and there isn't one environment that makes sense for everything ...

Automation often starts with the people closest to an immediate need. A developer may schedule a script with cron or Amazon EventBridge Scheduler; a business team may adopt an RPA bot, low-code platform, or agent. Many tools stay useful within one domain. Some become consequential across organizational boundaries because other systems and teams depend on them or business outcomes rely on their timing and recovery. Those cases may call for broader coordination ...

The conventional wisdom in our industry says that AI demands a new operating model and specific data tailored for it. I disagree. AI is changing the rules, but not in the way people think ...

Most organizations invest heavily in security tools, policies, and controls. Yet despite these efforts, one reality persists: employees often find ways around them. This isn't always malicious. In many cases, it’s a response to friction, urgency, or misalignment between how work actually happens and how systems are designed to protect it. When employees bypass security, it creates blind spots. Sensitive data may move through unmonitored channels, access controls may be weakened, and risks that security teams intended to mitigate reappear in new forms ...

Enterprise AI projects are often justified with a familiar metric: hours saved. A team automates a manual process, employees spend fewer hours reviewing documents or entering data, and the productivity gain becomes the headline. Time savings matter, but they capture only part of the financial impact AI can have inside an organization ... A more useful way to evaluate enterprise AI is to examine the financial decisions and risks surrounding the workflows it supports ...

We run an internal AI assistant that answers engineering questions in Slack ... Each question kicks off what we call a turn ... We could already audit a single turn after the fact. Every reply carries a short lookup token in its footer, and a slash command pulls up the full receipt for that turn. What we could not do was answer questions about the whole fleet. How long do turns take? What do they cost? Which tools does the agent reach for? Where does the time go inside a single answer? Those are APM questions. So we treated the agent like any other service and instrumented it with OpenTelemetry, end to end. This post covers what we built, the design choices that turned out to matter, and the things that surprised us ...