7

Mins

Multi-Agent Observability: Tracing, Debugging & Evaluation in Production

When a team of AI agents gets an answer wrong, you need to find the step that slipped. This guide shows how: trace every request, measure what matters, and test your agents before and after launch, so they can't slip the same way twice.

Observability-Multi-Agents

CONTENT

Key Takeaways

  • Give each request one trace ID, and pass it through every agent, tool and queue.

  • Record every step: what each agent saw, decided and did.

  • Watch for silent failures. A wrong answer often comes with no error, so check the path, not just the answer: right agent, right tools, no wasted steps.

  • Test before launch, then grade a sample of live answers. Turn every failure into a new test.

  • Guard risky actions in code, not just in prompts, and mask personal data before you store traces.

Why observability is important in multi-agent systems

Here's a made-up support ticket: "Your bot said my refund was approved. A week later, still no money." Somewhere in a team of AI agents, one step failed, and nothing flagged it. You open the logs: thousands of lines from a dozen agents. Which ones belong to this customer?

In plain words: Observability means you can see inside a running system and explain what it did. With a team of AI agents, that gets hard fast.

One request, many moving parts

We'll reuse the made-up support team from our guardrails article. A lead agent, the orchestrator, passes work to three helper agents, or sub-agents: triage, research and refunds.

  • One message can lead to many model calls and tool calls.

  • A mistake can start in one agent and show up two agents later.

  • Anthropic found agents use about 4 times the tokens of a chat. Multi-agent systems use about 15 times.

why-agents-are-harder-to-watch

Figure 1. Tokens per task vs. a chat. Anthropic data, June 2025.

Why pay for 15 times the tokens?

Because for the right job, the extra work pays off.

  • Better answers: in Anthropic's tests, a lead Claude Opus 4 agent with Claude Sonnet 4 helpers beat a single Claude Opus 4 agent by 90.2% on its internal research test.

  • Tokens do the work: Anthropic wrote that multi-agent systems "work mainly because they help spend enough tokens to solve the problem."

  • Right jobs only: they do best on big tasks that can be split up and run at the same time, need more information than one model can hold, or use many complex tools. They fit poorly when every agent needs the same context or the agents depend heavily on each other. Anthropic gives most coding tasks as an example.

  • Worth the cost: Anthropic says the task's value must be "high enough to pay for the increased performance."

💡 Why observability matters here: You're paying for 15 times the tokens, so you need to see where they go. Tracing shows which agent spent them, on which step, and whether the answer got better. Without it, you can't tell if the extra cost is paying off.

Same question, different path

  • Agents don't follow the same steps every time. Anthropic's engineers wrote that this "makes debugging harder."

  • Rerunning the request may not show the bug.

  • Logs show single events, not which request each belongs to.

Words you'll see in this article

Term

What it means

LLM

Large language model, the AI behind each agent.

Token

A small chunk of text, often part of a word.

MCP server

An add-on that gives agents extra tools.

Orchestrator

The lead agent. It splits a job and hands the pieces to helper agents.

Sub-agent

A helper agent that does one part of the job.

Tool call

When an agent uses an outside function, like "look up order".

Trace

The full record of one request, from start to finish.

Span

One step inside a trace, like a single model call or tool call.

Logging vs. monitoring vs. tracing vs. evaluation

In plain words: These are four ways to watch your system. Each answers a different question.

Method

The question it answers

Example

Logging

What happened at this moment?

"Refund tool failed at 14:02"

Monitoring

Is the system healthy right now?

Errors jumped to 6% since 2 p.m.

Tracing

What path did this one request take?

Message, triage, research, refund, reply

Evaluation

Was the answer actually good?

A grader scores the reply 1 out of 5

Tracing is the backbone: logs, scores and numbers all link back to traces.

four-ways-to-watch-a-system

Figure 2. Tracing links the other three. Conceptual diagram.

Anatomy of a trace

In plain words: A trace is the full story of one request. A span is one step in that story. Think of a shop receipt, with one line per item.

How spans fit together

  • The first span covers the whole request. It's called the root span.

  • Spans sit inside other spans, and each knows its parent.

  • Expect five kinds of span: orchestrator decisions, sub-agent calls, LLM calls, tool calls and retrievals. A retrieval is a search through documents.

What every span should record

Inputs and outputs, tokens, latency (how long it took), cost and any error.

anatomy-of-a-trace

Figure 3. Our refund request as a trace.

Tracing agent decisions

In plain words: A good trace shows what each agent did, and the reason it gave. Think of a plane's flight recorder.

  • Routing: record which helper got the work, and the reason given.

  • Tool choice: record the tools on offer, the one picked, and what it sent.

  • Loops and overruns: watch for an agent repeating the same step, or not stopping. Anthropic saw its agents "continuing when they already had sufficient results".

  • Context: everything the agent could see, like its instructions and documents. If it ignores a fact, check it was there.

Tracing across agent boundaries

In plain words: Every agent, tool and server should tag its work with the same trace ID. Think of a parcel tracking number that every depot scans.

  • Handoffs: when one agent passes work on, it passes the trace ID too. This is called context propagation.

  • The format: a short text called traceparent, from the W3C web standards group. It holds the trace ID and the caller's step ID.

  • Parallel agents: helpers working at once each get their own span, under the same parent.

  • Queues: a queue holds jobs to run later. Put the trace ID in each job, or the story splits in two.

  • MCP servers: each MCP request has a set place for the trace ID, a field called params._meta.

  • Other APIs: an API is how programs talk to each other. Send the traceparent with each call (see the code).

What it looks like in code

A short Python sketch, assuming OpenTelemetry is set up as in its Python guide:

from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer("support.team")

with tracer.start_as_current_span("invoke_agent refund"):
    with tracer.start_as_current_span("execute_tool issue_refund"):
        headers = {}
        TraceContextTextMapPropagator().inject(headers)
from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer("support.team")

with tracer.start_as_current_span("invoke_agent refund"):
    with tracer.start_as_current_span("execute_tool issue_refund"):
        headers = {}
        TraceContextTextMapPropagator().inject(headers)
from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer("support.team")

with tracer.start_as_current_span("invoke_agent refund"):
    with tracer.start_as_current_span("execute_tool issue_refund"):
        headers = {}
        TraceContextTextMapPropagator().inject(headers)

The tool span sits inside the agent span. inject adds the traceparent to the headers, so the payment API can join the trace.


tracing

Figure 4. One trace ID everywhere.

Key metrics to monitor

In plain words: Metrics are numbers that show how all requests are doing, like a car dashboard.

Area

What to watch

Latency (speed)

How long requests, agents and tools take. Watch the slowest 5% (p95), not just the average.

Cost

Tokens and money per request and per agent.

Reliability

Errors, retries, timeouts and stuck loops.

Quality

Tasks finished, cases passed to a person (escalation rate), grader scores, user ratings.

Guardrails

How often safety checks block, hide data or ask a person. See our guardrails article.

Offline evaluation

In plain words: Offline evaluation means testing before launch, on cases you pick. Think of a driving test. Our evaluation article goes deeper.

Build a test set from real cases

  • A test set is a list of example requests and expected results.

  • Start small. Anthropic began with "about 20 queries representing real usage patterns."

  • Test each agent alone, then the whole team together.

Pick the right grader

A grader decides whether an answer passed.

Grader

Best for

Watch out for

Code checks

One right answer, like a refund amount

Can't judge open-ended writing

LLM-as-judge

Open-ended answers, like customer replies

Can favour longer answers, answer order or its own writing

Human review

High stakes, like money, health or legal

Slow and costly

LLM-as-judge means one AI model grades another's answer against a checklist. In a 2023 study, strong judges like GPT-4 agreed with people over 80% of the time. That's about as often as people agree with each other.

People still catch what graders miss. Anthropic's human testers saw early agents pick "SEO-optimized content farms" over academic PDFs. These are low-quality sites built for search.

Test again after every change

  • Rerun every test after any prompt, model or tool change. This catches regressions: things that used to work but now don't.

  • Small changes spread between agents. Anthropic found "small changes to the lead agent can unpredictably change how subagents behave."

Online evaluation (in production)

In plain words: Online evaluation means checking real answers after launch. Real users find cases your tests missed.

  • Score a sample: grade a small share of answers. Tools like Langfuse let you pick which.

  • Use feedback: attach thumbs-downs and reopened tickets to the trace.

  • Watch for drift: drift is quality slowly getting worse, with no errors. Model updates or new kinds of questions can cause it. In one 2023 study, GPT-4's accuracy on a maths task fell from 84% to 51% between two versions. Chart the score weekly and alert on drops.

online-evaluation

Figure 5. Drift that error counts miss.

Debugging with traces

In plain words: Start from the complaint, find its trace, and walk down to the step that broke.

Back to our missing refund (Figure 3):

  1. Find the trace. Search by ticket number or time.

  2. Scan the timeline. Look for errors, slow steps or repeats.

  3. Spot the break. The refund tool timed out. Yet the next step told the customer the refund was approved.

  4. Find the root cause, the real reason. Two things went wrong. The agent's instructions never said what to do if the tool fails. And nothing in the code stopped the reply from saying "approved" when the refund didn't go through.

  5. Fix both, then add a test. Tell the agent what to do when a tool fails. Then add a code check: the reply can only say "approved" if the refund tool returned success. Prompts guide the agent, but code should guard the money. Finally, add this case to your test set so the bug can't come back.

result = issue_refund(order_id, amount)

if result.status != "success":
    reply = "We couldn't process your refund yet. A team member will follow up."
    escalate_to_human(ticket_id)
else:
    reply = refund_agent.write_reply(result)
result = issue_refund(order_id, amount)

if result.status != "success":
    reply = "We couldn't process your refund yet. A team member will follow up."
    escalate_to_human(ticket_id)
else:
    reply = refund_agent.write_reply(result)
result = issue_refund(order_id, amount)

if result.status != "success":
    reply = "We couldn't process your refund yet. A team member will follow up."
    escalate_to_human(ticket_id)
else:
    reply = refund_agent.write_reply(result)

The agent only writes the "approved" message after the code confirms the refund worked. If it failed, the customer gets an honest reply and a person takes over.

This is a guardrail: a rule in code that the agent can't talk its way around. See our guardrails article for more.

debugging-with-traces

Figure 6. Every failure becomes a new test.

Privacy and retention

In plain words: Traces contain whatever your users typed, like names and emails. Treat them like customer data.

  • Only record text if you need it: OpenTelemetry's guidelines say not to record prompts and answers by default. This text is "likely to contain sensitive information".

  • Mask before you store: masking means swapping personal data for a placeholder, like [EMAIL]. Langfuse can mask data before it leaves your app.

  • Set a retention period: how long you keep traces. Europe's privacy law, GDPR, says to keep personal data "for no longer than is necessary". LangSmith's cloud keeps traces 14 or 180 days.

  • Limit access: only people who need raw traces should see them.

privacy-and-retention

Figure 7. Masking one span.

Platforms for agent observability

In plain words: You need a standard way to record traces, and a place to view them. Some platforms let you self-host, meaning run them on your own servers.

The standard: OpenTelemetry

  • OpenTelemetry (OTel) is a free, open standard for recording traces, metrics and logs.

  • It works with many tools, so you're not tied to one company.

  • To view the data, you send it to a platform.

Three popular platforms

Platform

What it covers

Free option

Paid plans from

Langfuse

Tracing, prompt management and evaluation

Self-host free, or a free cloud plan: 50k units (traces, steps or scores) a month

$29 a month

Arize Phoenix

Tracing, evaluation and experiments

Self-host free

Arize AX (managed): see Arize's pricing page

LangSmith

Tracing, evaluation and monitoring, from the LangChain team

Free plan: 1 seat, 5k traces a month included

$39 per seat a month

Table 1. All three work with OpenTelemetry. Prices checked November 20, 2026.

  • Langfuse uses an MIT license for its core. Some extras need a license key.

  • Phoenix is called open source by Arize. Its Elastic License 2.0 allows self-hosting, but not offering it to others as a service.

  • LangSmith is mainly a cloud service. Self-hosting is an Enterprise add-on.

How to choose

  • Need data on your own servers? Pick Langfuse or Phoenix.

  • Built with LangChain or LangGraph (popular agent toolkits)? Pick LangSmith.

  • Whatever you pick, check it supports OpenTelemetry, so you can switch later.

Closing checklist

In plain words: Start small, then tick off the full list.

Where to start

  1. Add OpenTelemetry to your agents.

  2. Pass one trace ID through every handoff.

  3. Turn on masking for personal data.

  4. Send traces to a platform to view them.

  5. Write 20 test cases from real requests, including a few where a tool fails.

  6. Set one alert, like refund tool errors.

  7. Grade a sample of live answers weekly.

Before launch

checklist

Figure 8. Pre-launch checklist.

If you do one thing this week, give every request one trace ID. Without it, the missing refund is a mystery. With it, you can point to the step that broke, fix it, and add a test so it can’t break the same way again.

Frequently asked questions

Frequently asked questions

Q1. What is observability in multi-agent AI systems?

Observability helps track agent decisions, tool calls, errors and performance across an entire multi-agent workflow.

Q2. How do you trace and debug multi-agent systems?

Use distributed tracing with a shared trace ID to track agent handoffs, tool calls and failures across the workflow.

Q3. What are the best tools for AI agent observability?

Langfuse, LangSmith and Arize Phoenix provide tracing, monitoring and evaluation. OpenTelemetry enables standardized instrumentation.

Q4. What metrics should you monitor in multi-agent systems?

Monitor latency, token usage, cost, error rates, task completion, agent decisions and evaluation scores.

Q5. How do you evaluate multi-agent AI systems in production?

Use offline test cases, automated evaluations, LLM-as-judge scoring, human reviews and continuous monitoring of live agent performance.

SHARE