LLM Observability
LLM Observability records a single request to an application with an embedded LLM as a trace, tracks the inputs and outputs, tokens, latency, and errors of each model call, and turns response quality into numbers through evaluations. The only thing billed is the LLM span, which represents a model call.
What it does
It turns a single request received by an LLM application such as a chatbot into one trace, and records the model calls within it (LLM spans) and the surrounding tool calls and preprocessing (spans such as Tool / Task / Retrieval) in a parent-child relationship. Each span carries input and output, token count, latency, and errors, and evaluations (Evaluations) score the quality and safety of the response and tie the score to the same trace.
When to use it
When you want to isolate whether a complaint that answers are slow is about the model's response time or the search (Retrieval) that precedes it. When you want to track a surge in token consumption by model / by provider. When you want to continuously confirm, on production traffic, that no personal information is mixed into inputs and outputs and that the answer is not in a language different from the user's.
An everyday analogy
It is the minutes of a business negotiation conducted through an interpreter. For each utterance, it records what was told to the interpreter (the LLM), what came back, how many seconds the exchange took, and how much the interpreter's fee (tokens) was. On top of that, a third party (evaluation) later grades the accuracy of the translation and whether there were any inappropriate remarks, and writes the grade on the same line of the minutes.
The Bottom Line - Only Model Calls Are Counted
The first thing to grasp is how billing is counted. The billable unit is the LLM span only, that is, the span representing a model call; the surrounding tool call (tool), workflow (workflow), and search (retrieval) spans are not billed (as of August 2026; source: FAQ on Datadog Pricing). Even for an application that calls the model once per request and performs five searches and tool calls around it, what is counted is 1 span. Since breaking the surrounding processing into fine-grained spans does not change the price, there is no reason to hesitate about making traces more detailed for the sake of isolation. One note on naming as well. As of September 2026, the official documentation page for this product is published under the heading Agent Observability, while the SKU name on the pricing page remains LLM Observability (source: Agent Observability). This article uses the pricing page's name throughout.
What Is the Same as APM Spans, and What Is Different
An APMspan represents a single unit of processing within a service, and the kind of processing, such as an HTTP request or a database query, is determined automatically. LLM Observability spans are the same in that they form a single trace through parent-child relationships, but the span has an attribute called span kind, which represents what role that processing plays within the LLM application. The official documentation lists seven kinds (as of September 2026; source: Terms and Concepts).
| Kind | What It Represents | Can It Be the Root of a Trace? | Billable? |
|---|---|---|---|
| LLM | A single call to a model | Yes | Billable |
| Workflow | A fixed sequence of steps including an LLM call and the processing before and after it | Yes | Not billable |
| Agent | A series of decisions and actions made by an autonomous agent | Yes | Not billable |
| Tool | Processing that calls an external program or service with arguments generated by the LLM | No | Not billable |
| Task | Standalone processing that does not call an external service (preprocessing, etc.) | No | Not billable |
| Embedding | A call to a model or function that returns an embedding vector | No | Not billable |
| Retrieval | Processing that retrieves data from an external knowledge base | No | Not billable |
Whereas APM answers which processing in which service is slow, what these kinds answer is whether the slow part is the model's response, the search before it, or the tool call. Using the same vessel, the trace, the unit of the question changes from the service to the stages of the LLM application.
A Single Trace Comes in Three Shapes
How a trace is composed splits into three shapes depending on the kind of the root span: an inference trace consisting of a single LLM span, a workflow trace with a Workflow span as the root nesting LLM / Task / Tool / Embedding / Retrieval spans, and an agent trace with an Agent span as the root that further nests Workflows (as of September 2026; source: Terms and Concepts).
- Inference trace: a single LLM span. This is where you get the record of input and output, tracking of token usage / error rate / latency, and the breakdown by model / by provider
- Workflow trace: root = Workflow. Bundles a fixed sequence of steps into one trace, such as receiving a URL, fetching the page (Tool), formatting it (Task), and summarizing it (LLM)
- Agent trace: root = Agent. Records processing whose steps change according to the agent's decisions, such as a chatbot answering inquiries, together with the nested Workflows
For the first step, an inference trace is enough. Just recording a model call as a single span gives you input and output, tokens, and latency together. Once you find that the cause of slow answers lies outside the model, you add the surrounding processing as Tool or Retrieval spans and grow it into a workflow trace. Since only LLM spans are billable, growing it this way does not increase the price.
Evaluations That Turn Quality into Numbers Have Three Entry Points
Latency and errors can be understood by counting, but the quality of an answer cannot be counted. Evaluations (Evaluations) are the mechanism that fills this hole: they tie evaluation results to individual spans and let you trace directly from the span to the input and output that led to that evaluation. There are three entry points (as of September 2026; source: Terms and Concepts).
- Managed evaluations: enable evaluations provided by Datadog. The list as of September 2026 is Language Mismatch, which detects responses in a language different from the user's input, and Sensitive Data Scanning, which detects whether inputs and outputs contain confidential or regulated information (source: Managed Evaluations)
- Custom evaluations: send in results graded by your own criteria and tie them to spans
- External framework integration: ingest results from evaluation frameworks such as NeMo
Separately from evaluations, there are also Patterns, which automatically bundles the inputs and outputs of production traffic by topic, and Insights, which surfaces outliers in latency and error rate over the past week along the axes of span name / workflow kind / topic (source: Agent Observability). The value of this layer is being able to grasp what users are asking and where quality dropped without reading individual logs.
Three Ingestion Routes - SDK / HTTP API / OpenTelemetry
There are three instrumentation routes. The recommended one is the SDK, which supports Python 3.7 or later / Node.js 16 or later / Java 8 or later, and all of them offer both automatic and manual instrumentation. For languages and environments where the SDK cannot be used, there is a route that sends spans directly through the HTTP API (as of September 2026; source: Instrumentation).
| Route | Support | Suited Situation |
|---|---|---|
| SDK (automatic instrumentation) | Python / Node.js / Java. Captures frameworks and providers such as OpenAI / LangChain / AWS Bedrock / Anthropic without code changes | When you use a supported framework and want to see inputs, outputs, and tokens first |
| SDK (manual instrumentation) | Same 3 languages. You create spans and assign kinds such as Workflow / Tool / Retrieval yourself | When you want the processing before and after the model call in a single trace as well |
| HTTP API | Send spans directly as JSON | Languages without an SDK, or when you want to consolidate sending onto your own platform |
| OpenTelemetry | Send traces that follow the GenAI semantic conventions of OpenTelemetry 1.37 or later, or the OpenInference conventions, by configuring the OTLP exporter, without the Datadog SDK | Organizations already instrumented with OpenTelemetry |
One point for organizations already instrumented with OpenTelemetry. Under the instrumentation page there is a dedicated page for OpenTelemetry instrumentation, which explains that traces following the GenAI semantic conventions of OpenTelemetry 1.37 or later or the OpenInference conventions can be sent with only OTLP exporter configuration, without using the Datadog SDK (as of September 2026; source: OpenTelemetry Instrumentation). Check the same page for the details of the supported scope, such as evaluations and prompt tracking.
Pricing Is Determined Solely by the Number of LLM Spans
Pricing has two tiers: a monthly allowance and additional charges for LLM spans beyond it. Free is free up to 40,000 LLM spans per month; Pro includes 100,000 LLM spans in its monthly fee, and the excess is added at a unit price per 10,000 LLM spans. Both the Pro monthly fee and the additional unit price differ across the three forms of annual / monthly / on-demand (plan breakdown as of August 2026; source: Datadog Pricing). For the actual amounts, check the same page.
| Plan / SKU | Included Volume | Form of billing | Main caveats |
|---|---|---|---|
| Free | 40,000 LLM spans per month; 15-day retention | Free | Exceeding the allowance assumes moving to Pro |
| Pro (monthly) | 100,000 LLM spans per month; 15-day retention | Monthly fee (annual / monthly / on-demand) | Anything beyond the allowance is added via the additional SKU |
| Additional LLM spans | Per 10,000 LLM spans | Metered (unit price differs across the three forms) | Determined solely by LLM span count, not by host count or request count |
For an application that calls the model 300,000 times a month, the 200,000 spans beyond Pro's 100,000 allowance are the additional portion, and the monthly charge reads as "the Pro monthly fee + 20 × (the additional unit price per 10,000 LLM spans)." The number of searches and tool calls does not enter the calculation. Conversely, with a design that internally calls the model 3 times per question, LLM spans triple. It is a separate SKU from APM's host billing; check its place within the overall pricing in how to read the pricing structure.
Is APM Enough, or Do You Need LLM Observability?
It is not the case that every application calling an LLM necessarily needs this product. If what you want to know stops at how many times a month you call the model's API, how many milliseconds the response took, and how many times it failed, APM traces show it as external API calls. LLM Observability is needed when the question goes inside the call.
| What You Want to Know | Sufficient Tool | Reason |
|---|---|---|
| Model API call count, response time, failure count | APM | Visible as external API call spans |
| How it answered which prompt (input and output) | LLM Observability | LLM spans carry input and output |
| How many tokens were used in which processing | LLM Observability | Per-span token usage and breakdown by model |
| Continuously measuring language mismatches in responses and leakage of confidential information | LLM Observability | Evaluations (Evaluations) tie scores to spans |
| Tracking steps that change with an agent's decisions, stage by stage | LLM Observability | Stages are separated by the Agent / Workflow / Tool kinds |
A rule of thumb for the decision is to picture the incident investigation. If investigating slowness ends with checking the model's response time, APM; if investigating wrong answers has to be done on production inputs and outputs, LLM Observability. Free's 40,000 LLM spans per month is a trial allowance wide enough to make this decision from actual measurement. It is complementary to Continuous Profiler, which looks at the same application from the CPU and memory side: time waiting on the model does not show up in the profiler, and conversely the CPU load of embedding computation does not show up in LLM Observability.
Things to watch out for
- This article is based on the official documentation as of September 2026. Span kinds, the list of managed evaluations, and supported languages may change, so confirm the official documentation before implementing.
- This article explains how billing works. For current unit prices and plans, refer to Datadog's official pricing page.
- As of September 2026, the product name in the official documentation has been renamed to Agent Observability (docs: https://docs.datadoghq.com/llm_observability/). This article uses LLM Observability, the SKU name on the pricing page, throughout.