Distributed Tracing
Distributed tracing is a technique that follows the path a single user request takes across multiple services and databases, linking it with a shared trace ID into one record. It is used in architectures where processing crosses boundaries, such as microservices, to pinpoint which segment produced slowness or errors.
Overview
Distributed tracing is a technique that gathers the processing path of a single request, from the frontend through backend services and databases, into one record. A unique trace ID is issued at the request's entry point and passed along each time a service is called, so that records of work born in separate processes can later be assembled into a single whole. It is the concept you care about when comparing per-service logs leaves the story cut off at the boundaries, and when, faced with a report that something is "slow," you want to name which operation in which service caused it, at the granularity of an operation rather than a minute.
Linking one request across services into a single record
Distributed tracing is a technique for tracking how a request received by an application flows from the frontend device through backend services and databases (as of September 2026; Source: Distributed Tracing). With a monolith that runs entirely on one server, you can understand the flow of processing by looking at that process's stack. In microservices, however, a single order flow calls separate services one after another, such as authentication, inventory, payment, and notification, so to track where the time went you need a mechanism that identifies the same request across process boundaries. That mechanism is the trace ID; the single record assembled with it as the key is a Trace, and each unit of work within it is a Span.
Carrying the trace ID and stacking spans in parent-child relationships
The core of the mechanism is continuously passing the identifier to the next service. When a request reaches the entry point, a unique trace ID and the first parent span are created, and when calling another service, the trace ID and parent span ID are embedded in HTTP headers and sent along. The receiving side extracts them, attaches the same trace ID to its own span, and passes it further downstream (as of September 2026; Source: APM Terms and Concepts). This handoff is called trace context propagation.
| Element | What it represents | Where it originates |
|---|---|---|
| Trace ID | An identifier that stays constant throughout one request | Issued once at the entry service |
| Span | One unit of work such as an API call or query, with its duration and errors | Each service that performed the work |
| Parent span ID | The parent-child relationship of which work called this one | Passed downstream in headers at call time |
| Metadata | Supplementary information such as customer ID or region | Attached to each span |
Because the spans share the same trace ID, the collecting side can reassemble fragments arriving from separate processes into a single whole. Drawing the parent span on top and the child spans below in order of occurrence gives you a flame graph, where long segments and the location of errors can be read at a glance.
Logs alone leave the story cut off at the boundaries
It is natural to ask whether the same thing could be followed with logs. Logs come from any layer, application, infrastructure, or network, and record events line by line with timestamps. However, because each service writes them out independently, there is no key from the outset that ties a request's line to a line from another service. Without it, incident investigation becomes the work of staring at the logs of 5 services side by side by timestamp while a human guesses which lines belong to the same order. Distributed tracing, in exchange for targeting only the application layer, shows the path the request takes across boundaries as-is and points to where it got stuck at the granularity of an operation (as of September 2026; Source: Distributed Tracing). From there, when digging into why it was slow, you read the logs tied to that trace; that division of roles is the practical pattern.
The sampling compromise: you cannot keep everything
In high-traffic systems, millions of spans can be generated per minute (as of September 2026; Source: Distributed Tracing). Because storing all of them continuously is not realistic on cost grounds, sampling is performed to keep only a portion. There are two representative ways to decide. The approach that decides probabilistically at the request's entry point whether to keep or discard is called head-based sampling, and it is also the default mechanism in Datadog (as of September 2026; Source: Ingestion Mechanisms). It is lightweight, but in exchange it may have discarded from the start the one request that later turns out to matter. The approach that first collects everything and then selects and keeps the slow ones or those containing errors misses less, but in exchange requires a vessel that holds everything until the selection is made. What is easily lost with the probabilistic-at-entry approach is "the rare bad one," while the shape of the average request is sufficiently preserved by either approach. The combination Datadog recommends, "send 100% and then keep with retention filters," is explained in the Trace entry. The mechanisms on the side of narrowing ingestion are summarized in the Ingestion Controls entry.
The standard is OpenTelemetry, and what receives it is APM
If the propagation format and instrumentation API differ from tool to tool, a service instrumented with one tool cannot read the trace ID attached by another service, and the trace breaks there. To prevent this fragmentation, OpenTelemetry is the standard that defines the instrumentation API and data format in a vendor-independent way. Datadog also presents it as the option when you want vendor-neutral instrumentation (as of September 2026; Source: Application Instrumentation). For the types of receiving endpoints and differences in available features, see the OpenTelemetry explainer. What realizes distributed tracing in Datadog is APM, and if you use Single Step Instrumentation when installing the Agent, trace collection for supported frameworks begins without changing code. As needed, you can layer on manual SDK configuration and custom spans written in code. The sequence with the least rework is to first make the entire path visible with automatic instrumentation, and only after the slow segment is identified add your own spans to that service alone.