Ingestion Controls
The mechanism that decides which of the traces your applications send to Datadog are actually ingested. Even if you reduce the ingested volume, APM metrics are still calculated from all traces.
Overview
Ingestion Controls refers to the mechanism that decides which of the traces your applications send to Datadog are ingested, and to the page that gives an overview of that state. Ingestion volume and its breakdown are listed per service, and colors let you tell apart the share dropped by sampling, the share discarded by rate limiting at the source, and the share lost because the Agent ran short of resources. There are two ways to decide rates: telling Datadog a monthly volume target, or setting rates manually per resource. If the conditions are met, changes take effect without redeploying.
What gets reduced is the volume sent, not the volume aggregated
Ingestion Controls is the mechanism that decides which of the traces your applications send to Datadog are actually ingested, together with the page that gives an overview of that state. As the name suggests, it is a feature for reducing volume, but only the incoming traces are reduced; APM metrics such as latency and error rate are always calculated from all traces. This two-layer design is why the numbers on your graphs do not shrink when you change your sampling settings.The top of the page shows the ingested volume for the last hour and the monthly usage estimated from it. The per-service table is built from the usage metrics datadog.estimated_usage.apm.ingested_spans and datadog.estimated_usage.apm.ingested_bytes, and it can be broken down by service name, environment, and ingestion reason tags. Think of it as the place where you can confirm, with numbers rather than guesses, which service is sending how much.Because the volume of ingested traces feeds directly into cost as APM usage, the official documentation also positions this page as the entry point for managing your ingestion volume budget (as of August 2026). The reason this mechanism exists is that it lets you decide, while looking at the numbers, the balance between how much visibility to keep and where to start cutting.
The colors in the breakdown show why traces were dropped
Each service row shows a breakdown, colored by where the sent traces ended up. What matters is that the colors tell you not just that the volume decreased, but why it decreased.
| Category | What happened | Main cause |
|---|---|---|
| Ingested | Stored on the Datadog side | None (the expected outcome) |
| Not retained | Dropped by sampling | A rate calculated automatically by the Agent, or a sampling rule you configured |
| Dropped by rate limiting | The sending library stopped sending | A default limit of 100 traces per second. Enabled automatically when you set a rate manually |
| Dropped due to resource shortage | The Agent could not keep up and discarded them | The Agent's CPU and memory limits. Needs attention because traces end up with gaps |
Only the last, red category is an accident rather than the result of configuration. Traces with gaps can no longer be followed end to end, so you resolve this by increasing the CPU and memory allocated to the Agent.
There are two ways to decide rates
Two methods are available for adjusting the ingestion volume of each service. One is adaptive sampling: you tell Datadog a monthly ingestion volume target, and Datadog keeps adjusting the service's rate automatically. This suits cases where you want to keep every service and endpoint visible while only capping the volume.The other is manual configuration per resource. You write a rate for each resource name, such as an endpoint, and a wildcard (*) lets you cover similar names at once. If you want to decide the gradation yourself, such as keeping 100% of important payment processing and only a few percent of health checks, this is the one to use.With either method, in environments where Remote Configuration is available, changes can take effect from the Datadog UI without redeploying the service. The conditions are Datadog Agent 7.41.1 or later, Remote Configuration enabled, permission to write APM Remote Configuration, and the minimum SDK version defined for each language (for example v1.34.0 for Java) (as of August 2026). The path of generating a configuration file, applying it locally, and redeploying is available only for manual configuration per resource; to use adaptive sampling, you must meet the Remote Configuration conditions.
Three defaults that apply when there are no rules
If no rules are written on the service side, the decision is left to the Datadog Agent. By default, the rate is calculated automatically with a target of 10 traces per second per Agent (as of August 2026), and you can change this target with the environment variable DD_APM_TARGET_TPS or through Remote Configuration.Because that alone would miss failures and unusual operations, two supplements are provided. Traces containing errors are picked up at up to 10 per second per Agent even beyond the target (DD_APM_ERROR_TPS), and rarely seen traces can be picked up at up to 5 per second, though this one is disabled by default (DD_APM_ENABLE_RARE_SAMPLER). What you want to see during incident investigation is failures and rare cases rather than average successes, so these two are worth remembering.These values, too, can take effect without restarting the Agent if you use Remote Configuration (Agent 7.42.0 or later). Values specified remotely take precedence over local values written in environment variables or configuration files.
Which setting wins when settings overlap
It is not unusual for the same service to receive settings from several places. The order of precedence is as follows, with items higher up overriding those below.
- Per-resource rules set through Remote Configuration
- Adaptive sampling rules
- Sampling rules written locally (
DD_TRACE_SAMPLING_RULES) - A global rate set through Remote Configuration
- A global rate written locally (
DD_TRACE_SAMPLE_RATE) - The rate determined indirectly by Agent settings (
DD_APM_TARGET_TPS)
These six levels can be restated as three principles: settings in the sending library are stronger than Agent settings, individual rules are stronger than global rates, and remote settings are stronger than local settings. When you do not get the rate you intended, first suspect that another setting remains at a higher layer.
Fluctuation in the effective rate, and volume decided upstream
The first point is the effective rate over short periods. In a window of about 1 to 4 hours, the displayed effective rate can fall below 100 percent even when you set it to 100 percent. This is a matter of statistical convergence, not a sign that the setting is not working. Make your judgment after widening the window.The second point is that some services have a volume that your own settings alone do not determine. When the service that first receives a trace makes the sampling decision, that decision is inherited by downstream services. As a result, for a service called from upstream, changing settings on its own page may not change the volume. You can trace where the volume comes from by looking at the ingestion reason on the page and the list of services that made the decision.For details, see Datadog's Ingestion Controls.