Observability Pipelines
A preprocessing layer in which a Worker running in your own environment aggregates, transforms, and routes logs and metrics to destinations before they are sent to Datadog. Decisions to reduce, redact, and distribute to multiple destinations can be made before the data leaves your environment.
What it does
You run the Observability Pipelines Worker on your own servers or Kubernetes; it takes logs received from the Agent, Syslog, Kafka, and other sources, transforms them with processors for filtering, redaction, and conversion to metrics, and then sends them to multiple destinations such as Datadog Logs, object storage, and SIEM products. Pipeline design and Worker deployment are done from the Datadog UI.
When to use it
Use it when you want to redact logs containing personal or secret information before they leave the company, distribute the same logs to both Datadog and your own SIEM, or drop debug logs and duplicate logs before ingestion to keep the bill down. Dual-sending to a legacy platform and Datadog during a log management migration is also among the official templates.
An everyday analogy
It is the in-house mailroom before mail goes out of the company. Envelopes gathered from each department are received in one place, sorted by destination, parts that must not be shown are blacked out, the same notice is copied and sent to multiple recipients, and unwanted advertising is thrown away. Because this work is finished before anything leaves the building, the outside carriers (each destination, including Datadog) receive only what has been tidied up.
Observability Pipelines is the layer that processes data before it reaches Datadog
Observability Pipelines is a product that collects and transforms logs and metrics inside your own infrastructure and routes them to multiple destinations. The official documentation positions it as "giving you control over your observability data before it leaves your environment," and explains that redaction of sensitive information, enrichment, noise exclusion, and routing to Datadog, SIEMs, and cloud storage can be assembled from templates (as of September 2026, source: Observability Pipelines). Whereas the log pipeline in Log Management is a step after logs reach Datadog, this is a step before they arrive. Billing is per uncompressed GB ingested by the Worker, with a unit price published for each of the three contract forms, annual contract, monthly contract, and on-demand (as of August 2026, source: Datadog Pricing). This article explains how billing works. For current unit prices and plans, refer to Datadog's official pricing page.
- Sources: Datadog Agent, Syslog, Kafka, Fluentd, HTTP, cloud log delivery services, and more (24 kinds in the official list)
- Worker (your environment): a set of processors for exclusion, sampling, redaction, field removal, conversion to metrics, and enrichment
- Destinations: Datadog Logs, Datadog Archives, Amazon S3, Google Cloud Storage, Azure Storage, Splunk HEC, Microsoft Sentinel, Elasticsearch, and more (24 kinds in the official list)
How it differs from the cloud-side log pipeline
Even though both carry the name "pipeline," they differ in where they run and at what point they process. Log Management pipelines run after Datadog receives the logs and handle parsing, attribute assignment, and routing to indexes. Observability Pipelines processing runs inside the Worker in your environment, and only the transformed data reaches Datadog. The official documentation likewise distinguishes that the Worker's processors are "for on-premises logging environments," while cloud-side log structuring and enrichment are done on the Log Management side (as of September 2026, source: Processors). Billing is also a separate line: the unit price on GB ingested by the Worker and the Datadog-side ingestion (Ingest) unit price apply independently. In other words, even when logs pass through the Worker, Ingest billing remains for what is sent to Datadog; what decreases is only the portion dropped at the Worker.
| Aspect | Observability Pipelines (Worker) | Log Management pipelines |
|---|---|---|
| Where it runs | Your own servers, Kubernetes, ECS Fargate | Datadog side |
| Point of processing | Before reaching Datadog | After Datadog receives the data |
| Main processing | Exclusion, sampling, redaction, field removal, aggregation, conversion to metrics, routing | Parsing, attribute assignment, normalization to standard attributes, routing to indexes |
| Destinations | Multiple external destinations including Datadog (up to 20 per pipeline) | Datadog indexes and archives |
| Billing unit | Uncompressed GB ingested by the Worker (three unit prices for annual / monthly / on-demand, as of August 2026) | Ingest by GB ingested, Index by event count and retention days |
The uses shown by the official templates fall into 3 families: reduce, hide, and distribute
The template list in the official documentation as of September 2026 has 7 for logs and 1 for metrics, and the uses can be organized into 3 families (source: Observability Pipelines). The reduce family is Log Volume Control and Generate Log-based Metrics; the hide family is Sensitive Data Redaction; the distribute family is Dual Ship Logs, Split Logs, and Archive Logs. Log Enrichment is a supporting template that adds information and spans all 3 families, and MetricTag Governance on the metrics side is for dropping unneeded tags to prevent high cardinality. Of the 3 families, the ones hard to substitute with Datadog-side features are hide and distribute; the reduce family can be partly covered by exclusion filters. This difference ties directly to the cost-benefit discussion in the following 2 sections.
| Template | Official description (summarized) | Family |
|---|---|---|
| Log Volume Control | Exclude low-value logs before storage to reduce indexed volume | Reduce |
| Generate Log-based Metrics | Convert high-volume logs into count or distribution metrics | Reduce |
| Sensitive Data Redaction | Detect and remove personal and secret information with built-in or custom rules | Hide |
| Dual Ship Logs | Send the same log stream to multiple destinations | Distribute |
| Split Logs | Route logs to different tools by log type | Distribute |
| Archive Logs | Store raw logs long-term in object storage and rehydrate later | Distribute |
| Log Enrichment | Add information from reference tables or fixed mapping tables | Supporting |
| Metric Tag Governance (metrics) | Keep only the metrics you need and drop unneeded tags to prevent high cardinality | Reduce |
Where to place the Worker and how to distribute configuration
The Worker is software you place in your own environment, and the official installation guides target Linux (RHEL and CentOS 8.0 or later), Docker, Kubernetes (major distributions), and ECS Fargate (as of September 2026, source: Install the Worker). Pipelines are built in the Datadog UI, which plays 3 roles: creation, Worker deployment and management, and enabling health monitors. When starting the Worker you pass the pipeline ID and an API key, and the key must have Remote Configuration enabled. Rather than distributing configuration files to each machine, the Worker receives what you edit in the UI. Processors are evaluated in order from the top, and there are limits of 25 processor groups and 150 processors per canvas, and 20 destinations per pipeline. Updating the Worker at least once a month is recommended.
- 1. Open Observability Pipelines in the UI and choose a template that fits your use
- 2. Select the log source
- 3. Configure processors (an initial set matching the template is included)
- 4. Add 1 or more destinations
- 5. Install the Worker in your environment (pipeline ID and an API key with Remote Configuration enabled)
- 6. Enable monitors that track pipeline health
A map of where log billing can be reduced - the only place to drop before Ingest is here
The means of reducing log billing fall into place when divided into 4 layers. Narrowing log levels in app or Agent configuration is layer 1, dropping at the Worker is layer 2, removing from indexes with exclusion filters after arrival at Datadog is layer 3, and the design of retention days and archives is layer 4. Because ingestion (Ingest) billing applies to GB that reach Datadog, layer 3 reduces only Index billing while Ingest remains. Only layers 1 and 2 can cut before Ingest, and the only one that cuts without app-side changes is the Worker. The official log volume reduction guide lists 7 Worker-side means: sampling, filtering, attribute removal, aggregation, deduplication, quotas, and direct-to-archive (as of September 2026, source: Strategies for Reducing Log Volume). For the overall picture see the 3 layers of log pricing, and for procedures see the cost review checklist.
| Layer | Means | Billing that decreases | Additional costs |
|---|---|---|---|
| 1. App and Agent configuration | Reviewing log levels and output destinations | Ingest and Index | None (requires app-side changes) |
| 2. Worker (Observability Pipelines) | Filtering, sampling, attribute removal, aggregation, deduplication, quotas, direct-to-archive | Ingest and Index | Worker ingestion GB billing and Worker operations |
| 3. After Datadog ingestion | Exclusion filters, index design | Index only | None |
| 4. Retention and archives | Shortening retention days, archiving and rehydration | Index (for the retention days) | Storage cost at the archive destination |
Conditions under which standing up a Worker pays off, and under which it does not
The unit price on GB ingested by the Worker and the Datadog-side Ingest unit price are both published per GB, and the pricing page places them at a similar level (both annual contract values, as of August 2026, source: Datadog Pricing). What this relationship means is that simply passing the full volume through the Worker and dropping a portion is unlikely to pay for itself. As formulas, the Worker-side increase is "full volume x Worker unit price" and the Ingest-side decrease is "volume dropped x Ingest unit price," so as long as the two unit prices are close, the gap in ingestion billing is not closed unless the proportion dropped is quite high. It pays off when any of the following holds: you calculate including the Index-side reduction (logs dropped at the Worker are not indexed either, and indexing cost tends to be larger than ingestion), you have requirements that cannot be substituted on the Datadog side such as redaction or delivery to multiple destinations, or you have destinations other than Datadog and want to do the same transformation once. If you only want to reduce indexed volume within Datadog alone, exclusion filters and a review of log levels are enough. There is also vCPU-based pricing for scales above 30 TB per month, but the unit price does not appear on the public price list.
Things to watch out for
- This article is based on the official documentation as of September 2026. The kinds and limits of sources, processors, destinations, and templates may change, so give priority to the official documentation's lists when designing.
- This article explains how billing works. For current unit prices and plans, refer to Datadog's official pricing page.
- Billing on GB ingested by the Worker applies separately from Datadog-side log ingestion (Ingest) billing. When logs that passed through the Worker are sent to Datadog Logs, Ingest billing remains for what is sent (based on the pricing structure as of August 2026).