Log Pipeline
The process in which Datadog passes received logs through processors in order, extracts attributes from the body, and shapes them into a form usable for search and aggregation. Every arriving log is matched against the filters of all pipelines, and the processing of the matching ones is applied.
Overview
A Log Pipeline is the process in which Datadog passes received logs through processors in order and extracts attributes from an unstructured body so that they can be used for search and aggregation. Logs in JSON format have a preprocessing stage that maps your own attribute names to the reserved attributes of date, status, host, service, and message. Because the filter is evaluated before the processors of that pipeline, attributes extracted within the pipeline cannot be used as conditions. All configuration is completed in the UI, with no changes needed on the application side.
The process that gives structure to logs after they are received
A Log Pipeline (Log Pipeline) is the process in which Datadog passes received logs through processors in order, parsing them and adding information. It extracts values from an unstructured body, turns them into attributes, and makes those attributes usable for search and aggregation. Think of it as the shaping work that happens before a log enters a Log Index.An arriving log is matched in order against the filters of all pipelines. The processors of each pipeline whose conditions match are applied, and the log then proceeds to the next pipeline. Logs in JSON format are parsed without additional configuration, so emitting JSON on the sending side reduces the work later.All configuration is completed in the Datadog UI. There is no need to rewrite the log output settings on the application side or redistribute anything to servers. Note that this mechanism targets logs in the cloud; if you want processing to finish inside your own facilities, you use a separate mechanism called Observability Pipelines.
Preprocessing runs before pipelines
Logs in JSON format have a preprocessing stage that runs before pipelines. Here, your own attribute names are mapped to the reserved attributes that Datadog treats specially (date, status, host, service, and message). The default settings are aligned with common forwarding methods, so in most cases they work without being touched.
| Reserved attribute | Keys read by default | Points to watch |
|---|---|---|
| Date | @timestamp, timestamp, eventTime, published_date, and others | Logs whose official date is older than 18 hours are not accepted. Formats are ISO8601, UNIX (milliseconds), and RFC3164 |
| Status | status, severity, level, syslog.severity | Becomes a value usable for narrowing searches |
| Host | host, hostname, syslog.hostname | In Kubernetes, this key overrides the hostname held by the Agent, and the host-side Tags you expected are no longer attached |
| Service | service, syslog.appname, dd.service | Filled in automatically when sent via the Datadog Agent |
| Message | message, msg, log | Treated as the body that is the target of full-text search |
Specifying the host is the one thing that can only be done in this preprocessing. Even if you line up processors that rewrite attributes afterward, the logs will not be treated as grouped per host, so a mistaken hostname is fixed here. If you want to use keys other than those in the table above, specify them in the corresponding remapper processor.
The filter is evaluated before the pipeline's own processors
When you create a pipeline, you first write a filter that selects the target. What to be careful about here is that the filter is evaluated before the processors of that pipeline. Attributes extracted within the pipeline cannot be used as conditions, so if you want to split by such an attribute, route it to a nested pipeline.Besides the name, you can add a description and tags. Tags here do not affect logs; they are markers for narrowing down the pipeline list. Making it possible to trace which team placed what and for what purpose makes maintenance easier as the number grows.For commonly used log sources, Datadog provides integration pipelines. When you configure the corresponding source and the first log arrives, the pipeline is added automatically, with parsing suited to that format already in place from the start. They are read-only, so if you want to modify one, clone it first and then edit. They cannot be deleted, only disabled. How they are processed by default can be checked in the list of integration pipelines.
When multiple processors write the same attribute
Sometimes multiple pipelines match the conditions and try to put a value into the same attribute. Which one remains is determined by the type of processor, and there are three behaviors.
| Behavior | Result | Main processors |
|---|---|---|
| The later write remains | Overwritten by the value of the processor later in the order | Grok parser, category, arithmetic, string builder, lookup, URL parser, user agent parser, geo IP parser |
| Depends on configuration | You can choose whether to overwrite. By default, no overwrite if the target is already filled | Attribute remapper, array copy |
| The earlier write remains | Only the first match takes effect. Across pipelines, too, the first one encountered takes effect | Status, service, message, trace, and Span remappers (the date remapper alone is the exception, where the last one takes effect) |
If you reorder without knowing this difference, an attribute you thought you had fixed appears to revert. The shortcut is to first confirm which behavior the processor you are touching has.There are also guidelines on numbers. Datadog recommends up to 20 processors per pipeline and up to 10 Grok parsing rules (as of August 2026), and extremely heavy rules or pipelines may be stopped on the Datadog side. Splitting into nested pipelines is safer than piling up a long chain.
Check the effect before you change
When you rebuild pipelines or processors, you can check the effect before applying. A page is provided that uses the logs currently flowing as they are and compares before and after the change side by side. You can choose between two comparisons: comparing the currently running version with your change, and comparing logs before they enter the pipeline with logs after passing through the whole thing.The list can also be filtered by whether there was an impact: all logs, logs whose contents were changed by the modification, and logs that were unchanged. When the count is large, looking only at the changed ones lets you notice unintended side effects.On the operational side, each pipeline shows who last changed it and when, and you can filter the list by that. Editing permissions have two levels: a role that can touch the whole pipeline, and a role that can touch only processors and nested pipelines. The latter cannot change the filter conditions or the ordering, which suits organizations that want to divide responsibilities. Each pipeline also shows an estimate of ingested volume and count, and the detailed breakdown can be checked from the default usage dashboard.For details, see Datadog's Log Pipelines.