Log Management

The Overall Picture of Collection Routes

Datadog Log Management begins with ingesting logs from sources such as hosts, containers, and cloud providers (as of August 2026; source: Datadog Log Management documentation). The role of collecting log files on servers and containers falls to the Datadog Agent, while logs emitted by cloud services are ingested through integrations. Because arriving logs can be checked in Live Tail the moment they stream in, verifying that your collection settings are taking effect is also completed here. Building a table of where each log originates and which route brings it in is the foundation for the billing design described later.

Shaping Logs with Pipelines

Ingested logs can be processed with pipelines and processors to enrich their content (source: same documentation). As raw text, operations such as filtering to errors only or counting per user are impossible, so the earlier you extract attributes and normalize the shape, the more effective everything downstream becomes: search, exclusion filters, and metric generation. If you cut corners here, you later end up wanting to filter with no axis to filter on, and you also lose the ability to decide how to reduce the volume going into the index. Think of pipelines as the center of log design.

Two-Tier Billing for Ingestion and Indexing - Ingest and Index Are Different Things

The biggest characteristic of Datadog log billing is that ingestion (Ingest) and indexing (Index) are separated. Datadog calls this Logging without Limits and explains it as a structure in which you ingest all logs while choosing only the portion to keep for search (as of August 2026). The unit prices differ by an order of magnitude, as follows.

Note also that they are counted with different yardsticks: ingestion by GB and indexing by event count. The official FAQ provides no formula for converting GB to event count, and instead directs you to count the number of log events in the most recent 24 hours and extrapolate to 30 days for an estimate. Since only what goes into the index is searchable, ingest everything, index selectively is the starting point of the design.

Retention Periods and Archives

You can create multiple indexes in an account (1 by default; the default upper limit is 100), and set a retention period and a daily quota separately for each. Because a log goes into the first index whose defined filter it matches, a split such as errors for 30 days and access logs for 3 days is achieved within a single flow (source: Indexes documentation). Logs you want to keep beyond the index retention period are routed to an archive. An archive is a storage location optimized for long-term retention, and if you later want to search it, you can dig it up by rehydrating from the archive; however, in addition to the charge per volume of compressed data scanned (0.10 dollars/GB on an annual contract as of August 2026), the re-indexed portion separately incurs the index unit price of your contract. Archive Search, which searches the archive directly without restoring it to an index, is also provided, at 0.05 dollars per GB scanned on an annual contract (as of August 2026; source: Datadog Pricing / Rehydrating from Archives). Remembering that archives are cheap but digging them up is not free makes retention design decisions easier.

The Search Toolkit

The entry point for search is the Log Explorer, which searches across the logs placed in your indexes. Also provided are Live Tail for watching logs as they stream, Patterns for grouping similar logs to grasp the overall picture, Analytics for aggregating and viewing, and Saved Views for saving and sharing search conditions (as of August 2026; source: Log Management documentation). In actual incident response, the efficient sequence is to first use Patterns to find, as a cluster, which kind of log surged, then narrow to that pattern and read the individual logs. If you turn frequently used filters into Saved Views and share them with the team, even a response in the middle of the night can start the investigation from the same viewpoint.

Typical Patterns of Log Explosions and How to Prevent Them

Log billing accidents tend to happen not as a quiet increase in volume but as a sudden eruption one day. There are two typical patterns: one is leaving debug-level logging on in production and then receiving high traffic, and the other is a process or container repeatedly restarting and endlessly emitting startup logs. As a breakwater on the Datadog side, you can first set a per-index daily quota as a hard limit. The quota is specified in units of 1 million events, and an event is generated when the warning threshold (50% or more) is reached, so connecting this to a monitor lets you receive approaching the quota as a notification. Furthermore, exclusion filters let you trim the volume going into the index with a combination of a query and a sampling rate (0 to 100%). Excluded logs are merely left out of the index and remain usable for Live Tail, metric generation, and archives, so you can build a graceful degradation in which the logs are not searchable but the trail remains (source: Indexes documentation).

Dividing Responsibilities with CloudWatch Logs

In an AWS-centered setup, the logs of many services first land in CloudWatch Logs (VPC flow logs, Lambda logs, and so on). CloudWatch Logs lets you set retention per log group from 1 day to 10 years, or indefinitely, and includes search with a dedicated query language called Logs Insights (as of August 2026; source: AWS CloudWatch Logs documentation). The strength of Datadog Log Management, on the other hand, is gathering all sources, including those outside AWS, onto a single search surface and cross-referencing them on the same screen as metrics and traces. A realistic division is to keep long-term preservation that is contained within AWS (such as indefinite retention of audit logs) on the CloudWatch Logs side, and forward and index in Datadog the portion used for daily investigation, cross-source search, and alerting. Retaining all logs long-term in both places means paying storage costs on both, so deciding which one is the primary repository before building the forwarding configuration is what separates gain from loss.