Alert

Overview

In Datadog, an alert refers to the message sent out to recipients when a monitor's state changes. The monitor is what holds the conditions and thresholds; the alert is the layer responsible for "who receives it, with what text, and how many times." Thinking of these as two separate layers lets you fix two different failures as separate tasks: the monitoring was running but nobody noticed, and conversely, it fired so often that nobody read it anymore. How recipients are specified, the variables you can embed in the body, and how renotification and aggregation take effect are the pillars of the design.

Drawing the line between alerts and monitors

The configuration that defines a monitoring condition is itself called a Monitor, and the message sent outward when that condition is met and the state changes is the alert. The monitor holds the state; the alert is the output produced by that state transition.Where this distinction pays off is in incident retrospectives. If the state correctly changed to alert but nobody acted, what needs fixing is not the threshold but the recipients and the text. Conversely, if the state never changed, no amount of polishing the notification will get it delivered. In Datadog's configuration screen, too, the evaluation condition section and the notification section sit side by side as separate blocks.Seen from the receiving side, a single monitor may send zero notifications, or it may deliver many times depending on the renotification settings. It matches reality better to think of the count as determined by the notification design rather than by the number of monitors.

Deciding who receives it

There are three layers to specifying recipients. Rather than writing recipients by hand into each individual monitor, Datadog recommends using monitor notification rules to add recipients automatically according to conditions.

The integration format is @<INTEGRATION_NAME>-<VALUES>, and prefixes are provided for Jira, PagerDuty, Slack, Webhooks, Microsoft Teams, ServiceNow, and others. The format has two pitfalls: a space is required immediately before the recipient specification, and handles containing parentheses are not interpreted, so no alert is created (as of August 2026). For email, writing an address after @ lets you send to people who are not Datadog users, but users with pending invitations or disabled users are not eligible recipients.For details, see Datadog's Notifications.

Write the body as a runbook

The notification body field supports Markdown syntax and variables, so you can write steps the recipient can act on right away. The basic pattern combines conditional variables such as {{#is_alert}} to switch the text by state with tag variables such as {{host.name}} to insert the name of the target that fired. For a disk space warning, the official example lists numbered steps from removing unneeded packages to cleaning up duplicate files, and places the recipient for the responsible channel at the end.Markdown interpretation differs by route, however. Bold, italics, inline code, and links all work in both Slack and email, but tables are not formatted and arrive as raw text, and in Slack heading syntax also comes out as raw text. Even where you would like to organize things in a table, reducing it to a bulleted list in the notification body delivers it in a readable form.Severity can be expressed as a priority from P1 to P5. Using {{override_priority 'P1'}} against the default priority lets the same monitor send a different priority in the warning state and the alert state.

Tuning notifications that keep firing and ones that never fire

When you want a reminder about an unresolved problem, enable renotification. What you can configure is the interval, the states that trigger renotification (alert, warn, and no data), and the maximum number of times to send. Setting the maximum to 1 gives you a single follow-up after the first alert. The recommended way to write escalated text is in a {{#is_renotify}} block; in this case the original body is sent along with it, so write only the additional information inside the block.On the other side, suppressing excess noise is where aggregation takes effect. When the monitor's query is grouped, you can drop some of the axes from notification grouping, or drop all of them to consolidate into a single notification. You can also choose what to include in the notification. Options are provided to hide only the query, hide only the list of recipients, or remove the query, recipients, snapshot, and the related links at the end all at once, so if you do not want the contents of your monitoring flowing into external channels, this is where you narrow it down.Once you have finished building it, confirm the real thing with a test notification. In a test you can choose the state transition and group and actually send it, and the notification that is sent carries a mark identifying it as a test and is recorded as an event.

Design order and common mix-ups

Working backward from "what does a person do in the 3 minutes after it fires" reduces rework. The order is: first decide the recipient, then write the text that lets that recipient act, and finally adjust the volume with renotification and aggregation. Starting from fine-tuning thresholds tends to leave a state where it keeps firing while the people who receive it do not know what they should do.There are two points that are easy to mix up. One is equating an alert not arriving with a missed detection. Route-side reasons, such as the recipient being a disabled user or the format being broken, produce the same symptom. The other is reading the number of notifications as the number of incidents. Depending on the aggregation and renotification settings, the same single incident can become dozens of notifications or just one. If you treat the count as a metric, fix the aggregation settings before comparing.Building out monitors as a whole, including alerts, is covered in the Monitors and Alerts explainer.