Monitors and Alerts
A mechanism that watches metrics and logs with thresholds or anomaly detection and delivers signs of trouble as notifications. Monitoring is not complete with just collecting; this is where it first takes the form of moving people to act.
What it does
Define monitoring conditions (monitors) on collected metrics, logs, traces, and other data, and send alerts to email, Slack, and other channels when the conditions are met. Beyond threshold comparison, you can also choose history-based anomaly detection and forecast-type monitoring.
When to use it
Set this up as the finishing touch on monitoring when you want to build a setup where you notice problems without watching screens all the time. The aim is to catch the signs before users are affected.
An everyday analogy
It is like a smoke detector in a house. When it senses smoke (a sign of an anomaly), it sounds loudly even if nobody is there. If you get the sensitivity wrong, it also becomes the annoying device that goes off every time you cook.
Monitor types - more than threshold comparison
Datadog monitors are a mechanism for detecting problems in advance and responding in real time, and the official documentation as of August 2026 lists 29 monitor types (source: Monitor Types). The centerpiece is the Metric monitor, which compares a metric value against a threshold, but there are also Logs monitors that watch log counts, APM monitors that watch trace metrics, Host monitors that detect when reporting from a host has stopped, Live Process monitors that check whether a process is alive, Composite monitors that combine multiple monitors with an expression, and SLO Alerts that watch an SLO's error budget, among others. If you keep in mind the structure that "every data source has a monitor type that watches it," you can pick the type by working backward from what you want to monitor. For a sorting-out of the individual concepts, see also the Monitor entry in the glossary.
Threshold design - think in terms of the 3 points: warning, alert, and recovery
Thresholds consist of a required alert threshold and an optional warning threshold; the warning is treated separately as a notification that comes before the real alert. You can additionally set an optional recovery threshold, and if it is not set, the monitor is automatically treated as recovered the moment the value falls below (or rises above) the threshold (as of August 2026, source: Configure Monitors). The trick to design is to decide these 3 points as one story. The warning is "the level at which a person deals with it in a planned way during business hours," the alert is "the level at which someone gets called right now," and the recovery threshold is "the margin that keeps notifications from oscillating as the value moves back and forth near the threshold." If you leave recovery at the same value as the alert, fluctuations of a value sitting on the boundary turn directly into flickering notifications. Placing a deliberate gap between alert and recovery is the first move in the alert fatigue countermeasures described later.
How to write notifications - can the recipient act next?
Notifications can be sent to email or Slack, with a graph snapshot attached and the affected target and value inserted into the body via template variables. Routing to deliver notifications to the appropriate person in charge is also covered in the official documentation (as of August 2026, source: Datadog Monitors documentation). The quality of the body is decided not by the mechanism but by how it is written. A useful notification states not only "what, where, and how much" but also "what to check first." If you picture the person receiving it in the middle of the night and embed the entry point for investigation (the dashboard to look at and where to check recent changes) in the notification text, the first response shrinks by minutes. Conversely, a notification that just pastes the condition expression makes the recipient pay a decoding cost every time it arrives.
The toolkit for preventing alert fatigue
The official documentation itself sets the goal of "reducing alert fatigue so you can focus on responding when it matters," and the tools for that are provided (as of August 2026). First is the unit of aggregation: for grouped queries you can choose Multi Alert, which notifies separately per target, or Simple Alert, which bundles everything into 1 notification, and narrowing the grouping dimensions of notifications reduces noise. Second is downtime, which lets you mute alerts during maintenance in a planned way. Third is evaluation delay, which lets you delay evaluation by up to 86,400 seconds so that late-arriving data is not misdetected (a 15-minute delay is recommended for metrics originating from cloud providers). Fourth, you can explicitly choose how missing data (No data) is handled, such as evaluating it as zero, keeping the previous state, or notifying it as missing. As a principle that comes before choosing among the tools, regularly carrying out "delete or downgrade to warning any alert that fires but nobody acts on" is the substance of alert fatigue countermeasures.
Where anomaly-detection monitors fit
For metrics where a static threshold is hard to decide, there are types that judge by comparison with history or with a population.
| Type | Basis for judgment | Suitable situation |
|---|---|---|
| Anomaly | The metric's own history | Values with day-of-week or time-of-day periodicity where a fixed threshold is hard to decide |
| Outlier | Comparison with other members of the same group | Detecting that just 1 host in a homogeneous host group behaves differently |
| Forecast | Whether the future forecast will cross the threshold | Values such as disk usage where you want an advance warning of exhaustion |
| Change Alert | The amount of change over a fixed period | Values where "it changed suddenly" matters more than the absolute value |
Convenient as they are, anomaly-detection types make "why did it fire" harder to explain than static thresholds, and if the history they learn from is dirty, the judgment is dirty too. Setting the main values with straightforward thresholds first, and replacing only the spots where periodicity causes repeated false alarms with anomaly-detection types, is the order that keeps operations explainable.
How to answer "what should we monitor?"
The approach of filling in from a checklist of monitoring items can miss the failures that matter despite the number of items. What to decide first is monitoring of "symptoms visible to users." Watch the symptom side first, such as slow responses, returned errors, and processing that never finishes, with APM or log counts, and position the resource side, such as CPU and memory, as secondary monitoring used to investigate the cause of symptoms. If you set up large numbers of resource-side alerts alone, an inversion tends to happen: a midnight CPU spike with no user impact wakes someone up, while nothing fires during a real outage. On top of that, adding a small number of "is monitoring itself alive" checks, such as a host that stopped reporting (Host monitor) or process liveness (Live Process), tightens the foundation of monitoring. If you are torn between watching Infrastructure Monitoring and Log Management, the practical way to decide is "which screen does the person who receives that notification open first?"
Putting it into operation - naming conventions and periodic review
A monitor's value is decided not at the moment it is created but by what it looks like six months later. So that management does not collapse as the count grows, decide a naming convention at the start. If the target system, environment, and symptom can be read from the name alone (what is wrong with which service), the notification list becomes a status report as is. Also, in a quarterly review, identify "monitors that never fired during this period" and "monitors that fired but nobody acted on"; consider revising thresholds for the former and deleting or downgrading to warning for the latter. Monitors left silenced by manual mute instead of downtime are the main search target of the review. Design your operations on the premise that keeping alerts meaningful when they fire is harder than setting them up.
Things to watch out for
- This article is based on the official documentation as of August 2026. Specifications may change, so always check the latest official documentation for configuration details.
- If no recovery threshold is set, the monitor is automatically treated as recovered the moment the value crosses back over the threshold. When values fluctuate near the boundary, notifications will oscillate, so leave a deliberate margin.
- Evaluation delay is especially important for metrics originating from cloud providers (the official recommendation is a 15-minute delay, as of August 2026). Insufficient delay misdetects "the data just has not arrived yet" as an anomaly.