SLO (Service Level Objective)
A target that declares how far a service's reliability must be met, as a percentage over a period; as the flip side of the target, the amount of failure you can tolerate is decided at the same time.
Overview
A Datadog SLO is a feature that declares, for an indicator such as the percentage of successful responses or the percentage returned within a prescribed speed, "how much of it must be met over this period", and lets you follow attainment and the remaining margin on the same screen. The moment you set a target, the amount you do not need to meet (the error budget) is decided automatically as well. There are three types: one calculated from indicator counts, one calculated from the uptime of existing monitors, and one that paints time slices as good or bad. The period is chosen from rolling windows such as 7, 30, or 90 days. Because you can see not only whether you are meeting the target but also how fast you are consuming what remains, it becomes a shared yardstick when discussing whether to devote effort to stabilization or to adding features.
Separating four words: indicator, objective, agreement, and remaining margin
The foundation is the measuring-side indicator called an SLI. You shape it so that good or bad can be expressed as a single number, such as the percentage of responses that succeeded or the percentage returned faster than a prescribed number of seconds. An SLO is that SLI with a level placed on it, such as "99.9% over a rolling 30 days"; an SLA is the same level made explicit as a promise to customers, with the consequences of failing to meet it also defined. The official glossary defines these three words separately, and if you use them interchangeably, internal targets and external liabilities end up carrying the same weight.The fourth is the error budget, officially defined as the "amount of tolerable failure" derived from the target value, that is, 100% minus the target. If you aim for 99.9% over 30 days, what remains is the 0.1% share, and whether from an outage or a side effect of a change, the time or count of failures is subtracted from it. The advantage of lining up these four words is that attainment can be treated as a dwindling balance rather than viewed as a single percentage.
Choose among the three types by how the SLI is built
When you create an SLO in Datadog, the first thing to decide is the type. The difference among the three lies in how attainment is calculated, and choosing based on the monitoring assets you already have keeps you from getting lost.
| Type | Attainment calculation | Prerequisite |
|---|---|---|
| Metric-based | Divide the good count by the total count | An indicator that can be counted as events |
| Monitor-based | The percentage of time the underlying monitor was in a healthy state | The target monitor must be created first |
| Time-slice | Divide the time matching your own definition of healthy by the total time | Thresholds and target filtering must be defined in an expression |
The time-slice type needs no monitor, and on the creation screen you can change the filter and threshold and confirm on the spot how much time was unhealthy. The number of decimal places you can write in the target differs by type and period: monitor-based allows 2 places for 7 and 30 days and 3 places for 90 days, while metric-based allows up to 3 places for any period. The limit on decimal places affects design in the form of "there are periods where 99.95% can be written but 99.995% cannot", so if you decide the level before choosing the type, you will get stuck.
Why the target cannot be set to 100%
The official documentation states explicitly that to gain the benefits of the error budget and its alerts, the target must be set strictly below 100%. Two reasons are given. At 100%, the error budget becomes 0% and the quantity representing acceptable risk disappears, so you can no longer build agreement between the conflicting priorities of maintaining customer-visible reliability and investing in feature development. The other is that a target of 100% causes division by zero when alerts are evaluated.This is also the part that makes setting an SLO worthwhile. Even in the official definition, the error budget is a quantity "intended to be spent on product development"; if there is budget left, you may proceed with uncertain changes such as new features or migrations, and if it is used up, you put stabilization first. You can state that judgment in terms of the remaining balance rather than impressions. Not having to redo the debate between the cautious camp and the ambitious camp every time is a byproduct of fixing the target to a single number.
Watch not just the balance but how fast it is falling
On the list screen, a burn rate indicator appears next to the SLO name. This uses a rolling window of the most recent 2 hours to single out SLOs that are consuming their error budget too fast. If the burn rate over the past 2 hours exceeds 6, it is marked red for critical; between 1 and 6 it is marked yellow for elevated; and you can filter by critical, elevated, and healthy. For SLOs carrying a service tag, you can navigate directly from the display to the page for that service.To send notifications, you create a monitor separately from the SLO. Two kinds are provided: an error budget alert notifies you when a specified percentage of the balance has been consumed, and a burn rate alert notifies you when the speed of consumption exceeds a specified level and stays there for a set duration. The variables you can put in the message text are also divided: the consumed percentage is exclusive to error budget alerts, and the short-window and long-window burn rate values are exclusive to burn rate alerts. "This month is already in danger" and "the current pace is bad" are different questions, so you use both for their own purposes.
Removing planned outages from the balance
If time stopped for planned maintenance is also subtracted from the balance, the target stops reflecting operational reality. For this there is a mechanism called status corrections, which lets you drop a specified period from the attainment and error budget calculations. The uses the official documentation lists are not reducing the balance for scheduled outages, ignoring hours outside business hours where there is no premise of upholding the SLO, and not carrying in transient problems caused by deployments.How a period is dropped differs by type: monitor-based does not count the corrected period, metric-based counts neither good nor bad events in that period, and time-slice treats the corrected period as healthy time. Corrections come in one-time and recurring forms, and either way you must choose a reason category (scheduled maintenance, outside business hours, deployment, or other). Because the category is retained, you can later trace "why was this period excluded."
Creating it alone does not alert, and it does not reset at the start of the month
The first point is that creating an SLO alone does not make anything fire. Attainment is calculated and shown on screen, but notifications arrive only after you separately create an error budget alert or a burn rate alert. The underlying monitor used by a monitor-based SLO firing and the SLO's balance being in danger are separate events.The second is how the period is counted. An SLO with a rolling 30 days selected is, whenever you look at it, evaluating "the 30 days looking back from that moment", so it is not a number that resets at the start of the month. If you want to review by calendar boundaries, switch the list screen to the daily, weekly, or monthly calendar view to follow 12 months of history. Note, however, that this view supports metric-based and time-slice types; monitor-based is not supported.