obsdog

Datadog Fundamentals Certification / 05 / 06

Troubleshooting Datadog

What this domain covers

In this domain you learn how to isolate the cause yourself in situations such as data not arriving after you installed the Agent, an integration not working, or the Agent failing to start. There are three pillars. First, reading the output of the status command to understand the situation (the Running Checks, Collector, and Clocks sections). Second, safely handing the information support needs to them with a flare. Third, understanding the behavior of the Agent logs (location, log level, runtime changes) and of the configuration files (the loading rules of conf.d, .default files, temporary disabling). On the exam these topics are asked in forms such as "where do you look first for this symptom," "what does this subcommand diagnose," and "how does the Agent behave with files placed this way." Numbers and defaults are based on the official documentation as of September 2026.

Topics listed in the official guide

  • Agent commands (status / flare)
  • Agent logs
  • Agent configuration files

Key points

  • The official checklist verifies reachability to the internet or a proxy, that the API key and site match, that only one Agent is running, and that the Agent was restarted after YAML edits.
  • If an integration appears in the Running Checks section of the status output with no warnings or errors, that integration is configured correctly.
  • Gaps in graphs, metric delays, and incorrect alert triggers can be caused by NTP drift. Check the NTP offset in the Clocks section of the status output.
  • A flare is the mechanism that bundles configuration files and logs and sends them to support; sensitive information such as passwords and API keys is removed, and you are asked to confirm before upload.
  • diagnose checks connectivity to the destination, and configcheck displays the loaded and resolved configuration. Choose between them depending on whether the symptom is network or configuration.
  • The Agent logs on Linux are in /var/log/datadog/, and the default log level is INFO. Enable DEBUG only for a limited period and set it back to INFO when you are done.
  • In Agent 6.19 / 7.19 and later, config set log_level debug changes the log level without a restart, but the change is lost on restart.
  • All valid YAML files inside <CHECK_NAME>.d/ under conf.d are loaded, and zero-length files are ignored. .default files define the default core checks.
  • To temporarily stop a check, rename conf.yaml to an extension that is neither .yaml nor .yml (such as conf.yaml.disable).
  • Agent 7.40 and later containers exit right after startup if they cannot resolve the hostname. Verify reachability to the Kubelet API and the cloud metadata endpoint.

Terms and concepts

The troubleshooting checklist

The official documentation lists 7 items to check first when you suspect a problem with the Agent. Whether the Agent container has stopped right after startup, whether the internet or a proxy is reachable, whether the Agent has the proxy settings if a proxy is used, whether the API key belongs to your platform, whether site matches the organization's site, whether only one Agent is running, and whether you restarted after editing YAML. If none of these is the problem, move on to the status command, the Agent logs, and DEBUG mode, and if that still does not explain it, contact support with a flare attached.

How to read the output of the status command

datadog-agent status displays the state of the Agent and its integrations in one place. A correctly configured integration appears in the Running Checks section with no warnings or errors, along with values such as Total Runs and Metric Samples. The Collector section shows a summary of the running check instances and the number of metrics collected, which helps narrow down the cause when the Agent's CPU or memory usage is high. The NTP offset in the Clocks section lets you check for clock drift, which can cause gaps in graphs and false alerts.

flare

A flare is the mechanism that gathers the Agent's configuration files and logs into a single archive and sends it to support. Sensitive information such as passwords, API keys, and proxy credentials is removed, and you are asked to confirm before upload. The command is sudo datadog-agent flare <CASE_ID>, after which you enter the email address associated with the case. If you have no case number, a new case is created using your login email address. When the Agent cannot connect to Datadog, you submit the archive created in /tmp manually.

Choosing between the diagnostic subcommands

The Agent CLI is subcommand based. The ones used in troubleshooting are check, which runs the specified check; configcheck, which displays all the configuration the running Agent has loaded and resolved; diagnose, which diagnoses connectivity; health, which displays the Agent's health; hostname, which displays the hostname the Agent uses; and stream-logs, which streams the logs being processed. The basic split is: if checks are running but data is not arriving, use diagnose; if a check does not appear in Running Checks, verify with configcheck.

Agent log files and log levels

The Agent log on Linux is agent.log in /var/log/datadog/. On macOS it is /opt/datadog-agent/logs from 7.28 onward, and on Windows it is C:\ProgramData\Datadog\logs. By default the log rolls over every 10 MB and keeps 1 backup. The default log level is INFO; you choose from 7 levels ranging from OFF to TRACE, and the specified level and above are output. Because DEBUG increases the log volume, enable it only for a limited period and set it back to INFO when you are done. OFF must be enclosed in quotation marks.

Runtime configuration management

In Agent 6.19 / 7.19 and later, the config subcommand can change some settings without a restart. config list-runtime lists the items that can be changed, config set log_level debug changes the value, and config get log_level shows the current value. The advantage is that you can switch the log level to DEBUG without recreating the container, but dynamic changes do not persist and revert when the Agent restarts. A trace-agent running in a separate container does not support runtime changes; set DD_LOG_LEVEL and redeploy instead.

Loading rules of conf.d

Integration configurations go in conf.d/<CHECK_NAME>.d/, and because the Agent loads every valid YAML file in that folder, you can split them across multiple files. Zero-length files are ignored. Files with the .default suffix define core checks that are always enabled by default, such as CPU and memory; they are ignored if the same check has another configuration, and you delete them when you want to stop the check. To stop a check temporarily, rename conf.yaml to a different extension. For log collection, multiple YAML files pointing to the same source are not accepted, and only the first file in alphabetical order is used.

Permissions and ownership (the dd-agent user)

On Linux the Agent runs as the dd-agent user, and /var/log/datadog/ and the files inside it must be owned by dd-agent. If the owner is different, the Agent cannot write its logs and reports Permission denied, so check with ls -l, fix it with sudo chown -R dd-agent:dd-agent /var/log/datadog/, and restart. If an EACCES socket error appears at startup and there is neither a port conflict nor a leftover process, suspect the owner of /opt/datadog-agent/run. Running as root is nothing more than a workaround.

Hostname resolution failures in containers

From Agent 7.40 onward, if the hostname cannot be determined reliably in a container environment, the Agent emits an error and exits right after startup, and the log records Error while getting hostname, exiting. On Kubernetes the cause is being unable to reach one of the Kubelet API, the cloud metadata endpoint, or the container runtime API, and the most common case is a failure to verify the Kubelet's TLS certificate. In that case, set DD_KUBELET_TLS_VERIFY to false. In CI or sidecar setups, set DD_HOSTNAME explicitly.

Check your understanding

Check what you have learned with 5 questions