obsdog

Datadog Fundamentals Certification / 03 / 06

Networking and Agent Configuration

What this domain covers

In this domain you learn in which direction, to which destinations, and on which ports the Datadog Agent communicates, and you build the knowledge needed to run the Agent reliably on a restricted network. The first half covers SSL traffic that is initiated one way from the Agent to Datadog, the per-site destination domains and allowlisting with ip-ranges, outbound ports such as 443/TCP and 123/UDP, host-local ports such as 8125/UDP, and proxy configuration. The second half covers Autodiscovery, which finds monitoring targets automatically in environments where containers move, and you learn where to place templates, how to narrow the targets with ad_identifiers, template variables, and include and exclude rules. On the exam these topics are asked as practical decisions: what to register in the allowlist, which port to open for what purpose, and where to write a template.

Topics listed in the official guide

  • Ports used by Datadog
  • IP addresses (allowlist)
  • Autodiscovery

Key points

  • Agent traffic is always initiated by the Agent to Datadog, and Datadog never opens a session to the Agent. All traffic is sent over SSL.
  • The metrics destination takes the form <VERSION>-app.agent.<SITE>, so register *.agent.<SITE> as a wildcard entry on the firewall.
  • The destination domains are CNAMEs to a set of static IPs, and the list is the JSON at https://ip-ranges.<SITE>. Put the entire range in the allowlist even if only part of it is used.
  • Most outbound traffic uses 443/TCP. NTP time synchronization uses 123/UDP, and blocking it leads to clock drift warnings and abnormal metric timestamps.
  • The default host-local ports are DogStatsD 8125/UDP, APM 8126/TCP, IPC 5001/TCP, and GUI 5002/TCP. They can be changed in datadog.yaml.
  • DogStatsD listens only on 127.0.0.1 by default. To receive from another container or another host, set dogstatsd_non_local_traffic to true.
  • A proxy is configured in the proxy section of datadog.yaml or with environment variables such as DD_PROXY_HTTPS, and the environment variables take precedence. Restart the Agent after a change.
  • Autodiscovery watches container start and stop events and, for containers whose ad_identifiers match, generates a static configuration with the template variables replaced.
  • Templates can be placed in any of pod annotations, Docker labels, files in a mounted conf.d, ConfigMaps, and key-value stores.
  • Container exclusion and inclusion are written in DD_CONTAINER_EXCLUDE / INCLUDE as regular expressions with a prefix, and in principle include takes precedence over exclude.

Terms and concepts

Direction of outbound traffic and ports

Agent traffic is always initiated by the Agent to Datadog, and no connection is ever opened from Datadog to the Agent, so on the firewall you only need to think about allowing outbound traffic. All traffic is SSL, and most of it, including the Agent itself, APM, containers, Live Processes, and metrics, uses 443/TCP. In addition, 123/UDP is used for NTP time synchronization, and blocking it leads to clock drift warnings and abnormal timestamps. Sending logs over TCP has no delivery guarantee and is not recommended; HTTP is the recommended path. The destinations change with the site you are contracted to (US1, EU, and so on).

Destination domains and the allowlist

The destination differs by data type. Metrics, service checks, and events go to <VERSION>-app.agent.<SITE> (for example 7-31-0-app.agent.<SITE>) and flares go to <VERSION>-flare.agent.<SITE>, so register *.agent.<SITE> as a wildcard entry on the firewall. APM uses trace.agent.<SITE>, processes use process.<SITE>, and API key validation uses api.<SITE>. At installation time, also allow install.datadoghq.com and the apt / yum repositories.

ip-ranges (publication of static IP addresses)

All destination domains are CNAMEs that point to a set of static IPs. For environments that write their allowlist by IP, a JSON list is published at https://ip-ranges.<SITE>. The JSON is divided into sections such as agents / api / apm / logs / process, contains IPv4 and IPv6 CIDRs, and can also be fetched per section, as in /apm.json. Even though only part of the set is in use, the active IPs rotate during maintenance, so the official guidance is to register the entire range. webhooks and synthetics are source IPs on the Datadog side and are not needed for the Agent.

Host-local ports (inbound) and how to change them

The Agent listens on several ports within the host. The defaults are DogStatsD 8125/UDP, APM receiver 8126/TCP, IPC API 5001/TCP, go_expvar 5000/TCP, and browser GUI 5002/TCP, and in case of a conflict you change them with dogstatsd_port, receiver_port, cmd_port, expvar_port, and GUI_port in datadog.yaml. GUI_port can be disabled with -1, and the default on Linux is -1. If you change the DogStatsD or APM port, the sender side configuration must be adjusted to match.

dogstatsd_non_local_traffic

By default, DogStatsD 8125/UDP listens only on the IPv4 localhost address 127.0.0.1. Metrics from applications on the same host arrive, but to send from another container or another host you set dogstatsd_non_local_traffic to true in datadog.yaml or set the environment variable DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true. Many of the reports of custom metrics not arriving in container environments come down to this. At the same time, the official documentation asks you to restrict the sources that can reach the Agent so that it cannot be reached from untrusted networks.

Proxy configuration

On hosts that cannot reach the outside directly, you make the Agent send through a proxy. It is configured in the proxy section of datadog.yaml (http / https / no_proxy) or with environment variables such as DD_PROXY_HTTPS, and the environment variables take precedence. The Agent must be restarted after a change. no_proxy uses exact matching by default, so setting no_proxy_nonexact_match to true is recommended. Squid is the recommended proxy; HAProxy and NGINX require manual maintenance of the domain list and are not recommended.

How Autodiscovery works

Containers move between hosts and their IPs change, so static check configurations cannot keep up with monitoring. Autodiscovery is a mechanism in which you define check templates and the identifiers of their targets in advance, and the Agent watches container creation, start, stop, and destruction events and, for each matching container, generates, enables, and disables a static configuration in which the template variables are replaced with actual values. The Agent automatically detects the Docker, containerd, and Kubernetes API sockets and enables itself accordingly, and you can check the state with agent status. Auto-configuration templates for Apache and Redis are bundled.

Where to place integration templates

On Kubernetes, you write the check configuration as a single JSON in the pod annotation ad.datadoghq.com/<CONTAINER_NAME>.checks (the v2 format for 7.36 and later), and logs in .logs. On Docker, you use the com.datadoghq.ad.checks label. Other options are writing a file with ad_identifiers in conf.d/<NAME>.d/conf.yaml mounted into the Agent, ConfigMaps, and key-value stores. The file method requires restarting the Agent container on every change.

ad_identifiers and template variables

ad_identifiers are the identifiers of the target containers, and by default they are matched against the short image name (for example httpd). The short name does not distinguish registry or tag, so it matches both foo/httpd:latest and bar/httpd:v2. To apply different configurations to the same image, attach a custom identifier with a Docker label such as com.datadoghq.ad.check.id, which takes precedence over the image name. As for variables, %%host%% is the container IP, %%port%% is the highest exposed port in ascending numerical order, and %%env_<ENV_VAR>%% is an environment variable as seen by the Agent.

Including and excluding containers

By default, all containers are monitored. To narrow the scope, write regular expressions on name / image / kube_namespace in the environment variables DD_CONTAINER_EXCLUDE / DD_CONTAINER_INCLUDE. Because include takes precedence over exclude, the standard approach is to exclude everything with image:.* and include only the images you need. However, the pod annotation ad.datadoghq.com/exclude: true is stronger than include, and a global exclusion cannot be undone by an individual include. Pause containers are excluded by default.

Check your understanding

Check what you have learned with 5 questions