obsdog

Nvidia DCGM Exporter integration metrics

The official Datadog integration for monitoring Nvidia DCGM Exporter. It collects 45 metrics.

Official summary

Monitors the exposed GPU metrics leveraged by the Nvidia DCGM Exporter

Metric types

  • gauge (value at a point in time): 37
  • count (occurrences in the interval): 8

Metrics

Descriptions are the official definitions, unedited.

MetricTypeUnit
dcgm.clock_throttle_reasonsCurrent clock throttle reasons (bitmask of DCGM_CLOCKS_THROTTLE_REASON_*)gauge-
dcgm.correctable_remapped_rows.countNumber of remapped rows for correctable errors.countrow
dcgm.dec_utilizationDecoder utilization (in %).gaugepercent
dcgm.device.countChange in number Devices on the node.countdevice
dcgm.device.totalNumber of Devices on the node.gauge-
dcgm.dram.activeRatio of cycles the device memory interface is active sending or receiving data (in %).gaugefraction
dcgm.enc_utilizationEncoder utilization (in %).gaugepercent
dcgm.fan_speedFan speed for the device in percent 0-100.gaugepercent
dcgm.frame_buffer.freeFree Frame Buffer in MB.gaugemegabyte
dcgm.frame_buffer.reservedReserved Frame Buffer in MB.gaugemegabyte
dcgm.frame_buffer.totalTotal Frame Buffer of the GPU in MB.gaugemegabyte
dcgm.frame_buffer.usedUsed Frame Buffer in MB.gaugemegabyte
dcgm.frame_buffer.used_percentPercentage used of Frame Buffer: Used/(Total - Reserved). Range 0.0-1.0gaugefraction
dcgm.gpu_utilizationGPU utilization (in %).gaugepercent
dcgm.gr_engine_activeRatio of time the graphics engine is active (in %).gaugefraction
dcgm.mem.clockMemory clock frequency (in MHz).gaugemegahertz
dcgm.mem.copy_utilizationMemory utilization (in %).gaugepercent
dcgm.mem.temperatureMemory temperature (in C).gaugedegree celsius
dcgm.nvlink_bandwidth.countChange in number of NVLink bandwidth counters for all lanescount-
dcgm.nvlink_bandwidth.totalTotal number of NVLink bandwidth counters for all lanesgauge-
dcgm.pcie_replay.countChange in number of PCIe retries.count-
dcgm.pcie_replay.totalTotal number of PCIe retries.gauge-
dcgm.pcie_rx_throughput.countChange in PCIe Rx utilization information.count-
dcgm.pcie_rx_throughput.totalPCIe Rx utilization information.gauge-
dcgm.pcie_tx_throughput.countChange PCIe Tx utilization information.count-
dcgm.pcie_tx_throughput.totalPCIe Tx utilization informationgauge-
dcgm.pipe.fp16_activeRatio of cycles the fp16 pipes are active (in %).gaugefraction
dcgm.pipe.fp32_activeRatio of cycles the fp32 pipes are active (in %).gaugefraction
dcgm.pipe.fp64_activeRatio of cycles the fp64 pipes are active (in %).gaugefraction
dcgm.pipe.tensor_activeRatio of cycles the tensor (HMMA) pipe is active (in %).gaugefraction
dcgm.power_management_limitCurrent power limit for the device.gaugewatt
dcgm.power_usagePower draw (in W).gaugewatt
dcgm.pstatePerformance state (P-State) 0-15. 0=highestgauge-
dcgm.row_remap_failureWhether remapping of rows has failed.gauge-
dcgm.slowdown_temperatureSlowdown temperature for the device.gaugedegree celsius
dcgm.sm_activeThe ratio of cycles an SM has at least 1 warp assigned (in %).gaugefraction
dcgm.sm_clockSM clock frequency (in MHz).gaugemegahertz
dcgm.sm_occupancyThe ratio of number of warps resident on an SM (in %).gaugefraction
dcgm.temperatureGPU temperature (in C).gaugedegree celsius
dcgm.total_energy_consumption.countChange in energy consumption (in mJ).countmillijoule
dcgm.total_energy_consumption.totalTotal energy consumption since boot (in mJ)gauge-
dcgm.uncorrectable_remapped_rows.countChange in number of remapped rows for uncorrectable errors.countrow
dcgm.uncorrectable_remapped_rows.totalTotal number of remapped rows for uncorrectable errors.gauge-
dcgm.vgpu_license_statusvGPU License statusgauge-
dcgm.xid_errorsValue of the last XID error encountered.gauge-

Official documentation

Official documentation for the Nvidia DCGM Exporter integration

Metric names and descriptions are the official Datadog integration definitions (Copyright Datadog, Inc., BSD-3-Clause), reproduced unedited. The full license terms are on the About page.

Related products

IntegrationsIntegrations are the bundle of entry points that pour data from external systems such as AWS and MySQL into Datadog. Where you configure them splits into two families, the Agent side and the cloud account side, and which one you choose changes both what you can see and how billing is counted.Foundation and data collectionLLM ObservabilityLLM Observability records a single request to an application with an embedded LLM as a trace, tracks the inputs and outputs, tokens, latency, and errors of each model call, and turns response quality into numbers through evaluations. The only thing billed is the LLM span, which represents a model call.APM and tracingDatadog AgentSoftware that runs resident on the servers and containers you monitor and delivers metrics, logs, and traces to Datadog.Foundation and data collectionMetricsFrom CPU utilization to user sign-ups, metrics are the core data of Datadog, recording every number in your environment as a time series. If you understand how they are counted before how they are collected, you avoid aggregation mistakes and unexpected charges.Foundation and data collection

Other AI/ML integrations

Hugging Face TGI54 metricsKubeflow48 metricsLangChain6 metricsMicrosoft Copilot21 metricsMilvus386 metricsNvidia NIM34 metricsNvidia Triton42 metricsOpenAI29 metricsRay146 metricsSGLang99 metricsTorchServe33 metricsvLLM62 metrics