Slurm integration metrics
The official Datadog integration for monitoring Slurm. It collects 87 metrics.
Official summary
Monitor Slurm cluster resource usage, job statuses, and system performance.
Metric types
- gauge (value at a point in time): 87
Metrics
Descriptions are the official definitions, unedited.
| Metric | Type | Unit |
|---|---|---|
slurm.Number of megabytes allocated on the node. | gauge | megabyte |
slurm.Number of CPUs allocated on the node for job-related tasks. | gauge | cpu |
slurm.Number of idle CPUs on the node. | gauge | cpu |
slurm.Number of CPUs performing other or non-job-related tasks on the node. | gauge | cpu |
slurm.Total number of CPUs on the node. | gauge | cpu |
slurm.CPU load on the node as reported by the OS. | gauge | - |
slurm.Free memory on the node as reported by the OS. | gauge | megabyte |
slurm.Total number of GPUs on the node. | gauge | - |
slurm.Number of GPUs used on the node. | gauge | - |
slurm.Information about the Slurm node. | gauge | - |
slurm.Total memory on the node as reported by the OS. | gauge | megabyte |
slurm.Temporary disk space on the node as reported by the OS. | gauge | megabyte |
slurm.(Deprecated) Number of CPUs allocated on the partition for job-related tasks. | gauge | cpu |
slurm.(Deprecated) Number of idle CPUs on the partition. | gauge | cpu |
slurm.(Deprecated) Number of CPUs performing other or non-job-related tasks on the partition. | gauge | cpu |
slurm.(Deprecated) Total number of CPUs on the partition. | gauge | cpu |
slurm.Total number of GPUs on the partition. | gauge | - |
slurm.Number of GPUs used on the partition. | gauge | - |
slurm.Information about the Slurm partition. | gauge | - |
slurm.Number of nodes allocated on the partition for job-related tasks. | gauge | node |
slurm.Number of idle nodes on the partition. | gauge | node |
slurm.Number of nodes performing other or non-job-related tasks on the partition. | gauge | node |
slurm.Total number of nodes on the partition. | gauge | node |
slurm.Number of nodes in the partition. | gauge | node |
slurm.Shows whether we're collecting sacct metrics or not for this host. | gauge | - |
slurm.Duration of the job in seconds. | gauge | second |
slurm.Information about the Slurm job in sacct. | gauge | - |
slurm.Average number of bytes read from disk by a job or step. | gauge | byte |
slurm.Average (system + user) CPU time of all tasks in job. | gauge | second |
slurm.Average resident set size of all tasks in job. | gauge | byte |
slurm.Time used (Elapsed time * CPU count) by a job or step in cpu-seconds. | gauge | second |
slurm.Maximum number of bytes read from disk by a job or step. | gauge | byte |
slurm.Maximum resident set size of all tasks in job. | gauge | byte |
slurm.Maximum virtual memory size of all tasks in job. | gauge | byte |
slurm.Status of running jobs on worker node. | gauge | job |
slurm.Number of agent threads. | gauge | thread |
slurm.Number of enqueued outgoing RPC requests in an internal retry list. | gauge | request |
slurm.Total count of active threads created by all the agent threads. | gauge | thread |
slurm.Mean count of jobs processed during all backfilling scheduling cycles since last reset. | gauge | job |
slurm.The subset of Depth Mean that the backfill scheduler attempted to schedule. | gauge | job |
slurm.Time in microseconds of last backfill scheduling cycle. | gauge | microsecond |
slurm.Time in seconds since the last scheduling cycle. | gauge | second |
slurm.Number of processed jobs during last backfilling scheduling cycle. | gauge | job |
slurm.Number of processed jobs during last backfilling scheduling cycle. | gauge | job |
slurm.Number of jobs pending to be processed by backfilling algorithm. | gauge | job |
slurm.Number of different time slots tested by the backfill scheduler in its last iteration. | gauge | - |
slurm.Time in microseconds of maximum backfill scheduling cycle execution since last reset. | gauge | microsecond |
slurm.Mean time in microseconds of backfilling scheduling cycles since last reset. | gauge | microsecond |
slurm.Mean count of different time slots tested by the backfill scheduler. | gauge | - |
slurm.Mean count of jobs pending to be processed by backfilling algorithm. | gauge | job |
slurm.Number of backfill scheduling cycles since last reset. | gauge | - |
slurm.Number of heterogeneous job components started thanks to backfilling since last Slurm start. | gauge | - |
slurm.Total backfilled jobs since last stats cycle restart. | gauge | job |
slurm.Total backfilled jobs since last slurm restart. | gauge | job |
slurm.Scheduling executions per minute. | gauge | - |
slurm.DBD Agent message queue size for SlurmDBD. | gauge | message |
slurm.Shows whether we're collecting sdiag metrics or not for this host. | gauge | - |
slurm.Number of jobs canceled since last reset. | gauge | job |
slurm.Number of jobs completed since last reset. | gauge | job |
slurm.Number of jobs failed since last reset. | gauge | job |
slurm.Number of jobs pending since last reset. | gauge | job |
slurm.Number of jobs running since last reset. | gauge | job |
slurm.Number of jobs started since last reset. | gauge | job |
slurm.Number of jobs submitted since last reset. | gauge | job |
slurm.Time in microseconds for last scheduling cycle. | gauge | microsecond |
slurm.Length of jobs pending queue. | gauge | job |
slurm.Maximum time in microseconds for any scheduling cycle since last reset. | gauge | microsecond |
slurm.Mean time in microseconds for all scheduling cycles since last reset. | gauge | microsecond |
slurm.Mean of cycle depth. Depth means number of jobs processed in a scheduling cycle. | gauge | job |
slurm.The number of current active slurmctld threads. | gauge | thread |
slurm.The total run time in microseconds for all scheduling cycles since the last reset. | gauge | microsecond |
slurm.The CPU efficiency of the job. | gauge | percent |
slurm.The CPU utilized by the job. | gauge | second |
slurm.The memory efficiency of the job. | gauge | percent |
slurm.The memory utilized by the job. | gauge | megabyte |
slurm.The association's usage normalized with its parent. | gauge | - |
slurm.The Fair-Share factor, based on a user or account's assigned shares and the effective usage charged to them or their accounts. | gauge | - |
slurm.This is the association's fairshare value compared to its siblings,calculated as norm_shares / effective_usage. | gauge | - |
slurm.The shares assigned to the user or account normalized to the total number of assigned shares. | gauge | - |
slurm.The Raw Usage normalized to the total number of tres-seconds of all jobs run on the cluster. | gauge | - |
slurm.The raw shares assigned to the user or account. | gauge | - |
slurm.The number of tres-seconds (cpu-seconds if TRESBillingWeights is not defined) of all the jobs charged to the account or user. | gauge | - |
slurm.Shows whether we're collecting node metrics or not for this host. | gauge | - |
slurm.Shows whether we're collecting partition metrics or not for this host. | gauge | - |
slurm.Shows whether we're collecting squeue metrics or not for this host. | gauge | - |
slurm.Information about the Slurm job in squeue. | gauge | - |
slurm.Shows whether we're collecting sshare metrics or not for this host. | gauge | - |
Official documentation
Metric names and descriptions are the official Datadog integration definitions (Copyright Datadog, Inc., BSD-3-Clause), reproduced unedited. The full license terms are on the About page.