Ray integration metrics

The official Datadog integration for monitoring Ray. It collects 146 metrics.

Definitions as of 2026-08-05

Official summary

Monitor the health and performance of Ray

Metric types

gauge (value at a point in time): 100
count (occurrences in the interval): 46

Metrics

Descriptions are the official definitions, unedited.

Metric	Type	Unit
`ray.actors`Current number of actors currently in a particular state.	gauge	-
`ray.cluster.active_nodes`Active nodes on the cluster	gauge	node
`ray.cluster.failed_nodes`Failed nodes on the cluster	gauge	node
`ray.cluster.pending_nodes`Pending nodes on the cluster	gauge	node
`ray.component.cpu_percentage`Total CPU usage of the components on a node.	gauge	percent
`ray.component.mem_shared`SHM usage of all components of the node. It is equivalent to the top command's SHR column.	gauge	byte
`ray.component.rss`RSS usage of all components on the node.	gauge	megabyte
`ray.component.uss`USS usage of all components on the node.	gauge	megabyte
`ray.gcs.actors`Number of actors per state {Created, Destroyed, Unresolved, Pending}	gauge	-
`ray.gcs.placement_group`Number of placement groups broken down by state in {Registered, Pending, Infeasible}	gauge	-
`ray.gcs.storage_operation.count`Number of operations invoked on Gcs storage	count	-
`ray.gcs.storage_operation.latency.bucket`Time to invoke an operation on Gcs storage	count	millisecond
`ray.gcs.storage_operation.latency.count`Time to invoke an operation on Gcs storage	count	-
`ray.gcs.storage_operation.latency.sum`Time to invoke an operation on Gcs storage	count	millisecond
`ray.gcs.task_manager.task_events.dropped`Number of task events dropped per type {PROFILE_EVENT, STATUS_EVENT}	gauge	event
`ray.gcs.task_manager.task_events.reported`Number of all task events reported to gcs.	gauge	event
`ray.gcs.task_manager.task_events.stored`Number of task events stored in GCS.	gauge	event
`ray.gcs.task_manager.task_events.stored_bytes`Number of bytes of all task events stored in GCS.	gauge	byte
`ray.grpc_server.req.finished.count`Finished request number in grpc server	count	request
`ray.grpc_server.req.handling.count`Request number are handling in grpc server	count	request
`ray.grpc_server.req.new.count`New request number in grpc server	count	request
`ray.grpc_server.req.process_time`Request latency in grpc server	gauge	millisecond
`ray.health_check.rpc_latency.bucket`Latency of rpc request for health check.	count	millisecond
`ray.health_check.rpc_latency.count`Latency of rpc request for health check.	count	-
`ray.health_check.rpc_latency.sum`Latency of rpc request for health check.	count	millisecond
`ray.internal_num.infeasible_scheduling_classes`The number of unique scheduling classes that are infeasible.	gauge	-
`ray.internal_num.processes.skipped.job_mismatch`The total number of cached workers skipped due to job mismatch.	gauge	process
`ray.internal_num.processes.skipped.runtime_environment_mismatch`The total number of cached workers skipped due to runtime environment mismatch.	gauge	process
`ray.internal_num.processes.started`The total number of worker processes the worker pool has created.	gauge	process
`ray.internal_num.processes.started.from_cache`The total number of workers started from a cached worker process.	gauge	process
`ray.internal_num.spilled_tasks`The cumulative number of lease requests that this raylet has spilled to other raylets.	gauge	request
`ray.memory_manager.worker_eviction`The number of tasks and actors killed by the Ray Out of Memory killer broken down by types (whether it is tasks or actors) and names (name of tasks and actors).	count	-
`ray.node.cpu`Total CPUs available on a ray node	gauge	-
`ray.node.cpu_utilization`Total CPU usage on a ray node	gauge	-
`ray.node.disk.free`Total disk free (bytes) on a ray node	gauge	byte
`ray.node.disk.io.read`Total read from disk	gauge	-
`ray.node.disk.io.read.count`Total read ops from disk	gauge	operation
`ray.node.disk.io.read.speed`Disk read speed	gauge	-
`ray.node.disk.io.write`Total written to disk	gauge	-
`ray.node.disk.io.write.count`Total write ops to disk	gauge	-
`ray.node.disk.io.write.speed`Disk write speed	gauge	-
`ray.node.disk.read.iops`Disk read iops	gauge	-
`ray.node.disk.usage`Total disk usage (bytes) on a ray node	gauge	byte
`ray.node.disk.utilization`Total disk utilization (percentage) on a ray node	gauge	percent
`ray.node.disk.write.iops`Disk write iops	gauge	-
`ray.node.gpus_utilization`The GPU utilization per GPU as a percentage quantity (0..NGPU*100). GpuDeviceName is a name of a GPU device (e.g., Nvidia A10G) and GpuIndex is the index of the GPU.	gauge	percent
`ray.node.gram_used`The amount of GPU memory used per GPU, in bytes.	gauge	byte
`ray.node.mem.available`Memory available on a ray node	gauge	byte
`ray.node.mem.shared`Total shared memory usage on a ray node	gauge	byte
`ray.node.mem.total`Total memory on a ray node	gauge	byte
`ray.node.mem.used`Memory usage on a ray node	gauge	byte
`ray.node.network.receive.speed`Network receive speed	gauge	-
`ray.node.network.received`Total network received	gauge	-
`ray.node.network.send.speed`Network send speed	gauge	-
`ray.node.network.sent`Total network sent	gauge	-
`ray.object_directory.added_locations`Number of object locations added per second., If this is high, a lot of objects have been added on this node.	gauge	-
`ray.object_directory.lookups`Number of object location lookups per second. If this is high, the raylet is waiting on a lot of objects.	gauge	-
`ray.object_directory.removed_locations`Number of object locations removed per second. If this is high, a lot of objects have been removed from this node.	gauge	-
`ray.object_directory.subscriptions`Number of object location subscriptions. If this is high, the raylet is attempting to pull a lot of objects.	gauge	-
`ray.object_directory.updates`Number of object location updates per second., If this is high, the raylet is attempting to pull a lot of objects and/or the locations for objects are frequently changing (e.g. due to many object copies or evictions).	gauge	update/second
`ray.object_manager.bytes`Number of bytes pushed or received by type {PushedFromLocalPlasma, PushedFromLocalDisk, Received}.	gauge	byte
`ray.object_manager.num_pull_requests`Number of active pull requests for objects.	gauge	-
`ray.object_manager.received_chunks`Number object chunks received broken per type {Total, FailedTotal, FailedCancelled, FailedPlasmaFull}.	gauge	-
`ray.object_store.available_memory`Amount of memory currently available in the object store.	gauge	byte
`ray.object_store.fallback_memory`Amount of memory in fallback allocations in the filesystem.	gauge	byte
`ray.object_store.memory`Object store memory by various sub-kinds on this node	gauge	byte
`ray.object_store.num_local_objects`Number of objects currently in the object store.	gauge	object
`ray.object_store.size.bucket`The distribution of object size in bytes	count	byte
`ray.object_store.size.count`The distribution of object size in bytes	count	-
`ray.object_store.size.sum`The distribution of object size in bytes	count	byte
`ray.object_store.used_memory`Amount of memory currently occupied in the object store.	gauge	byte
`ray.placement_groups`Current number of placement groups by state. The State label (e.g., PENDING, CREATED, REMOVED) describes the state of the placement group.	gauge	-
`ray.process.cpu_seconds.count`Total user and system CPU time spent in seconds.	count	second
`ray.process.max_fds`Maximum number of open file descriptors.	gauge	file
`ray.process.open_fds`Number of open file descriptors.	gauge	file
`ray.process.resident_memory`Resident memory size in bytes.	gauge	byte
`ray.process.start_time`Start time of the process since unix epoch in seconds.	gauge	second
`ray.process.virtual_memory`Virtual memory size in bytes.	gauge	byte
`ray.pull_manager.active_bundles`Number of active bundle requests	gauge	request
`ray.pull_manager.num_object_pins`Number of object pin attempts by the pull manager, can be {Success, Failure}.	gauge	attempt
`ray.pull_manager.object_request_time.bucket`Time between initial object pull request and local pinning of the object.	count	millisecond
`ray.pull_manager.object_request_time.count`Time between initial object pull request and local pinning of the object.	count	-
`ray.pull_manager.object_request_time.sum`Time between initial object pull request and local pinning of the object.	count	millisecond
`ray.pull_manager.requested_bundles`Number of requested bundles broken per type {Get, Wait, TaskArgs}.	gauge	-
`ray.pull_manager.requests`Number of pull requests broken per type {Queued, Active, Pinned}.	gauge	request
`ray.pull_manager.retries_total`Number of cumulative pull retries.	gauge	-
`ray.pull_manager.usage`The total number of bytes usage broken per type {Available, BeingPulled, Pinned}	gauge	byte
`ray.push_manager.chunks`Number of object chunks transfer broken per type {InFlight, Remaining}.	gauge	-
`ray.push_manager.in_flight_pushes`Number of in flight object push requests.	gauge	request
`ray.python.gc.collections.count`Number of times this generation was collected	count	-
`ray.python.gc.objects_collected.count`Objects collected during gc	count	object
`ray.python.gc.objects_uncollectable.count`Uncollectable objects found during GC	count	object
`ray.resources`Logical Ray resources broken per state {AVAILABLE, USED}	gauge	resource
`ray.scheduler.failed_worker_startup`Number of tasks that fail to be scheduled because workers were not available. Labels are broken per reason {JobConfigMissing, RegistrationTimedOut, RateLimited}	gauge	task
`ray.scheduler.placement_time.bucket`The time it takes for a workload (task, actor, placement group) to be placed. This is the time from when the tasks dependencies are resolved to when it actually reserves resources on a node to run.	count	second
`ray.scheduler.placement_time.count`The time it takes for a workload (task, actor, placement group) to be placed. This is the time from when the tasks dependencies are resolved to when it actually reserves resources on a node to run.	count	-
`ray.scheduler.placement_time.sum`The time it takes for a workload (task, actor, placement group) to be placed. This is the time from when the tasks dependencies are resolved to when it actually reserves resources on a node to run.	count	second
`ray.scheduler.tasks`Number of tasks waiting for scheduling broken per state {Cancelled, Executing, Waiting, Dispatched, Received}.	gauge	task
`ray.scheduler.unscheduleable_tasks`Number of pending tasks (not scheduleable tasks) broken per reason {Infeasible, WaitingForResources, WaitingForPlasmaMemory, WaitingForRemoteResources, WaitingForWorkers}.	gauge	task
`ray.serve.deployment.error`The number of exceptions that have occurred in this replica.	gauge	exception
`ray.serve.deployment.processing_latency.bucket`The latency for queries to be processed.	count	millisecond
`ray.serve.deployment.processing_latency.count`The latency for queries to be processed.	count	-
`ray.serve.deployment.processing_latency.sum`The latency for queries to be processed.	count	millisecond
`ray.serve.deployment.queued_queries`The current number of queries to this deployment waiting to be assigned to a replica.	gauge	query
`ray.serve.deployment.replica.healthy`Tracks whether this deployment replica is healthy. 1 means healthy, 0 means unhealthy.	gauge	-
`ray.serve.deployment.replica.starts`The number of times this replica has been restarted due to failure.	gauge	-
`ray.serve.deployment.request.counter`The number of queries that have been processed in this replica.	gauge	query
`ray.serve.grpc_request_latency.bucket`The end-to-end latency of GRPC requests (measured from the Serve GRPC proxy).	count	-
`ray.serve.grpc_request_latency.count`The end-to-end latency of GRPC requests (measured from the Serve GRPC proxy).	count	-
`ray.serve.grpc_request_latency.sum`The end-to-end latency of GRPC requests (measured from the Serve GRPC proxy).	count	-
`ray.serve.handle_request`The number of handle.remote() calls that have been made on this handle.	gauge	request
`ray.serve.http_request_latency.bucket`The end-to-end latency of HTTP requests (measured from the Serve HTTP proxy).	count	millisecond
`ray.serve.http_request_latency.count`The end-to-end latency of HTTP requests (measured from the Serve HTTP proxy).	count	-
`ray.serve.http_request_latency.sum`The end-to-end latency of HTTP requests (measured from the Serve HTTP proxy).	count	millisecond
`ray.serve.multiplexed_get_model_requests.count`The counter for get model requests on the current replica.	count	-
`ray.serve.multiplexed_model_load_latency.bucket`The time it takes to load a model.	count	millisecond
`ray.serve.multiplexed_model_load_latency.count`The time it takes to load a model.	count	-
`ray.serve.multiplexed_model_load_latency.sum`The time it takes to load a model.	count	millisecond
`ray.serve.multiplexed_model_unload_latency.bucket`The time it takes to unload a model.	count	millisecond
`ray.serve.multiplexed_model_unload_latency.count`The time it takes to unload a model.	count	-
`ray.serve.multiplexed_model_unload_latency.sum`The time it takes to unload a model.	count	millisecond
`ray.serve.multiplexed_models_load.count`The counter for loaded models on the current replica.	count	-
`ray.serve.multiplexed_models_unload.count`The counter for unloaded models on the current replica.	count	-
`ray.serve.num_deployment_grpc_error_requests`The number of errored GRPC responses returned by each deployment.	gauge	-
`ray.serve.num_deployment_http_error_requests`The number of non-200 HTTP responses returned by each deployment.	gauge	response
`ray.serve.num_grpc_error_requests`The number of errored GRPC responses.	gauge	-
`ray.serve.num_grpc_requests`The number of GRPC responses.	gauge	-
`ray.serve.num_http_error_requests`The number of non-200 HTTP responses.	gauge	response
`ray.serve.num_http_requests`The number of HTTP requests processed.	gauge	request
`ray.serve.num_multiplexed_models`The number of models loaded on the current replica.	gauge	-
`ray.serve.num_router_requests`The number of requests processed by the router.	gauge	request
`ray.serve.registered_multiplexed_model_id`The model id registered on the current replica.	gauge	-
`ray.serve.replica.pending_queries`The current number of pending queries.	gauge	query
`ray.serve.replica.processing_queries`The current number of queries being processed.	gauge	query
`ray.server.num_ongoing_grpc_requests`The number of ongoing requests in this GRPC proxy.	gauge	-
`ray.server.num_ongoing_http_requests`The number of ongoing requests in this HTTP proxy.	gauge	-
`ray.server.num_scheduling_tasks`The number of request scheduling tasks in the router.	gauge	-
`ray.server.num_scheduling_tasks_in_backoff`The number of request scheduling tasks in the router that are undergoing backoff.	gauge	-
`ray.spill_manager.objects`Number of local objects broken per state {Pinned, PendingRestore, PendingSpill}.	gauge	object
`ray.spill_manager.objects_size`Byte size of local objects broken per state {Pinned, PendingSpill}.	gauge	byte
`ray.spill_manager.request_total`Number of {spill, restore} requests.	gauge	request
`ray.tasks`Current number of tasks currently in a particular state.	gauge	task
`ray.unintentional_worker_failures.count`Number of worker failures that are not intentional. For example, worker failures due to system related errors.	count	error
`ray.worker.register_time.bucket`End to end latency of register a worker process.	count	millisecond
`ray.worker.register_time.count`End to end latency of register a worker process.	count	-
`ray.worker.register_time.sum`End to end latency of register a worker process.	count	millisecond

Official documentation

Official documentation for the Ray integration

Metric names and descriptions are the official Datadog integration definitions (Copyright Datadog, Inc., BSD-3-Clause), reproduced unedited. The full license terms are on the About page.

Official summary

Metric types

Metrics

Official documentation

Related products

Other AI/ML integrations