Ray インテグレーションのメトリクス
Ray の監視に使う Datadog の公式インテグレーションです。収集するメトリクスは 146 件あります。
公式の紹介文 (英語)
Monitor the health and performance of Ray
種別の内訳
- gauge (その時点の値): 100 件
- count (集計間隔内の発生回数): 46 件
メトリクスの一覧
説明は公式の定義の英文をそのまま載せています。
| メトリクス | 種別 | 単位 |
|---|---|---|
ray.Current number of actors currently in a particular state. | gauge | - |
ray.Active nodes on the cluster | gauge | node |
ray.Failed nodes on the cluster | gauge | node |
ray.Pending nodes on the cluster | gauge | node |
ray.Total CPU usage of the components on a node. | gauge | percent |
ray.SHM usage of all components of the node. It is equivalent to the top command's SHR column. | gauge | byte |
ray.RSS usage of all components on the node. | gauge | megabyte |
ray.USS usage of all components on the node. | gauge | megabyte |
ray.Number of actors per state {Created, Destroyed, Unresolved, Pending} | gauge | - |
ray.Number of placement groups broken down by state in {Registered, Pending, Infeasible} | gauge | - |
ray.Number of operations invoked on Gcs storage | count | - |
ray.Time to invoke an operation on Gcs storage | count | millisecond |
ray.Time to invoke an operation on Gcs storage | count | - |
ray.Time to invoke an operation on Gcs storage | count | millisecond |
ray.Number of task events dropped per type {PROFILE_EVENT, STATUS_EVENT} | gauge | event |
ray.Number of all task events reported to gcs. | gauge | event |
ray.Number of task events stored in GCS. | gauge | event |
ray.Number of bytes of all task events stored in GCS. | gauge | byte |
ray.Finished request number in grpc server | count | request |
ray.Request number are handling in grpc server | count | request |
ray.New request number in grpc server | count | request |
ray.Request latency in grpc server | gauge | millisecond |
ray.Latency of rpc request for health check. | count | millisecond |
ray.Latency of rpc request for health check. | count | - |
ray.Latency of rpc request for health check. | count | millisecond |
ray.The number of unique scheduling classes that are infeasible. | gauge | - |
ray.The total number of cached workers skipped due to job mismatch. | gauge | process |
ray.The total number of cached workers skipped due to runtime environment mismatch. | gauge | process |
ray.The total number of worker processes the worker pool has created. | gauge | process |
ray.The total number of workers started from a cached worker process. | gauge | process |
ray.The cumulative number of lease requests that this raylet has spilled to other raylets. | gauge | request |
ray.The number of tasks and actors killed by the Ray Out of Memory killer broken down by types (whether it is tasks or actors) and names (name of tasks and actors). | count | - |
ray.Total CPUs available on a ray node | gauge | - |
ray.Total CPU usage on a ray node | gauge | - |
ray.Total disk free (bytes) on a ray node | gauge | byte |
ray.Total read from disk | gauge | - |
ray.Total read ops from disk | gauge | operation |
ray.Disk read speed | gauge | - |
ray.Total written to disk | gauge | - |
ray.Total write ops to disk | gauge | - |
ray.Disk write speed | gauge | - |
ray.Disk read iops | gauge | - |
ray.Total disk usage (bytes) on a ray node | gauge | byte |
ray.Total disk utilization (percentage) on a ray node | gauge | percent |
ray.Disk write iops | gauge | - |
ray.The GPU utilization per GPU as a percentage quantity (0..NGPU*100). GpuDeviceName is a name of a GPU device (e.g., Nvidia A10G) and GpuIndex is the index of the GPU. | gauge | percent |
ray.The amount of GPU memory used per GPU, in bytes. | gauge | byte |
ray.Memory available on a ray node | gauge | byte |
ray.Total shared memory usage on a ray node | gauge | byte |
ray.Total memory on a ray node | gauge | byte |
ray.Memory usage on a ray node | gauge | byte |
ray.Network receive speed | gauge | - |
ray.Total network received | gauge | - |
ray.Network send speed | gauge | - |
ray.Total network sent | gauge | - |
ray.Number of object locations added per second., If this is high, a lot of objects have been added on this node. | gauge | - |
ray.Number of object location lookups per second. If this is high, the raylet is waiting on a lot of objects. | gauge | - |
ray.Number of object locations removed per second. If this is high, a lot of objects have been removed from this node. | gauge | - |
ray.Number of object location subscriptions. If this is high, the raylet is attempting to pull a lot of objects. | gauge | - |
ray.Number of object location updates per second., If this is high, the raylet is attempting to pull a lot of objects and/or the locations for objects are frequently changing (e.g. due to many object copies or evictions). | gauge | update/ |
ray.Number of bytes pushed or received by type {PushedFromLocalPlasma, PushedFromLocalDisk, Received}. | gauge | byte |
ray.Number of active pull requests for objects. | gauge | - |
ray.Number object chunks received broken per type {Total, FailedTotal, FailedCancelled, FailedPlasmaFull}. | gauge | - |
ray.Amount of memory currently available in the object store. | gauge | byte |
ray.Amount of memory in fallback allocations in the filesystem. | gauge | byte |
ray.Object store memory by various sub-kinds on this node | gauge | byte |
ray.Number of objects currently in the object store. | gauge | object |
ray.The distribution of object size in bytes | count | byte |
ray.The distribution of object size in bytes | count | - |
ray.The distribution of object size in bytes | count | byte |
ray.Amount of memory currently occupied in the object store. | gauge | byte |
ray.Current number of placement groups by state. The State label (e.g., PENDING, CREATED, REMOVED) describes the state of the placement group. | gauge | - |
ray.Total user and system CPU time spent in seconds. | count | second |
ray.Maximum number of open file descriptors. | gauge | file |
ray.Number of open file descriptors. | gauge | file |
ray.Resident memory size in bytes. | gauge | byte |
ray.Start time of the process since unix epoch in seconds. | gauge | second |
ray.Virtual memory size in bytes. | gauge | byte |
ray.Number of active bundle requests | gauge | request |
ray.Number of object pin attempts by the pull manager, can be {Success, Failure}. | gauge | attempt |
ray.Time between initial object pull request and local pinning of the object. | count | millisecond |
ray.Time between initial object pull request and local pinning of the object. | count | - |
ray.Time between initial object pull request and local pinning of the object. | count | millisecond |
ray.Number of requested bundles broken per type {Get, Wait, TaskArgs}. | gauge | - |
ray.Number of pull requests broken per type {Queued, Active, Pinned}. | gauge | request |
ray.Number of cumulative pull retries. | gauge | - |
ray.The total number of bytes usage broken per type {Available, BeingPulled, Pinned} | gauge | byte |
ray.Number of object chunks transfer broken per type {InFlight, Remaining}. | gauge | - |
ray.Number of in flight object push requests. | gauge | request |
ray.Number of times this generation was collected | count | - |
ray.Objects collected during gc | count | object |
ray.Uncollectable objects found during GC | count | object |
ray.Logical Ray resources broken per state {AVAILABLE, USED} | gauge | resource |
ray.Number of tasks that fail to be scheduled because workers were not available. Labels are broken per reason {JobConfigMissing, RegistrationTimedOut, RateLimited} | gauge | task |
ray.The time it takes for a workload (task, actor, placement group) to be placed. This is the time from when the tasks dependencies are resolved to when it actually reserves resources on a node to run. | count | second |
ray.The time it takes for a workload (task, actor, placement group) to be placed. This is the time from when the tasks dependencies are resolved to when it actually reserves resources on a node to run. | count | - |
ray.The time it takes for a workload (task, actor, placement group) to be placed. This is the time from when the tasks dependencies are resolved to when it actually reserves resources on a node to run. | count | second |
ray.Number of tasks waiting for scheduling broken per state {Cancelled, Executing, Waiting, Dispatched, Received}. | gauge | task |
ray.Number of pending tasks (not scheduleable tasks) broken per reason {Infeasible, WaitingForResources, WaitingForPlasmaMemory, WaitingForRemoteResources, WaitingForWorkers}. | gauge | task |
ray.The number of exceptions that have occurred in this replica. | gauge | exception |
ray.The latency for queries to be processed. | count | millisecond |
ray.The latency for queries to be processed. | count | - |
ray.The latency for queries to be processed. | count | millisecond |
ray.The current number of queries to this deployment waiting to be assigned to a replica. | gauge | query |
ray.Tracks whether this deployment replica is healthy. 1 means healthy, 0 means unhealthy. | gauge | - |
ray.The number of times this replica has been restarted due to failure. | gauge | - |
ray.The number of queries that have been processed in this replica. | gauge | query |
ray.The end-to-end latency of GRPC requests (measured from the Serve GRPC proxy). | count | - |
ray.The end-to-end latency of GRPC requests (measured from the Serve GRPC proxy). | count | - |
ray.The end-to-end latency of GRPC requests (measured from the Serve GRPC proxy). | count | - |
ray.The number of handle.remote() calls that have been made on this handle. | gauge | request |
ray.The end-to-end latency of HTTP requests (measured from the Serve HTTP proxy). | count | millisecond |
ray.The end-to-end latency of HTTP requests (measured from the Serve HTTP proxy). | count | - |
ray.The end-to-end latency of HTTP requests (measured from the Serve HTTP proxy). | count | millisecond |
ray.The counter for get model requests on the current replica. | count | - |
ray.The time it takes to load a model. | count | millisecond |
ray.The time it takes to load a model. | count | - |
ray.The time it takes to load a model. | count | millisecond |
ray.The time it takes to unload a model. | count | millisecond |
ray.The time it takes to unload a model. | count | - |
ray.The time it takes to unload a model. | count | millisecond |
ray.The counter for loaded models on the current replica. | count | - |
ray.The counter for unloaded models on the current replica. | count | - |
ray.The number of errored GRPC responses returned by each deployment. | gauge | - |
ray.The number of non-200 HTTP responses returned by each deployment. | gauge | response |
ray.The number of errored GRPC responses. | gauge | - |
ray.The number of GRPC responses. | gauge | - |
ray.The number of non-200 HTTP responses. | gauge | response |
ray.The number of HTTP requests processed. | gauge | request |
ray.The number of models loaded on the current replica. | gauge | - |
ray.The number of requests processed by the router. | gauge | request |
ray.The model id registered on the current replica. | gauge | - |
ray.The current number of pending queries. | gauge | query |
ray.The current number of queries being processed. | gauge | query |
ray.The number of ongoing requests in this GRPC proxy. | gauge | - |
ray.The number of ongoing requests in this HTTP proxy. | gauge | - |
ray.The number of request scheduling tasks in the router. | gauge | - |
ray.The number of request scheduling tasks in the router that are undergoing backoff. | gauge | - |
ray.Number of local objects broken per state {Pinned, PendingRestore, PendingSpill}. | gauge | object |
ray.Byte size of local objects broken per state {Pinned, PendingSpill}. | gauge | byte |
ray.Number of {spill, restore} requests. | gauge | request |
ray.Current number of tasks currently in a particular state. | gauge | task |
ray.Number of worker failures that are not intentional. For example, worker failures due to system related errors. | count | error |
ray.End to end latency of register a worker process. | count | millisecond |
ray.End to end latency of register a worker process. | count | - |
ray.End to end latency of register a worker process. | count | millisecond |
公式ドキュメント
メトリクスの名前と説明は Datadog の公式インテグレーションの定義 (Copyright Datadog, Inc.・BSD-3-Clause) を英文のまま載せています。利用条件の全文は サイトについてにあります。