obsdog

vLLM integration metrics

The official Datadog integration for monitoring vLLM. It collects 62 metrics.

Official summary

vLLM is a library for LLM inference and serving

Metric types

  • count (occurrences in the interval): 48
  • gauge (value at a point in time): 14

Metrics

Descriptions are the official definitions, unedited.

MetricTypeUnit
vllm.avg.generation_throughput.toks_per_sAverage generation throughput in tokens/sgauge-
vllm.avg.prompt.throughput.toks_per_sAverage prefill throughput in tokens/sgauge-
vllm.cache_config_infoInformation on cache configgauge-
vllm.cpu_cache_usage_percCPU KV-cache usage. 1 means 100 percent usagegaugepercent
vllm.e2e_request_latency.seconds.bucketThe observations of end to end request latency bucketed by seconds.count-
vllm.e2e_request_latency.seconds.countThe total number of observations of end to end request latency.count-
vllm.e2e_request_latency.seconds.sumThe sum of end to end request latency in seconds.countsecond
vllm.estimated_flops_per_gpu.countEstimated floating-point operations per GPU.countoperation
vllm.external_prefix_cache.hits.countExternal prefix cache hits measured in cached tokens.countitem
vllm.external_prefix_cache.queries.countExternal prefix cache queries measured in queried tokens.countitem
vllm.generation_tokens.countNumber of generation tokens processed.count-
vllm.gpu_cache_usage_percGPU KV-cache usage. 1 means 100 percent usagegaugepercent
vllm.num_preemptions.countCumulative number of preemption from the engine.count-
vllm.num_requests.runningNumber of requests currently running on GPU.gauge-
vllm.num_requests.swappedNumber of requests swapped to CPU.gauge-
vllm.num_requests.waitingNumber of requests waiting.gauge-
vllm.process.cpu_seconds.countTotal user and system CPU time spent in seconds.countsecond
vllm.process.max_fdsMaximum number of open file descriptors.gaugefile
vllm.process.open_fdsNumber of open file descriptors.gaugefile
vllm.process.resident_memory_bytesResident memory size in bytes.gaugebyte
vllm.process.start_time_secondsStart time of the process since unix epoch in seconds.gaugesecond
vllm.process.virtual_memory_bytesVirtual memory size in bytes.gaugebyte
vllm.prompt_tokens.cached.countNumber of cached prompt tokens from local and external caches.countitem
vllm.prompt_tokens.countNumber of prefill tokens processed.count-
vllm.python.gc.collections.countNumber of times this generation was collectedcount-
vllm.python.gc.objects.collected.countObjects collected during gccount-
vllm.python.gc.objects.uncollectable.countUncollectable objects found during GCcount-
vllm.python.infoPython platform informationgauge-
vllm.request.decode_time.seconds.bucketRequest decode-time histogram buckets.count-
vllm.request.decode_time.seconds.countNumber of request decode-time observations.count-
vllm.request.decode_time.seconds.sumTotal request decode time.countsecond
vllm.request.generation_tokens.bucketNumber of generation tokens processed.count-
vllm.request.generation_tokens.countNumber of generation tokens processed.count-
vllm.request.generation_tokens.sumNumber of generation tokens processed.count-
vllm.request.params.best_of.bucketHistogram of the best_of request parameter.count-
vllm.request.params.best_of.countHistogram of the best_of request parameter.count-
vllm.request.params.best_of.sumHistogram of the best_of request parameter.count-
vllm.request.params.n.bucketHistogram of the n request parameter.count-
vllm.request.params.n.countHistogram of the n request parameter.count-
vllm.request.params.n.sumHistogram of the n request parameter.count-
vllm.request.prefill_kv_computed_tokens.bucketHistogram buckets for new KV tokens computed during prefill.count-
vllm.request.prefill_kv_computed_tokens.countNumber of prefill KV token observations.count-
vllm.request.prefill_kv_computed_tokens.sumTotal new KV tokens computed during prefill.countitem
vllm.request.prefill_time.seconds.bucketRequest prefill-time histogram buckets.count-
vllm.request.prefill_time.seconds.countNumber of request prefill-time observations.count-
vllm.request.prefill_time.seconds.sumTotal request prefill time.countsecond
vllm.request.prompt_tokens.bucketNumber of prefill tokens processed.count-
vllm.request.prompt_tokens.countNumber of prefill tokens processed.count-
vllm.request.prompt_tokens.sumNumber of prefill tokens processed.count-
vllm.request.queue_time.seconds.bucketRequest queue-time histogram buckets.count-
vllm.request.queue_time.seconds.countNumber of request queue-time observations.count-
vllm.request.queue_time.seconds.sumTotal request queue time.countsecond
vllm.request.success.countCount of successfully processed requests.count-
vllm.spec_decode.num_accepted_tokens.countNumber of speculative decode tokens accepted.countitem
vllm.spec_decode.num_accepted_tokens_per_pos.countNumber of accepted speculative decode tokens by draft position.countitem
vllm.spec_decode.num_draft_tokens.countNumber of speculative decode draft tokens.countitem
vllm.time_per_output_token.seconds.bucketThe observations of time per output token bucketed by seconds.count-
vllm.time_per_output_token.seconds.countThe total number of observations of time per output token.count-
vllm.time_per_output_token.seconds.sumThe sum of time per output token in seconds.countsecond
vllm.time_to_first_token.seconds.bucketThe observations of time to first token bucketed by seconds.count-
vllm.time_to_first_token.seconds.countThe total number of observations of time to first token.count-
vllm.time_to_first_token.seconds.sumThe sum of time to first token in seconds.countsecond

Official documentation

Official documentation for the vLLM integration

Metric names and descriptions are the official Datadog integration definitions (Copyright Datadog, Inc., BSD-3-Clause), reproduced unedited. The full license terms are on the About page.

Related products

IntegrationsIntegrations are the bundle of entry points that pour data from external systems such as AWS and MySQL into Datadog. Where you configure them splits into two families, the Agent side and the cloud account side, and which one you choose changes both what you can see and how billing is counted.Foundation and data collectionLLM ObservabilityLLM Observability records a single request to an application with an embedded LLM as a trace, tracks the inputs and outputs, tokens, latency, and errors of each model call, and turns response quality into numbers through evaluations. The only thing billed is the LLM span, which represents a model call.APM and tracingLog ManagementLog Management is the product that gathers logs from servers, containers, and cloud services in one place and connects them to search, retention, and notification. It has a distinctive structure in which ingestion and indexing are billed separately.Log managementDatadog AgentSoftware that runs resident on the servers and containers you monitor and delivers metrics, logs, and traces to Datadog.Foundation and data collection

Other AI/ML integrations

Weaviate101 metricsAirbyte48 metricsAnthropic Usage and Costs8 metricsAWS Inferentia and AWS Trainium Monitoring33 metricsBentoML11 metricsDatabricks16 metricsNvidia DCGM Exporter45 metricsHugging Face TGI54 metricsKubeflow48 metricsLangChain6 metricsMicrosoft Copilot21 metricsMilvus386 metrics