SGLang integration metrics

The official Datadog integration for monitoring SGLang. It collects 99 metrics.

Definitions as of 2026-09-30

Official summary

Monitor SGLang request performance, token throughput, and KV cache efficiency

AI/ML

Metric types

gauge (value at a point in time): 57
count (occurrences in the interval): 42

Metrics

Descriptions are the official definitions, unedited.

Metric	Type	Unit
`sglang.cache_config_info`Cache configuration information.	gauge	-
`sglang.cached_tokens.count`Total number of cached prompt tokens.	count	token
`sglang.cuda_graph.active`Whether CUDA graph execution is active.	gauge	-
`sglang.cuda_graph.passes.count`Total number of CUDA graph execution passes.	count	-
`sglang.decode.sequence_length_sum`Sum of sequence lengths currently being decoded.	gauge	token
`sglang.disaggregation.bootstrap_failures.count`Total number of failed bootstrap requests.	count	-
`sglang.disaggregation.decode_preallocation_queue`Requests waiting for decode preallocation.	gauge	-
`sglang.disaggregation.decode_transfer_queue`Requests waiting for decode transfer.	gauge	-
`sglang.disaggregation.kv_transfer.allocation_milliseconds`Time spent allocating memory for KV transfers.	gauge	millisecond
`sglang.disaggregation.kv_transfer.bootstrap_milliseconds`Time spent bootstrapping KV transfers.	gauge	millisecond
`sglang.disaggregation.kv_transfer.gigabytes_per_second`KV cache transfer throughput.	gauge	gigabyte/second
`sglang.disaggregation.kv_transfer.latency_milliseconds`KV cache transfer latency.	gauge	millisecond
`sglang.disaggregation.kv_transfer.megabytes`KV cache data transferred.	gauge	megabyte
`sglang.disaggregation.prefill_bootstrap_queue`Requests in the prefill bootstrap queue.	gauge	-
`sglang.disaggregation.prefill_inflight_queue`Requests in the prefill inflight queue.	gauge	-
`sglang.disaggregation.prefill_preallocation_queue`Requests waiting for prefill preallocation.	gauge	-
`sglang.disaggregation.prefill_retries.count`Total number of retried prefill requests.	count	-
`sglang.disaggregation.transfer_failures.count`Total number of failed KV transfer requests.	count	-
`sglang.e2e_request_latency.seconds`End-to-end request latency (distribution).	gauge	second
`sglang.e2e_request_latency.seconds.count`End-to-end request latency observation count.	count	-
`sglang.e2e_request_latency.seconds.sum`Sum of end-to-end request latency.	count	second
`sglang.estimated_flops_per_gpu.count`Estimated floating-point operations per GPU.	count	operation
`sglang.estimated_read_bytes_per_gpu.count`Estimated bytes read per GPU.	count	byte
`sglang.estimated_write_bytes_per_gpu.count`Estimated bytes written per GPU.	count	byte
`sglang.forward_execution.seconds.count`Total model forward execution time.	count	second
`sglang.generation_throughput.tokens_per_second`Generation throughput.	gauge	token/second
`sglang.generation_tokens.count`Total number of generation tokens processed.	count	token
`sglang.http.requests.active`Currently active HTTP requests.	gauge	-
`sglang.http.requests.count`Total number of HTTP requests.	count	request
`sglang.http.responses.count`Total number of HTTP responses.	count	response
`sglang.inter_token_latency.seconds`Inter-token latency (distribution).	gauge	second
`sglang.inter_token_latency.seconds.count`Inter-token latency observation count.	count	-
`sglang.inter_token_latency.seconds.sum`Sum of inter-token latency.	count	second
`sglang.kv_cache.full_attention_usage`Full-attention KV cache usage as a fraction.	gauge	fraction
`sglang.kv_cache.hit_rate`Prefix KV cache hit rate.	gauge	fraction
`sglang.kv_cache.max_tokens`Maximum number of tokens in the KV cache pool.	gauge	token
`sglang.kv_cache.pending_preallocation_usage`KV cache usage reserved by pending preallocations.	gauge	fraction
`sglang.kv_cache.swa_usage`Sliding-window-attention KV cache usage as a fraction.	gauge	fraction
`sglang.kv_cache.usage`Overall KV cache usage as a fraction.	gauge	fraction
`sglang.kv_cache.used_tokens`Number of tokens currently stored in the KV cache.	gauge	token
`sglang.mamba_pool.usage`Mamba state pool usage as a fraction.	gauge	fraction
`sglang.num_requests.grammar_waiting`Requests waiting for grammar compilation.	gauge	request
`sglang.num_requests.offline_batch_running`Running low-priority offline batch requests.	gauge	request
`sglang.num_requests.running`Number of running inference requests.	gauge	request
`sglang.num_requests.waiting`Number of inference requests in the waiting queue.	gauge	request
`sglang.process.cpu_seconds.count`Total CPU time consumed by the SGLang process.	count	second
`sglang.process.max_fds`Maximum number of file descriptors.	gauge	-
`sglang.process.open_fds`Number of open file descriptors.	gauge	-
`sglang.process.resident_memory_bytes`Resident memory used by the SGLang process.	gauge	byte
`sglang.process.start_time_seconds`Unix timestamp when the SGLang process started.	gauge	second
`sglang.process.virtual_memory_bytes`Virtual memory used by the SGLang process.	gauge	byte
`sglang.prompt_tokens.count`Total number of prompt tokens processed.	count	token
`sglang.python.gc.collections.count`Python garbage collection runs.	count	-
`sglang.python.gc.objects.collected.count`Python objects collected by garbage collection.	count	object
`sglang.python.gc.objects.uncollectable.count`Python objects that garbage collection could not collect.	count	object
`sglang.python.info`Python runtime information.	gauge	-
`sglang.queue_time.seconds`Time requests spend waiting in a queue (distribution).	gauge	second
`sglang.queue_time.seconds.count`Request queue-time observation count.	count	-
`sglang.queue_time.seconds.sum`Sum of request queue time.	count	second
`sglang.realtime_tokens.count`Tokens processed in real time.	count	token
`sglang.request.generation_tokens`Generation tokens per request (distribution).	gauge	token
`sglang.request.generation_tokens.count`Generation-token histogram observation count.	count	-
`sglang.request.generation_tokens.sum`Sum of generation tokens observed per request.	count	token
`sglang.request.prompt_tokens`Prompt tokens per request (distribution).	gauge	token
`sglang.request.prompt_tokens.count`Prompt-token histogram observation count.	count	-
`sglang.request.prompt_tokens.sum`Sum of prompt tokens observed per request.	count	token
`sglang.request.stage_latency.seconds`Per-stage request latency (distribution).	gauge	second
`sglang.request.stage_latency.seconds.count`Per-stage request latency observation count.	count	-
`sglang.request.stage_latency.seconds.sum`Sum of per-stage request latency.	count	second
`sglang.request.uncached_prompt_tokens`Uncached prompt tokens per request (distribution).	gauge	token
`sglang.request.uncached_prompt_tokens.count`Uncached prompt-token histogram observation count.	count	-
`sglang.request.uncached_prompt_tokens.sum`Sum of uncached prompt tokens observed per request.	count	token
`sglang.requests.aborted.count`Total number of aborted requests.	count	request
`sglang.requests.count`Total number of inference requests processed.	count	request
`sglang.requests.paused`Number of paused requests.	gauge	request
`sglang.requests.retracted`Number of currently retracted requests.	gauge	request
`sglang.requests.retracted_total.count`Total number of retracted requests.	count	request
`sglang.retracted_input_tokens.count`Input tokens belonging to retracted requests.	count	token
`sglang.retracted_output_tokens.count`Output tokens belonging to retracted requests.	count	token
`sglang.routing_keys.active`Number of routing keys with active requests.	gauge	-
`sglang.scheduler.idle.seconds.count`Total time the scheduler spent idle.	count	second
`sglang.scheduler.new_token_ratio`Ratio used by the scheduler when allocating new tokens.	gauge	fraction
`sglang.scheduler.process_cpu.seconds.count`Total scheduler CPU processing time.	count	second
`sglang.scheduler.utilization`Scheduler utilization as a fraction.	gauge	fraction
`sglang.spec_decode.accept_length`Average speculative decoding acceptance length.	gauge	token
`sglang.spec_decode.accept_rate`Average speculative decoding acceptance rate.	gauge	fraction
`sglang.spec_decode.block_accept_length`Average accepted speculative tokens per block.	gauge	token
`sglang.spec_decode.capture_length`Speculative decoding capture length.	gauge	token
`sglang.spec_decode.draft_tokens`Number of draft tokens per speculative decoding step.	gauge	token
`sglang.spec_decode.steps`Number of speculative decoding steps.	gauge	-
`sglang.spec_decode.verify_calls.count`Total speculative decoding verification calls.	count	-
`sglang.startup.cuda_graph.seconds`Time spent capturing CUDA graphs during startup.	gauge	second
`sglang.startup.phase.seconds`Startup duration for the phase identified by the phase tag.	gauge	second
`sglang.startup.seconds`Time taken for the SGLang engine to start.	gauge	second
`sglang.time_to_first_token.seconds`Time from request receipt to first generated token (distribution).	gauge	second
`sglang.time_to_first_token.seconds.count`Time-to-first-token observation count.	count	-
`sglang.time_to_first_token.seconds.sum`Sum of time to first token.	count	second
`sglang.weight_load.seconds`Time taken to load model weights.	gauge	second
`sglang.weight_update.seconds`Duration of the most recent weight update from the source identified by the source tag.	gauge	second

Official documentation

Official documentation for the SGLang integration

Metric names and descriptions are the official Datadog integration definitions (Copyright Datadog, Inc., BSD-3-Clause), reproduced unedited. The full license terms are on the About page.

Official summary

Metric types

Metrics

Official documentation

Related products

Other AI/ML integrations