obsdog

Kafka Broker integration metrics

The official Datadog integration for monitoring Kafka Broker. It collects 125 metrics.

Official summary

Collect metrics for producers and consumers, replication, max lag, and more.

Metric types

  • gauge (value at a point in time): 121
  • rate (per second): 4

Metrics

Descriptions are the official definitions, unedited.

MetricTypeUnit
kafka.broker.start_timeThe start time of the Kafka broker in milliseconds since epoch.gaugemillisecond
kafka.consumer.bytes_consumedThe average number of bytes consumed per second for a specific topic.gaugebyte/second
kafka.consumer.bytes_inConsumer bytes in rate.gaugebyte/second
kafka.consumer.delayed_requestsNumber of delayed consumer requests.gaugerequest
kafka.consumer.expires_per_secondRate of delayed consumer request expiration.gaugeeviction/second
kafka.consumer.fetch_rateThe minimum rate at which the consumer sends fetch requests to a broker.gaugerequest
kafka.consumer.fetch_size_avgThe average number of bytes fetched per request for a specific topic.gaugebyte
kafka.consumer.fetch_size_maxThe maximum number of bytes fetched per request for a specific topic.gaugebyte
kafka.consumer.kafka_commitsRate of offset commits to Kafka.gaugewrite/second
kafka.consumer.max_lagMaximum consumer lag.gaugeoffset
kafka.consumer.messages_inRate of consumer message consumption.gaugemessage/second
kafka.consumer.records_consumedThe average number of records consumed per second for a specific topic.gaugerecord/second
kafka.consumer.records_per_request_avgThe average number of records in each request for a specific topic.gaugerecord/request
kafka.consumer.zookeeper_commitsRate of offset commits to ZooKeeper.gaugewrite/second
kafka.controller.active_broker_countNumber of active brokers in the cluster.gaugenode
kafka.controller.fenced_broker_countNumber of fenced brokers in the cluster.gaugenode
kafka.expires_secRate of delayed producer request expiration.gaugeeviction/second
kafka.follower.expires_per_secondRate of request expiration on followers.gaugeeviction/second
kafka.kraft.append_records_rateRecords appended per second by the Raft leader.gaugerecord/second
kafka.kraft.broker_metadata.last_applied_record_lag_msLag in milliseconds of the last applied metadata record.gaugemillisecond
kafka.kraft.broker_metadata.last_applied_record_offsetOffset of the last applied metadata record.gaugeoffset
kafka.kraft.broker_metadata.last_applied_record_timestampTimestamp of the last applied metadata record.gaugemillisecond
kafka.kraft.broker_metadata.metadata_apply_error_countNumber of errors applying the metadata log.rateerror
kafka.kraft.broker_metadata.metadata_load_error_countNumber of errors loading the metadata log.rateerror
kafka.kraft.commit_latency_avgAverage time to commit a Raft entry.gaugemillisecond
kafka.kraft.commit_latency_maxMaximum time to commit a Raft entry.gaugemillisecond
kafka.kraft.current_epochCurrent quorum epoch.gauge-
kafka.kraft.current_leaderCurrent quorum leader ID; -1 if unknown.gauge-
kafka.kraft.current_voteCurrent voted leader ID; -1 if not voted.gauge-
kafka.kraft.election_latency_avgAverage time for a leader election.gaugemillisecond
kafka.kraft.election_latency_maxMaximum time for a leader election.gaugemillisecond
kafka.kraft.fetch_records_rateRecords fetched per second from the Raft leader.gaugerecord/second
kafka.kraft.high_watermarkHigh watermark of the Raft quorum; -1 if unknown.gaugeoffset
kafka.kraft.log_end_epochLog end epoch of the Raft quorum.gauge-
kafka.kraft.log_end_offsetLog end offset of the Raft quorum.gaugeoffset
kafka.kraft.metadata_loader.current_controller_idID of the current controller; -1 if unknown.gauge-
kafka.kraft.metadata_loader.current_metadata_versionFeature level of the current effective metadata version.gauge-
kafka.kraft.metadata_loader.handle_load_snapshot_countNumber of metadata snapshots loaded since process start.rate-
kafka.kraft.poll_idle_ratio_avgAverage fraction of time the poll loop is idle.gaugefraction
kafka.kraft.snapshot_emitter.latest_snapshot_generated_age_msAge in milliseconds of the latest generated snapshot.gaugemillisecond
kafka.kraft.snapshot_emitter.latest_snapshot_generated_bytesSize in bytes of the latest generated snapshot.gaugebyte
kafka.kraft.unknown_voter_connectionsNumber of unknown voter connections whose information is not cached.gaugeconnection
kafka.log.directory.offlineWhether a Kafka log directory is offline. 0 means healthy.gauge-
kafka.log.flush_rate.rateLog flush rate.gaugeflush/second
kafka.log.partition.sizeThe size in bytes of a topic partition log on disk.gaugebyte
kafka.messages_in.rateIncoming message rate.gaugemessage
kafka.net.bytes_in.rateIncoming byte rate.gaugebyte/second
kafka.net.bytes_outOutgoing byte total.gaugebyte/second
kafka.net.bytes_out.rateOutgoing byte rate.gaugebyte/second
kafka.net.bytes_rejected.rateRejected byte rate.gaugebyte/second
kafka.net.processor.avg.idle.pct.rateAverage fraction of time the network processor threads are idle.gaugefraction
kafka.producer.available_buffer_bytesThe total amount of buffer memory that is not being used (either unallocated or in the free list)gaugebyte
kafka.producer.batch_size_avgThe average number of bytes sent per partition per-request.gaugebyte
kafka.producer.batch_size_maxThe max number of bytes sent per partition per-request.gaugebyte
kafka.producer.buffer_bytes_totalThe maximum amount of buffer memory the client can use (whether or not it is currently used).gaugebyte
kafka.producer.bufferpool_wait_ratioThe fraction of time an appender waits for space allocation.gauge-
kafka.producer.bufferpool_wait_timeThe fraction of time an appender waits for space allocation.gauge-
kafka.producer.bufferpool_wait_time_ns_totalThe total time in nanoseconds an appender waits for space allocation.gaugenanosecond
kafka.producer.bytes_outProducer bytes out rate.gaugebyte/second
kafka.producer.compression_rateThe average compression rate of record batches for a topicgaugefraction
kafka.producer.compression_rate_avgThe average compression rate of record batches.ratefraction
kafka.producer.delayed_requestsNumber of producer requests delayed.gaugerequest
kafka.producer.expires_per_secondsRate of producer request expiration.gaugeeviction/second
kafka.producer.io_waitProducer I/O wait time.gaugenanosecond
kafka.producer.message_rateProducer message rate.gaugemessage/second
kafka.producer.metadata_ageThe age in seconds of the current producer metadata being used.gaugesecond
kafka.producer.record_error_rateThe average per-second number of errored record sends for a topicgaugeerror/second
kafka.producer.record_queue_time_avgThe average time in ms record batches spent in the record accumulator.gaugemillisecond
kafka.producer.record_queue_time_maxThe maximum time in ms record batches spent in the record accumulator.gaugemillisecond
kafka.producer.record_retry_rateThe average per-second number of retried record sends for a topicgaugerecord/second
kafka.producer.record_send_rateThe average number of records sent per second for a topicgaugerecord/second
kafka.producer.record_size_avgThe average record size.gaugebyte
kafka.producer.record_size_maxThe maximum record size.gaugebyte
kafka.producer.records_per_requestThe average number of records sent per second.gaugerecord
kafka.producer.request_latency_avgProducer average request latency.gaugemillisecond
kafka.producer.request_latency_maxThe maximum request latency in ms.gaugemillisecond
kafka.producer.request_rateNumber of producer requests per second.gaugerequest/second
kafka.producer.requests_in_flightThe current number of in-flight requests awaiting a response.gaugerequest
kafka.producer.response_rateNumber of producer responses per second.gaugeresponse/second
kafka.producer.throttle_time_avgThe average time in ms a request was throttled by a broker.gaugemillisecond
kafka.producer.throttle_time_maxThe maximum time in ms a request was throttled by a broker.gaugemillisecond
kafka.producer.waiting_threadsThe number of user threads blocked waiting for buffer memory to enqueue their records.gaugethread
kafka.replication.active_controller_countNumber of active controllers in the cluster.gaugenode
kafka.replication.isr_expands.rateRate of replicas joining the ISR pool.gaugenode/second
kafka.replication.isr_shrinks.rateRate of replicas leaving the ISR pool.gaugenode/second
kafka.replication.leader_countNumber of leaders on this broker.gaugenode
kafka.replication.leader_elections.rateLeader election rate.gaugeevent/second
kafka.replication.max_lagMaximum lag in messages between the follower and leader replicas.gaugeoffset
kafka.replication.offline_partitions_countNumber of partitions that don't have an active leader.gauge-
kafka.replication.partition_countNumber of partitions across all topics in the cluster.gauge-
kafka.replication.unclean_leader_elections.rateUnclean leader election rate.gaugeevent/second
kafka.replication.under_min_isr_partition_countNumber of under min ISR partitions.gauge-
kafka.replication.under_replicated_partitionsNumber of under replicated partitions.gauge-
kafka.request.channel.queue.sizeNumber of queued requests.gaugerequest
kafka.request.fetch.failed.rateClient fetch request failures rate.gaugerequest
kafka.request.fetch_consumer.rateFetch consumer requests rate.gaugerequest
kafka.request.fetch_consumer.time.99percentileTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_consumer.time.avgTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_follower.rateFetch follower requests rate.gaugerequest
kafka.request.fetch_follower.time.99percentileTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_follower.time.avgTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_request_purgatory.sizeNumber of requests waiting in the producer purgatory.gaugerequest
kafka.request.handler.avg.idle.pct.rateAverage fraction of time the request handler threads are idle.gaugefraction
kafka.request.metadata.time.99percentileTime for metadata requests for 99th percentile.gaugemillisecond
kafka.request.metadata.time.avgAverage time for metadata request.gaugemillisecond
kafka.request.offsets.time.99percentileTime for offset requests for 99th percentile.gaugemillisecond
kafka.request.offsets.time.avgAverage time for an offset request.gaugemillisecond
kafka.request.produce.failed.rateFailed produce requests rate.gaugerequest
kafka.request.produce.rateProduce requests rate.gaugerequest
kafka.request.produce.time.99percentileTime for produce requests for 99th percentile.gaugemillisecond
kafka.request.produce.time.avgAverage time for a produce request.gaugemillisecond
kafka.request.producer_request_purgatory.sizeNumber of requests waiting in the producer purgatorygaugerequest
kafka.request.update_metadata.time.99percentileTime for update metadata requests for 99th percentile.gaugemillisecond
kafka.request.update_metadata.time.avgAverage time for a request to update metadata.gaugemillisecond
kafka.server.socket.connection_countNumber of currently open connections to the broker.gaugeconnection
kafka.session.fetch.countNumber of fetch sessions.gauge-
kafka.session.fetch.evictionEviction rate of fetch session.gaugeevent/second
kafka.session.zookeeper.disconnect.rateZookeeper client disconnect rate.gaugeevent/second
kafka.session.zookeeper.expire.rateZookeeper client session expiration rate.gaugeevent/second
kafka.session.zookeeper.readonly.rateZookeeper client readonly rate.gaugeevent/second
kafka.session.zookeeper.sync.rateZookeeper client sync rate.gaugeevent/second
kafka.topic.messages_in.rateIncoming message rate by topicgaugemessage
kafka.topic.net.bytes_in.rateIncoming byte rate by topic.gaugebyte/second
kafka.topic.net.bytes_out.rateOutgoing byte rate by topic.gaugebyte/second
kafka.topic.net.bytes_rejected.rateRejected byte rate by topic.gaugebyte/second

Official documentation

Official documentation for the Kafka Broker integration

Metric names and descriptions are the official Datadog integration definitions (Copyright Datadog, Inc., BSD-3-Clause), reproduced unedited. The full license terms are on the About page.

Related products

IntegrationsIntegrations are the bundle of entry points that pour data from external systems such as AWS and MySQL into Datadog. Where you configure them splits into two families, the Agent side and the cloud account side, and which one you choose changes both what you can see and how billing is counted.Foundation and data collectionLog ManagementLog Management is the product that gathers logs from servers, containers, and cloud services in one place and connects them to search, retention, and notification. It has a distinctive structure in which ingestion and indexing are billed separately.Log managementDatadog AgentSoftware that runs resident on the servers and containers you monitor and delivers metrics, logs, and traces to Datadog.Foundation and data collectionMetricsFrom CPU utilization to user sign-ups, metrics are the core data of Datadog, recording every number in your environment as a time series. If you understand how they are counted before how they are collected, you avoid aggregation mistakes and unexpected charges.Foundation and data collection

Other Message Queues integrations

Kafka Consumer39 metricsPulsar196 metricsRabbitMQ329 metricsTibco EMS50 metricsActiveMQ38 metricsActiveMQ XML16 metricsCelery10 metricsHiveMQ245 metricsIBM MQ94 metrics