obsdog

Kafka Broker インテグレーションのメトリクス

Kafka Broker の監視に使う Datadog の公式インテグレーションです。収集するメトリクスは 125 件あります。

公式の紹介文 (英語)

Collect metrics for producers and consumers, replication, max lag, and more.

種別の内訳

  • gauge (その時点の値): 121 件
  • rate (1 秒あたりの値): 4 件

メトリクスの一覧

説明は公式の定義の英文をそのまま載せています。

メトリクス種別単位
kafka.broker.start_timeThe start time of the Kafka broker in milliseconds since epoch.gaugemillisecond
kafka.consumer.bytes_consumedThe average number of bytes consumed per second for a specific topic.gaugebyte/second
kafka.consumer.bytes_inConsumer bytes in rate.gaugebyte/second
kafka.consumer.delayed_requestsNumber of delayed consumer requests.gaugerequest
kafka.consumer.expires_per_secondRate of delayed consumer request expiration.gaugeeviction/second
kafka.consumer.fetch_rateThe minimum rate at which the consumer sends fetch requests to a broker.gaugerequest
kafka.consumer.fetch_size_avgThe average number of bytes fetched per request for a specific topic.gaugebyte
kafka.consumer.fetch_size_maxThe maximum number of bytes fetched per request for a specific topic.gaugebyte
kafka.consumer.kafka_commitsRate of offset commits to Kafka.gaugewrite/second
kafka.consumer.max_lagMaximum consumer lag.gaugeoffset
kafka.consumer.messages_inRate of consumer message consumption.gaugemessage/second
kafka.consumer.records_consumedThe average number of records consumed per second for a specific topic.gaugerecord/second
kafka.consumer.records_per_request_avgThe average number of records in each request for a specific topic.gaugerecord/request
kafka.consumer.zookeeper_commitsRate of offset commits to ZooKeeper.gaugewrite/second
kafka.controller.active_broker_countNumber of active brokers in the cluster.gaugenode
kafka.controller.fenced_broker_countNumber of fenced brokers in the cluster.gaugenode
kafka.expires_secRate of delayed producer request expiration.gaugeeviction/second
kafka.follower.expires_per_secondRate of request expiration on followers.gaugeeviction/second
kafka.kraft.append_records_rateRecords appended per second by the Raft leader.gaugerecord/second
kafka.kraft.broker_metadata.last_applied_record_lag_msLag in milliseconds of the last applied metadata record.gaugemillisecond
kafka.kraft.broker_metadata.last_applied_record_offsetOffset of the last applied metadata record.gaugeoffset
kafka.kraft.broker_metadata.last_applied_record_timestampTimestamp of the last applied metadata record.gaugemillisecond
kafka.kraft.broker_metadata.metadata_apply_error_countNumber of errors applying the metadata log.rateerror
kafka.kraft.broker_metadata.metadata_load_error_countNumber of errors loading the metadata log.rateerror
kafka.kraft.commit_latency_avgAverage time to commit a Raft entry.gaugemillisecond
kafka.kraft.commit_latency_maxMaximum time to commit a Raft entry.gaugemillisecond
kafka.kraft.current_epochCurrent quorum epoch.gauge-
kafka.kraft.current_leaderCurrent quorum leader ID; -1 if unknown.gauge-
kafka.kraft.current_voteCurrent voted leader ID; -1 if not voted.gauge-
kafka.kraft.election_latency_avgAverage time for a leader election.gaugemillisecond
kafka.kraft.election_latency_maxMaximum time for a leader election.gaugemillisecond
kafka.kraft.fetch_records_rateRecords fetched per second from the Raft leader.gaugerecord/second
kafka.kraft.high_watermarkHigh watermark of the Raft quorum; -1 if unknown.gaugeoffset
kafka.kraft.log_end_epochLog end epoch of the Raft quorum.gauge-
kafka.kraft.log_end_offsetLog end offset of the Raft quorum.gaugeoffset
kafka.kraft.metadata_loader.current_controller_idID of the current controller; -1 if unknown.gauge-
kafka.kraft.metadata_loader.current_metadata_versionFeature level of the current effective metadata version.gauge-
kafka.kraft.metadata_loader.handle_load_snapshot_countNumber of metadata snapshots loaded since process start.rate-
kafka.kraft.poll_idle_ratio_avgAverage fraction of time the poll loop is idle.gaugefraction
kafka.kraft.snapshot_emitter.latest_snapshot_generated_age_msAge in milliseconds of the latest generated snapshot.gaugemillisecond
kafka.kraft.snapshot_emitter.latest_snapshot_generated_bytesSize in bytes of the latest generated snapshot.gaugebyte
kafka.kraft.unknown_voter_connectionsNumber of unknown voter connections whose information is not cached.gaugeconnection
kafka.log.directory.offlineWhether a Kafka log directory is offline. 0 means healthy.gauge-
kafka.log.flush_rate.rateLog flush rate.gaugeflush/second
kafka.log.partition.sizeThe size in bytes of a topic partition log on disk.gaugebyte
kafka.messages_in.rateIncoming message rate.gaugemessage
kafka.net.bytes_in.rateIncoming byte rate.gaugebyte/second
kafka.net.bytes_outOutgoing byte total.gaugebyte/second
kafka.net.bytes_out.rateOutgoing byte rate.gaugebyte/second
kafka.net.bytes_rejected.rateRejected byte rate.gaugebyte/second
kafka.net.processor.avg.idle.pct.rateAverage fraction of time the network processor threads are idle.gaugefraction
kafka.producer.available_buffer_bytesThe total amount of buffer memory that is not being used (either unallocated or in the free list)gaugebyte
kafka.producer.batch_size_avgThe average number of bytes sent per partition per-request.gaugebyte
kafka.producer.batch_size_maxThe max number of bytes sent per partition per-request.gaugebyte
kafka.producer.buffer_bytes_totalThe maximum amount of buffer memory the client can use (whether or not it is currently used).gaugebyte
kafka.producer.bufferpool_wait_ratioThe fraction of time an appender waits for space allocation.gauge-
kafka.producer.bufferpool_wait_timeThe fraction of time an appender waits for space allocation.gauge-
kafka.producer.bufferpool_wait_time_ns_totalThe total time in nanoseconds an appender waits for space allocation.gaugenanosecond
kafka.producer.bytes_outProducer bytes out rate.gaugebyte/second
kafka.producer.compression_rateThe average compression rate of record batches for a topicgaugefraction
kafka.producer.compression_rate_avgThe average compression rate of record batches.ratefraction
kafka.producer.delayed_requestsNumber of producer requests delayed.gaugerequest
kafka.producer.expires_per_secondsRate of producer request expiration.gaugeeviction/second
kafka.producer.io_waitProducer I/O wait time.gaugenanosecond
kafka.producer.message_rateProducer message rate.gaugemessage/second
kafka.producer.metadata_ageThe age in seconds of the current producer metadata being used.gaugesecond
kafka.producer.record_error_rateThe average per-second number of errored record sends for a topicgaugeerror/second
kafka.producer.record_queue_time_avgThe average time in ms record batches spent in the record accumulator.gaugemillisecond
kafka.producer.record_queue_time_maxThe maximum time in ms record batches spent in the record accumulator.gaugemillisecond
kafka.producer.record_retry_rateThe average per-second number of retried record sends for a topicgaugerecord/second
kafka.producer.record_send_rateThe average number of records sent per second for a topicgaugerecord/second
kafka.producer.record_size_avgThe average record size.gaugebyte
kafka.producer.record_size_maxThe maximum record size.gaugebyte
kafka.producer.records_per_requestThe average number of records sent per second.gaugerecord
kafka.producer.request_latency_avgProducer average request latency.gaugemillisecond
kafka.producer.request_latency_maxThe maximum request latency in ms.gaugemillisecond
kafka.producer.request_rateNumber of producer requests per second.gaugerequest/second
kafka.producer.requests_in_flightThe current number of in-flight requests awaiting a response.gaugerequest
kafka.producer.response_rateNumber of producer responses per second.gaugeresponse/second
kafka.producer.throttle_time_avgThe average time in ms a request was throttled by a broker.gaugemillisecond
kafka.producer.throttle_time_maxThe maximum time in ms a request was throttled by a broker.gaugemillisecond
kafka.producer.waiting_threadsThe number of user threads blocked waiting for buffer memory to enqueue their records.gaugethread
kafka.replication.active_controller_countNumber of active controllers in the cluster.gaugenode
kafka.replication.isr_expands.rateRate of replicas joining the ISR pool.gaugenode/second
kafka.replication.isr_shrinks.rateRate of replicas leaving the ISR pool.gaugenode/second
kafka.replication.leader_countNumber of leaders on this broker.gaugenode
kafka.replication.leader_elections.rateLeader election rate.gaugeevent/second
kafka.replication.max_lagMaximum lag in messages between the follower and leader replicas.gaugeoffset
kafka.replication.offline_partitions_countNumber of partitions that don't have an active leader.gauge-
kafka.replication.partition_countNumber of partitions across all topics in the cluster.gauge-
kafka.replication.unclean_leader_elections.rateUnclean leader election rate.gaugeevent/second
kafka.replication.under_min_isr_partition_countNumber of under min ISR partitions.gauge-
kafka.replication.under_replicated_partitionsNumber of under replicated partitions.gauge-
kafka.request.channel.queue.sizeNumber of queued requests.gaugerequest
kafka.request.fetch.failed.rateClient fetch request failures rate.gaugerequest
kafka.request.fetch_consumer.rateFetch consumer requests rate.gaugerequest
kafka.request.fetch_consumer.time.99percentileTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_consumer.time.avgTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_follower.rateFetch follower requests rate.gaugerequest
kafka.request.fetch_follower.time.99percentileTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_follower.time.avgTotal time in ms to serve the specified request.gaugemillisecond
kafka.request.fetch_request_purgatory.sizeNumber of requests waiting in the producer purgatory.gaugerequest
kafka.request.handler.avg.idle.pct.rateAverage fraction of time the request handler threads are idle.gaugefraction
kafka.request.metadata.time.99percentileTime for metadata requests for 99th percentile.gaugemillisecond
kafka.request.metadata.time.avgAverage time for metadata request.gaugemillisecond
kafka.request.offsets.time.99percentileTime for offset requests for 99th percentile.gaugemillisecond
kafka.request.offsets.time.avgAverage time for an offset request.gaugemillisecond
kafka.request.produce.failed.rateFailed produce requests rate.gaugerequest
kafka.request.produce.rateProduce requests rate.gaugerequest
kafka.request.produce.time.99percentileTime for produce requests for 99th percentile.gaugemillisecond
kafka.request.produce.time.avgAverage time for a produce request.gaugemillisecond
kafka.request.producer_request_purgatory.sizeNumber of requests waiting in the producer purgatorygaugerequest
kafka.request.update_metadata.time.99percentileTime for update metadata requests for 99th percentile.gaugemillisecond
kafka.request.update_metadata.time.avgAverage time for a request to update metadata.gaugemillisecond
kafka.server.socket.connection_countNumber of currently open connections to the broker.gaugeconnection
kafka.session.fetch.countNumber of fetch sessions.gauge-
kafka.session.fetch.evictionEviction rate of fetch session.gaugeevent/second
kafka.session.zookeeper.disconnect.rateZookeeper client disconnect rate.gaugeevent/second
kafka.session.zookeeper.expire.rateZookeeper client session expiration rate.gaugeevent/second
kafka.session.zookeeper.readonly.rateZookeeper client readonly rate.gaugeevent/second
kafka.session.zookeeper.sync.rateZookeeper client sync rate.gaugeevent/second
kafka.topic.messages_in.rateIncoming message rate by topicgaugemessage
kafka.topic.net.bytes_in.rateIncoming byte rate by topic.gaugebyte/second
kafka.topic.net.bytes_out.rateOutgoing byte rate by topic.gaugebyte/second
kafka.topic.net.bytes_rejected.rateRejected byte rate by topic.gaugebyte/second

公式ドキュメント

Kafka Broker インテグレーションの公式ドキュメント

メトリクスの名前と説明は Datadog の公式インテグレーションの定義 (Copyright Datadog, Inc.・BSD-3-Clause) を英文のまま載せています。利用条件の全文は サイトについてにあります。

関連する製品

インテグレーションインテグレーションは、AWS や MySQL といった外部システムのデータを Datadog に流し込む「入口の束」です。設定する場所は Agent 側とクラウドアカウント側の 2 系統に分かれ、どちらを選ぶかで見える範囲と課金の数え方が変わります。基盤とデータ収集ログ管理サーバー・コンテナ・クラウドサービスのログを 1 か所に集め、検索・保管・通知につなげる製品です。取り込みとインデックスを分けて課金する独特の構造を持ちます。ログ管理Datadog Agent監視対象のサーバーやコンテナに常駐し、メトリクス・ログ・トレースを Datadog へ送り届けるソフトウェアです。基盤とデータ収集メトリクスCPU 使用率からユーザー登録数まで、環境のあらゆる数値を時系列で記録する Datadog の中核データです。集め方より「数え方」を先に理解すると、集計の誤りと想定外の課金を避けられます。基盤とデータ収集

メッセージキューの他のインテグレーション

Kafka Consumerメトリクス 39 件Pulsarメトリクス 196 件RabbitMQメトリクス 329 件Tibco EMSメトリクス 50 件ActiveMQメトリクス 38 件ActiveMQ XMLメトリクス 16 件Celeryメトリクス 10 件HiveMQメトリクス 245 件IBM MQメトリクス 94 件