Spark インテグレーションのメトリクス

Spark の監視に使う Datadog の公式インテグレーションです。収集するメトリクスは 158 件あります。

2026-08-05 時点の定義

公式の紹介文 (英語)

Track failed task rates, shuffled bytes, and much more.

ログ収集

種別の内訳

count (集計間隔内の発生回数): 144 件
gauge (その時点の値): 14 件

メトリクスの一覧

説明は公式の定義の英文をそのまま載せています。

メトリクス	種別	単位
`spark.driver.active_tasks`Number of active tasks in the driver	count	task
`spark.driver.completed_tasks`Number of completed tasks in the driver	count	task
`spark.driver.disk_used`Amount of disk used in the driver	count	byte
`spark.driver.failed_tasks`Number of failed tasks in the driver	count	task
`spark.driver.max_memory`Maximum memory used in the driver	count	byte
`spark.driver.mem.total_off_heap_storage`Total available off heap memory for storage	count	byte
`spark.driver.mem.total_on_heap_storage`Total available on heap memory for storage	count	byte
`spark.driver.mem.used_off_heap_storage`Used off heap memory currently for storage	count	byte
`spark.driver.mem.used_on_heap_storage`Used on heap memory currently for storage	count	byte
`spark.driver.memory_used`Amount of memory used in the driver	count	byte
`spark.driver.peak_mem.direct_pool`Peak memory that the JVM is using for direct buffer pool	count	byte
`spark.driver.peak_mem.jvm_heap_memory`Peak memory usage of the heap that is used for object allocation	count	byte
`spark.driver.peak_mem.jvm_off_heap_memory`Peak memory usage of non-heap memory that is used by the Java virtual machine	count	byte
`spark.driver.peak_mem.major_gc_count`Total major GC count	count	byte
`spark.driver.peak_mem.major_gc_time`Elapsed total major GC time	count	millisecond
`spark.driver.peak_mem.mapped_pool`Peak memory that the JVM is using for mapped buffer pool	count	byte
`spark.driver.peak_mem.minor_gc_count`Total minor GC count	count	byte
`spark.driver.peak_mem.minor_gc_time`Elapsed total minor GC time	count	millisecond
`spark.driver.peak_mem.off_heap_execution`Peak off heap execution memory in use	count	byte
`spark.driver.peak_mem.off_heap_storage`Peak off heap storage memory in use	count	byte
`spark.driver.peak_mem.off_heap_unified`Peak off heap memory (execution and storage)	count	byte
`spark.driver.peak_mem.on_heap_execution`Peak on heap execution memory in use	count	byte
`spark.driver.peak_mem.on_heap_storage`Peak on heap storage memory in use	count	byte
`spark.driver.peak_mem.on_heap_unified`Peak on heap memory (execution and storage)	count	byte
`spark.driver.peak_mem.process_tree_jvm`Virtual memory size	count	byte
`spark.driver.peak_mem.process_tree_jvm_rss`Resident Set Size: number of pages the process has in real memory	count	byte
`spark.driver.peak_mem.process_tree_other`Virtual memory size for other kind of process	count	byte
`spark.driver.peak_mem.process_tree_other_rss`Resident Set Size for other kind of process	count	byte
`spark.driver.peak_mem.process_tree_python`Virtual memory size for Python	count	byte
`spark.driver.peak_mem.process_tree_python_rss`Resident Set Size for Python	count	byte
`spark.driver.rdd_blocks`Number of RDD blocks in the driver	count	block
`spark.driver.total_duration`Time spent in the driver	count	millisecond
`spark.driver.total_input_bytes`Number of input bytes in the driver	count	byte
`spark.driver.total_shuffle_read`Number of bytes read during a shuffle in the driver	count	byte
`spark.driver.total_shuffle_write`Number of shuffled bytes in the driver	count	byte
`spark.driver.total_tasks`Number of total tasks in the driver	count	task
`spark.executor.active_tasks`Number of active tasks in the application's executors	count	task
`spark.executor.completed_tasks`Number of completed tasks in the application's executors	count	task
`spark.executor.count`Number of executors	count	task
`spark.executor.disk_used`Amount of disk space used by persisted RDDs in the application's executors	count	byte
`spark.executor.failed_tasks`Number of failed tasks in the application's executors	count	task
`spark.executor.id.active_tasks`Number of active tasks in this executor	count	task
`spark.executor.id.completed_tasks`Number of completed tasks in this executor	count	task
`spark.executor.id.disk_used`Amount of disk space used by persisted RDDs in this executor	count	byte
`spark.executor.id.failed_tasks`Number of failed tasks in this executor	count	task
`spark.executor.id.max_memory`Total amount of memory available for storage for this executor	count	byte
`spark.executor.id.mem.total_off_heap_storage`Total available off heap memory for storage	count	byte
`spark.executor.id.mem.total_on_heap_storage`Total available on heap memory for storage	count	byte
`spark.executor.id.mem.used_off_heap_storage`Used off heap memory currently for storage	count	byte
`spark.executor.id.mem.used_on_heap_storage`Used on heap memory currently for storage	count	byte
`spark.executor.id.memory_used`Amount of memory used for cached RDDs in this executor.	count	byte
`spark.executor.id.peak_mem.direct_pool`Peak memory that the JVM is using for direct buffer pool	count	byte
`spark.executor.id.peak_mem.jvm_heap_memory`Peak memory usage of the heap that is used for object allocation	count	byte
`spark.executor.id.peak_mem.jvm_off_heap_memory`Peak memory usage of non-heap memory that is used by the Java virtual machine	count	byte
`spark.executor.id.peak_mem.major_gc_count`Total major GC count	count	byte
`spark.executor.id.peak_mem.major_gc_time`Elapsed total major GC time	count	millisecond
`spark.executor.id.peak_mem.mapped_pool`Peak memory that the JVM is using for mapped buffer pool	count	byte
`spark.executor.id.peak_mem.minor_gc_count`Total minor GC count	count	byte
`spark.executor.id.peak_mem.minor_gc_time`Elapsed total minor GC time	count	millisecond
`spark.executor.id.peak_mem.off_heap_execution`Peak off heap execution memory in use	count	byte
`spark.executor.id.peak_mem.off_heap_storage`Peak off heap storage memory in use	count	byte
`spark.executor.id.peak_mem.off_heap_unified`Peak off heap memory (execution and storage)	count	byte
`spark.executor.id.peak_mem.on_heap_execution`Peak on heap execution memory in use	count	byte
`spark.executor.id.peak_mem.on_heap_storage`Peak on heap storage memory in use	count	byte
`spark.executor.id.peak_mem.on_heap_unified`Peak on heap memory (execution and storage)	count	byte
`spark.executor.id.peak_mem.process_tree_jvm`Virtual memory size	count	byte
`spark.executor.id.peak_mem.process_tree_jvm_rss`Resident Set Size: number of pages the process has in real memory	count	byte
`spark.executor.id.peak_mem.process_tree_other`Virtual memory size for other kind of process	count	byte
`spark.executor.id.peak_mem.process_tree_other_rss`Resident Set Size for other kind of process	count	byte
`spark.executor.id.peak_mem.process_tree_python`Virtual memory size for Python	count	byte
`spark.executor.id.peak_mem.process_tree_python_rss`Resident Set Size for Python	count	byte
`spark.executor.id.rdd_blocks`Number of persisted RDD blocks in this executor	count	block
`spark.executor.id.total_duration`Time spent by the executor executing tasks	count	millisecond
`spark.executor.id.total_input_bytes`Total number of input bytes in the executor	count	byte
`spark.executor.id.total_shuffle_read`Total number of bytes read during a shuffle in the executor	count	byte
`spark.executor.id.total_shuffle_write`Total number of shuffled bytes in the executor	count	byte
`spark.executor.id.total_tasks`Total number of tasks in this executor	count	task
`spark.executor.max_memory`Max memory across all executors working for a particular application	count	byte
`spark.executor.mem.total_off_heap_storage`Total available off heap memory for storage	count	byte
`spark.executor.mem.total_on_heap_storage`Total available on heap memory for storage	count	byte
`spark.executor.mem.used_off_heap_storage`Used off heap memory currently for storage	count	byte
`spark.executor.mem.used_on_heap_storage`Used on heap memory currently for storage	count	byte
`spark.executor.memory_used`Amount of memory used for cached RDDs in the application's executors	count	byte
`spark.executor.peak_mem.direct_pool`Peak memory that the JVM is using for direct buffer pool	count	byte
`spark.executor.peak_mem.jvm_heap_memory`Peak memory usage of the heap that is used for object allocation	count	byte
`spark.executor.peak_mem.jvm_off_heap_memory`Peak memory usage of non-heap memory that is used by the Java virtual machine	count	byte
`spark.executor.peak_mem.major_gc_count`Total major GC count	count	byte
`spark.executor.peak_mem.major_gc_time`Elapsed total major GC time	count	millisecond
`spark.executor.peak_mem.mapped_pool`Peak memory that the JVM is using for mapped buffer pool	count	byte
`spark.executor.peak_mem.minor_gc_count`Total minor GC count	count	byte
`spark.executor.peak_mem.minor_gc_time`Elapsed total minor GC time	count	millisecond
`spark.executor.peak_mem.off_heap_execution`Peak off heap execution memory in use	count	byte
`spark.executor.peak_mem.off_heap_storage`Peak off heap storage memory in use	count	byte
`spark.executor.peak_mem.off_heap_unified`Peak off heap memory (execution and storage)	count	byte
`spark.executor.peak_mem.on_heap_execution`Peak on heap execution memory in use	count	byte
`spark.executor.peak_mem.on_heap_storage`Peak on heap storage memory in use	count	byte
`spark.executor.peak_mem.on_heap_unified`Peak on heap memory (execution and storage)	count	byte
`spark.executor.peak_mem.process_tree_jvm`Virtual memory size	count	byte
`spark.executor.peak_mem.process_tree_jvm_rss`Resident Set Size: number of pages the process has in real memory	count	byte
`spark.executor.peak_mem.process_tree_other`Virtual memory size for other kind of process	count	byte
`spark.executor.peak_mem.process_tree_other_rss`Resident Set Size for other kind of process	count	byte
`spark.executor.peak_mem.process_tree_python`Virtual memory size for Python	count	byte
`spark.executor.peak_mem.process_tree_python_rss`Resident Set Size for Python	count	byte
`spark.executor.rdd_blocks`Number of persisted RDD blocks in the application's executors	count	block
`spark.executor.total_duration`Time spent by the application's executors executing tasks	count	millisecond
`spark.executor.total_input_bytes`Total number of input bytes in the application's executors	count	byte
`spark.executor.total_shuffle_read`Total number of bytes read during a shuffle in the application's executors	count	byte
`spark.executor.total_shuffle_write`Total number of shuffled bytes in the application's executors	count	byte
`spark.executor.total_tasks`Total number of tasks in the application's executors	count	task
`spark.executor_memory`Maximum memory available for caching RDD blocks in the application's executors	count	byte
`spark.job.count`Number of jobs	count	task
`spark.job.num_active_stages`Number of active stages in the application	count	stage
`spark.job.num_active_tasks`Number of active tasks in the application	count	task
`spark.job.num_completed_stages`Number of completed stages in the application	count	stage
`spark.job.num_completed_tasks`Number of completed tasks in the application	count	task
`spark.job.num_failed_stages`Number of failed stages in the application	count	stage
`spark.job.num_failed_tasks`Number of failed tasks in the application	count	task
`spark.job.num_skipped_stages`Number of skipped stages in the application	count	stage
`spark.job.num_skipped_tasks`Number of skipped tasks in the application	count	task
`spark.job.num_tasks`Number of tasks in the application	count	task
`spark.rdd.count`Number of RDDs	count	-
`spark.rdd.disk_used`Amount of disk space used by persisted RDDs in the application	count	byte
`spark.rdd.memory_used`Amount of memory used in the application's persisted RDDs	count	byte
`spark.rdd.num_cached_partitions`Number of in-memory cached RDD partitions in the application	count	-
`spark.rdd.num_partitions`Number of persisted RDD partitions in the application	count	-
`spark.stage.count`Number of stages	count	task
`spark.stage.disk_bytes_spilled`Max size on disk of the spilled bytes in the application's stages	count	byte
`spark.stage.executor_run_time`Time spent by the executor in the application's stages	count	millisecond
`spark.stage.input_bytes`Input bytes in the application's stages	count	byte
`spark.stage.input_records`Input records in the application's stages	count	record
`spark.stage.memory_bytes_spilled`Number of bytes spilled to disk in the application's stages	count	byte
`spark.stage.num_active_tasks`Number of active tasks in the application's stages	count	task
`spark.stage.num_complete_tasks`Number of complete tasks in the application's stages	count	task
`spark.stage.num_failed_tasks`Number of failed tasks in the application's stages	count	task
`spark.stage.output_bytes`Output bytes in the application's stages	count	byte
`spark.stage.output_records`Output records in the application's stages	count	record
`spark.stage.shuffle_read_bytes`Number of bytes read during a shuffle in the application's stages	count	byte
`spark.stage.shuffle_read_records`Number of records read during a shuffle in the application's stages	count	record
`spark.stage.shuffle_write_bytes`Number of shuffled bytes in the application's stages	count	byte
`spark.stage.shuffle_write_records`Number of shuffled records in the application's stages	count	record
`spark.streaming.statistics.avg_input_rate`Average streaming input data rate	gauge	byte
`spark.streaming.statistics.avg_processing_time`Average application's streaming batch processing time	gauge	millisecond
`spark.streaming.statistics.avg_scheduling_delay`Average application's streaming batch scheduling delay	gauge	millisecond
`spark.streaming.statistics.avg_total_delay`Average application's streaming batch total delay	gauge	millisecond
`spark.streaming.statistics.batch_duration`Application's streaming batch duration	gauge	millisecond
`spark.streaming.statistics.num_active_batches`Number of active streaming batches	gauge	job
`spark.streaming.statistics.num_active_receivers`Number of active streaming receivers	gauge	object
`spark.streaming.statistics.num_inactive_receivers`Number of inactive streaming receivers	gauge	object
`spark.streaming.statistics.num_processed_records`Number of processed streaming records	count	record
`spark.streaming.statistics.num_received_records`Number of received streaming records	count	record
`spark.streaming.statistics.num_receivers`Number of streaming application's receivers	gauge	object
`spark.streaming.statistics.num_retained_completed_batches`Number of retained completed application's streaming batches	count	job
`spark.streaming.statistics.num_total_completed_batches`Total number of completed application's streaming batches	count	job
`spark.structured_streaming.input_rate`Average streaming input data rate	gauge	record/second
`spark.structured_streaming.latency`Average latency for the structured streaming application.	gauge	millisecond
`spark.structured_streaming.processing_rate`Number of received streaming records per second	gauge	row/second
`spark.structured_streaming.rows_count`Count of rows.	gauge	row
`spark.structured_streaming.used_bytes`Number of bytes used in memory.	gauge	byte

公式ドキュメント

Spark インテグレーションの公式ドキュメント

メトリクスの名前と説明は Datadog の公式インテグレーションの定義 (Copyright Datadog, Inc.・BSD-3-Clause) を英文のまま載せています。利用条件の全文はサイトについてにあります。

公式の紹介文 (英語)

種別の内訳

メトリクスの一覧

公式ドキュメント

関連する製品

ログ収集の他のインテグレーション