Get Pipeline Stats

Retrieve statistics (e.g., performance counters) of a running or paused pipeline.

Path Parameters
pipeline_name string — REQUIRED Unique pipeline name

Responses
200 Pipeline statistics retrieved successfully Schema — OPTIONAL global_metrics object Global controller metrics. bootstrap_in_progress boolean The pipeline has been resumed from a checkpoint and is currently bootstrapping new and modified views. buffered_input_bytes int64 Total number of bytes currently buffered by all endpoints. buffered_input_records int64 Total number of records currently buffered by all endpoints. cpu_msecs int64 CPU time used by the pipeline across all threads, in milliseconds. incarnation_uuid uuid Uniquely identifies the pipeline process that started at start_time. initial_start_time int64 Time at which the pipeline process from which we resumed started, in seconds since the epoch. pipeline_complete boolean True if the pipeline has processed all input data to completion. rss_bytes int64 Resident set size of the pipeline process, in bytes. runtime_elapsed_msecs int64 Time elapsed while the pipeline is executing a step, multiplied by the number of threads, in milliseconds. start_time int64 Time at which the pipeline process started, in seconds since the epoch. state string Possible values: [Paused, Running, Terminated] Pipeline state. storage_bytes int64 Current storage usage in bytes. storage_mb_secs int64 Storage usage integrated over time, in megabytes * seconds. total_completed_records int64 Total number of input records processed to completion. total_completed_steps int64 Number of steps whose input records have been processed to completion. A record is processed to completion if it has been processed by the DBSP engine and all outputs derived from it have been processed by all output connectors. Interpretation This is a count, not a step number. If total_completed_steps is 0, no steps have been processed to completion. If total_completed_steps > 0, then the last step whose input records have been processed to completion is total_completed_steps - 1. A record that was ingested when total_initiated_steps was n is fully processed when total_completed_steps >= n. total_initiated_steps int64 Number of steps that have been initiated. Interpretation This is a count, not a step number. If total_initiated_steps is 0, no steps have been initiated. If total_initiated_steps > 0, then step total_initiated_steps - 1 has been started and all steps previous to that have been completely processed by the circuit. total_input_bytes int64 Total number of bytes received from all endpoints. total_input_records int64 Total number of records received from all endpoints. total_processed_bytes int64 Total bytes of input records processed by the DBSP engine. total_processed_records int64 Total number of input records processed by the DBSP engine. transaction_id int64 ID of the current transaction or 0 if no transaction is in progress. transaction_initiators object Information about entities that initiated the current transaction. initiated_by_api string — OPTIONAL Possible values: [Started, Committed] Transaction phase. initiated_by_connectors object Transaction phases initiated by connectors, indexed by endpoint name. transaction_id int64 — OPTIONAL ID assigned to the transaction (None if no transaction is in progress). transaction_status string Possible values: [NoTransaction, TransactionInProgress, CommitInProgress] Transaction status summarized as a single value. uptime_msecs int64 Time since the pipeline process started, including time that the pipeline was running or paused. This is the elapsed time since start_time. inputs object[] Input endpoint configs and metrics. barrier boolean Endpoint is currently a barrier to checkpointing and suspend. completed_frontier object — OPTIONAL A watermark that has been fully processed by the pipeline. completed_at string Timestamp when all outputs produced from this input have been pushed to all output endpoints. ingested_at string Timestamp when the data was ingested from the wire. metadata object Metadata that describes the position in the input stream (e.g., Kafka partition/offset pairs). processed_at string Timestamp when the data was processed by the circuit. config object Schema definition for endpoint config that only includes the stream field. stream string The name of the stream. endpoint_name string Endpoint name. fatal_error string — OPTIONAL The first fatal error that occurred at the endpoint. metrics object Performance metrics for an input endpoint. buffered_bytes int64 Number of bytes currently buffered by the endpoint (not yet consumed by the circuit). buffered_records int64 Number of records currently buffered by the endpoint (not yet consumed by the circuit). end_of_input boolean True if end-of-input has been signaled. num_parse_errors int64 Number of parse errors. num_transport_errors int64 Number of transport errors. total_bytes int64 Total bytes pushed to the endpoint since it was created. total_records int64 Total records pushed to the endpoint since it was created. paused boolean Endpoint has been paused by the user. outputs object[] Output endpoint configs and metrics. config object Schema definition for endpoint config that only includes the stream field. stream string The name of the stream. endpoint_name string Endpoint name. fatal_error string — OPTIONAL The first fatal error that occurred at the endpoint. metrics object Performance metrics for an output endpoint. buffered_batches int64 Number of batches in the buffer. buffered_records int64 Number of records pushed to the output buffer. memory int64 Extra memory in use beyond that used for queuing records. num_encode_errors int64 Number of encoding errors. num_transport_errors int64 Number of transport errors. queued_batches int64 Number of queued batches. queued_records int64 Number of queued records. total_processed_input_records int64 The number of input records processed by the circuit. transmitted_bytes int64 Bytes sent on the underlying transport. transmitted_records int64 Records sent on the underlying transport. suspend_error — OPTIONAL Whether a pipeline supports checkpointing and suspend-and-resume.
404 Pipeline with that name does not exist Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
500 Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
503 Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.