Get Pipeline Stats

Retrieve statistics (e.g., performance counters) of a running or paused pipeline.

Path Parameters
pipeline_name string — REQUIRED Unique pipeline name

Responses
200 Pipeline statistics retrieved successfully Schema — OPTIONAL global_metrics object Global controller metrics. bootstrap_in_progress boolean The pipeline has been resumed from a checkpoint and is currently bootstrapping new and modified views. buffered_input_bytes int64 Total number of bytes currently buffered by all endpoints. buffered_input_records int64 Total number of records currently buffered by all endpoints. commit_progress object — OPTIONAL Summary of transaction commit progress. completed int64 Number of operators that have been fully flushed. in_progress int64 Number of operators that are currently being flushed. in_progress_processed_records int64 Number of records processed by operators that are currently being flushed. in_progress_total_records int64 Total number of records that operators that are currently being flushed need to process. remaining int64 Number of operators that haven't started flushing. cpu_msecs int64 CPU time used by the pipeline across all threads, in milliseconds. incarnation_uuid uuid Uniquely identifies the pipeline process that started at start_time. initial_start_time int64 Time at which the pipeline process from which we resumed started, in seconds since the epoch. output_stall_msecs int64 If the pipeline is stalled because one or more output connectors' output buffers are full, this is the number of milliseconds that the current stall has lasted. If this is nonzero, then the output connectors causing the stall can be identified by noticing ExternalOutputEndpointMetrics::queued_records is greater than or equal to ConnectorConfig::max_queued_records. In the ordinary case, the pipeline is not stalled, and this value is 0. pipeline_complete boolean True if the pipeline has processed all input data to completion. rss_bytes int64 Resident set size of the pipeline process, in bytes. runtime_elapsed_msecs int64 Time elapsed while the pipeline is executing a step, multiplied by the number of threads, in milliseconds. start_time int64 Time at which the pipeline process started, in seconds since the epoch. state string Possible values: [Paused, Running, Terminated] Pipeline state. storage_bytes int64 Current storage usage in bytes. storage_mb_secs int64 Storage usage integrated over time, in megabytes * seconds. total_completed_records int64 Total number of input records processed to completion. total_completed_steps int64 Number of steps whose input records have been processed to completion. A record is processed to completion if it has been processed by the DBSP engine and all outputs derived from it have been processed by all output connectors. Interpretation This is a count, not a step number. If total_completed_steps is 0, no steps have been processed to completion. If total_completed_steps > 0, then the last step whose input records have been processed to completion is total_completed_steps - 1. A record that was ingested when total_initiated_steps was n is fully processed when total_completed_steps >= n. total_initiated_steps int64 Number of steps that have been initiated. Interpretation This is a count, not a step number. If total_initiated_steps is 0, no steps have been initiated. If total_initiated_steps > 0, then step total_initiated_steps - 1 has been started and all steps previous to that have been completely processed by the circuit. total_input_bytes int64 Total number of bytes received from all endpoints. total_input_records int64 Total number of records received from all endpoints. total_processed_bytes int64 Total bytes of input records processed by the DBSP engine. total_processed_records int64 Total number of input records processed by the DBSP engine. transaction_id int64 ID of the current transaction or 0 if no transaction is in progress. transaction_initiators object Information about entities that initiated the current transaction. initiated_by_api string — OPTIONAL Possible values: [Started, Committed] Transaction phase. initiated_by_connectors object Transaction phases initiated by connectors, indexed by endpoint name. transaction_id int64 — OPTIONAL ID assigned to the transaction (None if no transaction is in progress). transaction_msecs int64 — OPTIONAL Elapsed time in milliseconds, according to transaction_status: [TransactionStatus::TransactionInProgress]: Time that this transaction has been in progress. [TransactionStatus::CommitInProgress]: Time that this transaction has been committing. transaction_records int64 — OPTIONAL Number of records in this transaction, according to transaction_status: [TransactionStatus::TransactionInProgress]: Number of records added so far. More records might be added. [TransactionStatus::CommitInProgress]: Final number of records. transaction_status string Possible values: [NoTransaction, TransactionInProgress, CommitInProgress] Transaction status summarized as a single value. uptime_msecs int64 Time since the pipeline process started, including time that the pipeline was running or paused. This is the elapsed time since start_time. inputs object[] Input endpoint configs and metrics. barrier boolean Endpoint is currently a barrier to checkpointing and suspend. completed_frontier object — OPTIONAL A watermark that has been fully processed by the pipeline. completed_at date-time Timestamp when all outputs produced from this input have been pushed to all output endpoints. ingested_at date-time Timestamp when the data was ingested from the wire. metadata object Metadata that describes the position in the input stream (e.g., Kafka partition/offset pairs). processed_at date-time Timestamp when the data was processed by the circuit. config object Schema definition for endpoint config that only includes the stream field. stream string The name of the stream. endpoint_name string Endpoint name. fatal_error string — OPTIONAL The first fatal error that occurred at the endpoint. health object — OPTIONAL description string — OPTIONAL status string Possible values: [Healthy, Unhealthy] metrics object Performance metrics for an input endpoint. buffered_bytes int64 Number of bytes currently buffered by the endpoint (not yet consumed by the circuit). buffered_records int64 Number of records currently buffered by the endpoint (not yet consumed by the circuit). end_of_input boolean True if end-of-input has been signaled. num_parse_errors int64 Number of parse errors. num_transport_errors int64 Number of transport errors. total_bytes int64 Total bytes pushed to the endpoint since it was created. total_records int64 Total records pushed to the endpoint since it was created. parse_errors object[] — OPTIONAL Recent parse errors on this endpoint. index int64 Sequence number of the error. The client can use this field to detect gaps in the error list reported by the pipeline. When the connector reports a large number of errors, the pipeline will only preserve and report the most recent errors of each kind. message string Error message. tag string — OPTIONAL Optional tag for the error. The tag is used to group errors by their type. timestamp date-time Timestamp when the error occurred, serialized as RFC3339 with microseconds. paused boolean Endpoint has been paused by the user. transport_errors object[] — OPTIONAL Recent transport errors on this endpoint. index int64 Sequence number of the error. The client can use this field to detect gaps in the error list reported by the pipeline. When the connector reports a large number of errors, the pipeline will only preserve and report the most recent errors of each kind. message string Error message. tag string — OPTIONAL Optional tag for the error. The tag is used to group errors by their type. timestamp date-time Timestamp when the error occurred, serialized as RFC3339 with microseconds. outputs object[] Output endpoint configs and metrics. config object Schema definition for endpoint config that only includes the stream field. stream string The name of the stream. encode_errors object[] — OPTIONAL Recent encoding errors on this endpoint. index int64 Sequence number of the error. The client can use this field to detect gaps in the error list reported by the pipeline. When the connector reports a large number of errors, the pipeline will only preserve and report the most recent errors of each kind. message string Error message. tag string — OPTIONAL Optional tag for the error. The tag is used to group errors by their type. timestamp date-time Timestamp when the error occurred, serialized as RFC3339 with microseconds. endpoint_name string Endpoint name. fatal_error string — OPTIONAL The first fatal error that occurred at the endpoint. health object — OPTIONAL description string — OPTIONAL status string Possible values: [Healthy, Unhealthy] metrics object Performance metrics for an output endpoint. buffered_batches int64 Number of batches in the buffer. buffered_records int64 Number of records pushed to the output buffer. memory int64 Extra memory in use beyond that used for queuing records. num_encode_errors int64 Number of encoding errors. num_transport_errors int64 Number of transport errors. queued_batches int64 Number of queued batches. queued_records int64 Number of queued records. total_processed_input_records int64 The number of input records processed by the circuit. This metric tracks the end-to-end progress of the pipeline: the output of this endpoint is equal to the output of the circuit after processing total_processed_input_records records. In a multihost pipeline, this count reflects only the input records processed on the same host as the output endpoint, which is not usually meaningful. total_processed_steps int64 The number of steps whose input records have been processed by the endpoint. This is meaningful in a multihost pipeline because steps are synchronized across all of the hosts. Interpretation This is a count, not a step number. If total_processed_steps is 0, no steps have been processed to completion. If total_processed_steps > 0, then the last step whose input records have been processed to completion is total_processed_steps - 1. A record that was ingested in step n is fully processed when total_processed_steps > n. transmitted_bytes int64 Bytes sent on the underlying transport. transmitted_records int64 Records sent on the underlying transport. transport_errors object[] — OPTIONAL Recent transport errors on this endpoint. index int64 Sequence number of the error. The client can use this field to detect gaps in the error list reported by the pipeline. When the connector reports a large number of errors, the pipeline will only preserve and report the most recent errors of each kind. message string Error message. tag string — OPTIONAL Optional tag for the error. The tag is used to group errors by their type. timestamp date-time Timestamp when the error occurred, serialized as RFC3339 with microseconds. suspend_error — OPTIONAL Whether a pipeline supports checkpointing and suspend-and-resume.
404 Pipeline with that name does not exist Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
500 Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
503 Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.