Stream Pipeline Logs

Required role: read or higher.

Retrieve logs of a pipeline as a stream.

The logs stream catches up to the extent of the internally configured per-pipeline circular logs buffer (limited to a certain byte size and number of lines, whichever is reached first). After the catch-up, new lines are pushed whenever they become available.

It is possible for the logs stream to end prematurely due to the API server temporarily losing connection to the runner. In this case, it is needed to issue again a new request to this endpoint.

The logs stream will end when the pipeline is deleted, or if the runner restarts. Note that in both cases the logs will be cleared.

Resuming a stream

Supplying the cursor parameter asks for only the lines the caller is missing, rather than the whole retained buffer. This matters on an unstable connection, where replaying the buffer on every reconnect can consume the entire link and leave the caller unable to reach the live tail.

A caller that supplies cursor is answered with three response headers:

feldera-logs-epoch: 0199c3f1-2d0a-7e84-b711-6f2c9a1d4e08
feldera-logs-seq: 41272
feldera-logs-gap: 0

• feldera-logs-epoch identifies this lifetime of the logs buffer. Pass it back unchanged.
• feldera-logs-seq is the sequence number of the line preceding the response's first log line.
• feldera-logs-gap counts lines that were discarded between the requested cursor and the sequence number above, and which the caller will therefore never receive. Zero means the resume is exact.

The body is log lines and nothing else, one per sequence number, so the caller's current position is feldera-logs-seq plus the number of lines it has received. To reconnect, pass cursor=<epoch>:<position>.

A 503 means the position could not be resolved. Retry it with the same cursor.

A pipeline's logs buffer is created when the runner first sees the pipeline and discarded when the runner restarts or the pipeline is deleted, which is also when the logs are cleared. Stopping and starting a pipeline leaves the buffer alone: the epoch stays the same and the numbering continues across runs.

A cursor carrying an epoch from a buffer that no longer exists is not an error: it is answered with a full catch-up and a gap naming what was lost, so a caller can never be locked out of its logs by an old cursor.

Callers that supply cursor receive no informational lines in the body, which is what makes the one-line-per-sequence-number correspondence exact. Callers that omit it get the body they have always received, and no position headers.

Path Parameters
pipeline_name string — REQUIRED Unique pipeline name

Query Parameters
cursor string Resume the stream after this position, formatted as <epoch>:<sequence> and derived from a previous response's feldera-logs-epoch and feldera-logs-seq headers. Omit for the whole retained buffer with no position headers. Pass empty to start from the beginning of the buffer and be told the position.

Responses
200 Pipeline logs retrieved successfully Schema — OPTIONAL string
400 Cursor is malformed Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
404 Pipeline with that name does not exist Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
500 Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.
503 Schema — OPTIONAL details Detailed error metadata. The contents of this field is determined by error_code. error_code string Error code is a string that specifies this error type. message string Human-readable error message.