feldera.rest package

Submodules

feldera.rest.config module

class feldera.rest.config.Config(url: str, api_key: str | None = None, version: str | None = None, timeout: float | None = None)

Bases: object

FelderaClient’s credentials and configuration parameters

feldera.rest.errors module

exception feldera.rest.errors.FelderaAPIError(error: str, request: Response)

Bases: FelderaError

Error sent by Feldera API

exception feldera.rest.errors.FelderaCommunicationError(err: str)

Bases: FelderaError

Error when connection to Feldera

exception feldera.rest.errors.FelderaError(message: str)

Bases: Exception

Generic class for Feldera error handling

exception feldera.rest.errors.FelderaTimeoutError(err: str)

Bases: FelderaError

Error when Feldera operation takes longer than expected

feldera.rest.feldera_client module

class feldera.rest.feldera_client.FelderaClient(url: str, api_key: str | None = None, timeout: float | None = None)

Bases: object

A client for the Feldera HTTP API

A client instance is needed for every Feldera API method to know the location of Feldera and its permissions.

checkpoint_pipeline(pipeline_name: str)

Checkpoint a fault-tolerant pipeline

Parameters:

pipeline_name – The name of the pipeline to checkpoint

create_or_update_pipeline(pipeline: Pipeline) → Pipeline

Create a pipeline if it doesn’t exist or update a pipeline and wait for it to compile

create_pipeline(pipeline: Pipeline) → Pipeline

Create a pipeline if it doesn’t exist and wait for it to compile

Name:

The name of the pipeline

delete_pipeline(name: str)

Deletes a pipeline by name

Parameters:

name – The name of the pipeline

get_pipeline(pipeline_name) → Pipeline

Get a pipeline by name

Parameters:

pipeline_name – The name of the pipeline

get_pipeline_stats(name: str) → dict

Get the pipeline metrics and performance counters

Parameters:

name – The name of the pipeline

get_runtime_config(pipeline_name) → dict

Get the runtime config of a pipeline by name

Parameters:

pipeline_name – The name of the pipeline

listen_to_pipeline(pipeline_name: str, table_name: str, format: str, backpressure: bool = True, array: bool = False, timeout: float | None = None)

Listen for updates to views for pipeline, yields the chunks of data

Parameters:

• pipeline_name – The name of the pipeline

• table_name – The name of the table to listen to

• format – The format of the data, either “json” or “csv”

• backpressure – When the flag is True (the default), this method waits for the consumer to receive each chunk and blocks the pipeline if the consumer cannot keep up. When this flag is False, the pipeline drops data chunks if the consumer is not keeping up with its output. This prevents a slow consumer from slowing down the entire pipeline.

• array – Set True to group updates in this stream into JSON arrays, used in conjunction with the “json” format, the default value is False

• timeout – The amount of time in seconds to listen to the stream for

static localhost(port: int = 8080) → FelderaClient

Create a FelderaClient that connects to the local Feldera instance

patch_pipeline(name: str, sql: str)

Incrementally update the pipeline SQL

Parameters:

• name – The name of the pipeline

• sql – The SQL snippet. Replaces the existing SQL code with this one.

pause_connector(pipeline_name, table_name, connector_name)

Pause the specified input connector.

Connectors allow feldera to fetch data from a source or write data to a sink. This method allows users to PAUSE a specific INPUT connector. All connectors are RUNNING by default.

Refer to the connector documentation for more information: <https://docs.feldera.com/connectors/#input-connector-orchestration>

Parameters:

• pipeline_name – The name of the pipeline.

• table_name – The name of the table associated with this connector.

• connector_name – The name of the connector.

Raises:

FelderaAPIError – If the connector cannot be found, or if the pipeline is not running.

pause_pipeline(pipeline_name: str, error_message: str | None = None, timeout_s: float | None = 300)

Stop a pipeline

Parameters:

• pipeline_name – The name of the pipeline to stop

• error_message – The error message to show if the pipeline is in FAILED state

• timeout_s – The amount of time in seconds to wait for the pipeline to pause. 300 seconds by default.

pipelines() → list[Pipeline]

Get all pipelines

push_to_pipeline(pipeline_name: str, table_name: str, format: str, data: list[list | str | dict], array: bool = False, force: bool = False, update_format: str = 'raw', json_flavor: str | None = None, serialize: bool = True)

Insert data into a pipeline

Parameters:

• pipeline_name – The name of the pipeline

• table_name – The name of the table

• format – The format of the data, either “json” or “csv”

• array – True if updates in this stream are packed into JSON arrays, used in conjunction with the “json” format

• force – If True, the data will be inserted even if the pipeline is paused

• update_format – JSON data change event format, used in conjunction with the “json” format, the default value is “insert_delete”, other supported formats: “weighted”, “debezium”, “snowflake”, “raw”

• json_flavor – JSON encoding used for individual table records, the default value is “default”, other supported encodings: “debezium_mysql”, “snowflake”, “kafka_connect_json_converter”, “pandas”

• data – The data to insert

• serialize – If True, the data will be serialized to JSON. True by default

query_as_json(pipeline_name: str, query: str) → Generator[dict, None, None]

Executes an ad-hoc query on the specified pipeline and returns the result as a generator that yields rows of the query as Python dictionaries. All floating-point numbers are deserialized as Decimal objects to avoid precision loss.

Parameters:

• pipeline_name – The name of the pipeline to query.

• query – The SQL query to be executed.

Returns:

A generator that yields each row of the result as a Python dictionary, deserialized from JSON.

query_as_parquet(pipeline_name: str, query: str, path: str)

Executes an ad-hoc query on the specified pipeline and saves the result to a parquet file. If the extension isn’t parquet, it will be automatically appended to path.

Parameters:

• pipeline_name – The name of the pipeline to query.

• query – The SQL query to be executed.

• path – The path including the file name to save the resulting parquet file in.

query_as_text(pipeline_name: str, query: str) → Generator[str, None, None]

Executes an ad-hoc query on the specified pipeline and returns a generator that yields lines of the table.

Parameters:

• pipeline_name – The name of the pipeline to query.

• query – The SQL query to be executed.

Returns:

A generator yielding the query result in tabular format, one line at a time.

resume_connector(pipeline_name: str, table_name: str, connector_name: str)

Resume the specified connector.

Connectors allow feldera to fetch data from a source or write data to a sink. This method allows users to RESUME / START a specific INPUT connector. All connectors are RUNNING by default.

Refer to the connector documentation for more information: <https://docs.feldera.com/connectors/#input-connector-orchestration>

Parameters:

• pipeline_name – The name of the pipeline.

• table_name – The name of the table associated with this connector.

• connector_name – The name of the connector.

Raises:

FelderaAPIError – If the connector cannot be found, or if the pipeline is not running.

shutdown_pipeline(pipeline_name: str, timeout_s: float | None = 300)

Shutdown a pipeline

Parameters:

• pipeline_name – The name of the pipeline to shut down

• timeout_s – The amount of time in seconds to wait for the pipeline to shut down. Default is 15 seconds.

start_pipeline(pipeline_name: str, timeout_s: float | None = 300)

Parameters:

• pipeline_name – The name of the pipeline to start

• timeout_s – The amount of time in seconds to wait for the pipeline to start. 300 seconds by default.

feldera.rest.pipeline module

class feldera.rest.pipeline.Pipeline(name: str, sql: str, udf_rust: str, udf_toml: str, program_config: Mapping[str, Any], runtime_config: Mapping[str, Any], description: str | None = None)

Bases: object

Represents a Feldera pipeline

classmethod from_dict(d: Mapping[str, Any])

feldera.rest.sql_table module

class feldera.rest.sql_table.SQLTable(name: str, fields: list[dict], case_sensitive: bool = False, materialized: bool = False)

Bases: object

Represents a SQL table in Feldera

classmethod from_dict(table_dict: dict)

feldera.rest.sql_view module

class feldera.rest.sql_view.SQLView(name: str, fields: list[dict], case_sensitive: bool = False, materialized: bool = False)

Bases: object

Represents a SQL view in Feldera

classmethod from_dict(view_dict: dict)

Module contents

This is the lower level REST client for Feldera.

This is a thin wrapper around the Feldera REST API.

It is recommended to use the higher level abstractions in the feldera package, instead of using the REST client directly.