feldera.rest package
Submodules
feldera.rest.config module
- class feldera.rest.config.Config(url: str, api_key: str | None = None, version: str | None = None, timeout: float | None = None, connection_timeout: float | None = None, requests_verify: bool = True)[source]
Bases:
object
FelderaClient
’s credentials and configuration parameters
feldera.rest.errors module
- exception feldera.rest.errors.FelderaAPIError(error: str, request: Response)[source]
Bases:
FelderaError
Error sent by Feldera API
- exception feldera.rest.errors.FelderaCommunicationError(err: str)[source]
Bases:
FelderaError
Error when connection to Feldera
- exception feldera.rest.errors.FelderaError(message: str)[source]
Bases:
Exception
Generic class for Feldera error handling
- exception feldera.rest.errors.FelderaTimeoutError(err: str)[source]
Bases:
FelderaError
Error when Feldera operation takes longer than expected
feldera.rest.feldera_client module
- class feldera.rest.feldera_client.FelderaClient(url: str, api_key: str | None = None, timeout: float | None = None, connection_timeout: float | None = None, requests_verify: bool = True)[source]
Bases:
object
A client for the Feldera HTTP API
A client instance is needed for every Feldera API method to know the location of Feldera and its permissions.
- activate_pipeline(pipeline_name: str, wait: bool = True, timeout_s: float | None = 300)[source]
- Parameters:
pipeline_name – The name of the pipeline to activate
wait – Set True to wait for the pipeline to activate. True by default
timeout_s – The amount of time in seconds to wait for the pipeline to activate. 300 seconds by default.
- checkpoint_pipeline(pipeline_name: str) int [source]
Checkpoint a pipeline.
- Parameters:
pipeline_name – The name of the pipeline to checkpoint
- checkpoint_pipeline_status(pipeline_name: str) dict [source]
Gets the checkpoint status
- Parameters:
pipeline_name – The name of the pipeline to check the checkpoint status of.
- clear_storage(pipeline_name: str, timeout_s: float | None = 300)[source]
Clears the storage from the pipeline. This operation cannot be canceled.
- Parameters:
pipeline_name – The name of the pipeline
timeout_s – The amount of time in seconds to wait for the storage to clear. Default is 300 seconds.
- commit_transaction(pipeline_name: str, transaction_id: int | None = None, wait: bool = True, timeout_s: float | None = None)[source]
Commits the currently active transaction.
- Parameters:
pipeline_name – The name of the pipeline.
transaction_id – If provided, the function verifies that the currently active transaction matches this ID. If the active transaction ID does not match, the function raises an error.
wait – If True, the function blocks until the transaction either commits successfully or the timeout is reached. If False, the function initiates the commit and returns immediately without waiting for completion. The default value is True.
timeout_s – Maximum time (in seconds) to wait for the transaction to commit when wait is True. If None, the function will wait indefinitely.
- Raises:
RuntimeError – If there is currently no transaction in progress.
ValueError – If the provided transaction_id does not match the current transaction.
TimeoutError – If the transaction does not commit within the specified timeout (when wait is True).
FelderaAPIError – If the pipeline fails to start a transaction.
- create_or_update_pipeline(pipeline: Pipeline) Pipeline [source]
Create a pipeline if it doesn’t exist or update a pipeline and wait for it to compile
- create_pipeline(pipeline: Pipeline) Pipeline [source]
Create a pipeline if it doesn’t exist and wait for it to compile
- Name:
The name of the pipeline
- delete_pipeline(name: str)[source]
Deletes a pipeline by name
- Parameters:
name – The name of the pipeline
- get_config() FelderaConfig [source]
Get general feldera configuration.
- get_pipeline(pipeline_name) Pipeline [source]
Get a pipeline by name
- Parameters:
pipeline_name – The name of the pipeline
- get_pipeline_logs(pipeline_name: str) Generator[str, None, None] [source]
Get the pipeline logs
- Parameters:
name – The name of the pipeline
- Returns:
A generator yielding the logs, one line at a time.
- get_pipeline_stats(name: str) dict [source]
Get the pipeline metrics and performance counters
- Parameters:
name – The name of the pipeline
- get_pipeline_support_bundle(pipeline_name: str, params: Dict[str, Any] | None = None) bytes [source]
Generate a support bundle containing diagnostic information from a pipeline.
This endpoint collects various diagnostic data from the pipeline including circuit profile, heap profile, metrics, logs, stats, and connector statistics, and packages them into a single ZIP file for support purposes.
- Parameters:
pipeline_name – The name of the pipeline
params – Optional query parameters to control data collection
- Returns:
The support bundle as bytes (ZIP file)
- Raises:
FelderaAPIError – If the pipeline does not exist or if there’s an error
- get_runtime_config(pipeline_name) dict [source]
Get the runtime config of a pipeline by name
- Parameters:
pipeline_name – The name of the pipeline
- listen_to_pipeline(pipeline_name: str, table_name: str, format: str, backpressure: bool = True, array: bool = False, timeout: float | None = None)[source]
Listen for updates to views for pipeline, yields the chunks of data
- Parameters:
pipeline_name – The name of the pipeline
table_name – The name of the table to listen to
format – The format of the data, either “json” or “csv”
backpressure – When the flag is True (the default), this method waits for the consumer to receive each chunk and blocks the pipeline if the consumer cannot keep up. When this flag is False, the pipeline drops data chunks if the consumer is not keeping up with its output. This prevents a slow consumer from slowing down the entire pipeline.
array – Set True to group updates in this stream into JSON arrays, used in conjunction with the “json” format, the default value is False
timeout – The amount of time in seconds to listen to the stream for
- static localhost(port: int = 8080) FelderaClient [source]
Create a FelderaClient that connects to the local Feldera instance
- patch_pipeline(name: str, sql: str | None = None, udf_rust: str | None = None, udf_toml: str | None = None, program_config: Mapping[str, Any] | None = None, runtime_config: Mapping[str, Any] | None = None, description: str | None = None)[source]
Incrementally update pipeline
- Parameters:
name – The name of the pipeline
- pause_connector(pipeline_name, table_name, connector_name)[source]
Pause the specified input connector.
Connectors allow feldera to fetch data from a source or write data to a sink. This method allows users to PAUSE a specific INPUT connector. All connectors are RUNNING by default.
Refer to the connector documentation for more information: <https://docs.feldera.com/connectors/#input-connector-orchestration>
- Parameters:
pipeline_name – The name of the pipeline.
table_name – The name of the table associated with this connector.
connector_name – The name of the connector.
- Raises:
FelderaAPIError – If the connector cannot be found, or if the pipeline is not running.
- pause_pipeline(pipeline_name: str, error_message: str = None, wait: bool = True, timeout_s: float | None = 300)[source]
Stop a pipeline
- Parameters:
pipeline_name – The name of the pipeline to stop
error_message – The error message to show if the pipeline is in STOPPED state due to a failure.
wait – Set True to wait for the pipeline to pause. True by default
timeout_s – The amount of time in seconds to wait for the pipeline to pause. 300 seconds by default.
- push_to_pipeline(pipeline_name: str, table_name: str, format: str, data: list[list | str | dict] | dict | str, array: bool = False, force: bool = False, update_format: str = 'raw', json_flavor: str | None = None, serialize: bool = True, wait: bool = True, wait_timeout_s: float | None = None) str [source]
Insert data into a pipeline
- Parameters:
pipeline_name – The name of the pipeline
table_name – The name of the table
format – The format of the data, either “json” or “csv”
array – True if updates in this stream are packed into JSON arrays, used in conjunction with the “json” format
force – If True, the data will be inserted even if the pipeline is paused
update_format – JSON data change event format, used in conjunction with the “json” format, the default value is “insert_delete”, other supported formats: “weighted”, “debezium”, “snowflake”, “raw”
json_flavor – JSON encoding used for individual table records, the default value is “default”, other supported encodings: “debezium_mysql”, “snowflake”, “kafka_connect_json_converter”, “pandas”
data – The data to insert
serialize – If True, the data will be serialized to JSON. True by default
wait – If True, blocks until this input has been processed by the pipeline
wait_timeout_s – The timeout in seconds to wait for this set of inputs to be processed by the pipeline. None by default
- Returns:
The completion token to this input.
- query_as_hash(pipeline_name: str, query: str) str [source]
Executes an ad-hoc query on the specified pipeline and returns a hash of the result.
- Parameters:
pipeline_name – The name of the pipeline to query.
query – The SQL query to be executed.
- Returns:
A string containing the hash of the query result.
- query_as_json(pipeline_name: str, query: str) Generator[dict, None, None] [source]
Executes an ad-hoc query on the specified pipeline and returns the result as a generator that yields rows of the query as Python dictionaries. All floating-point numbers are deserialized as Decimal objects to avoid precision loss.
- Parameters:
pipeline_name – The name of the pipeline to query.
query – The SQL query to be executed.
- Returns:
A generator that yields each row of the result as a Python dictionary, deserialized from JSON.
- query_as_parquet(pipeline_name: str, query: str, path: str)[source]
Executes an ad-hoc query on the specified pipeline and saves the result to a parquet file. If the extension isn’t parquet, it will be automatically appended to path.
- Parameters:
pipeline_name – The name of the pipeline to query.
query – The SQL query to be executed.
path – The path including the file name to save the resulting parquet file in.
- query_as_text(pipeline_name: str, query: str) Generator[str, None, None] [source]
Executes an ad-hoc query on the specified pipeline and returns a generator that yields lines of the table.
- Parameters:
pipeline_name – The name of the pipeline to query.
query – The SQL query to be executed.
- Returns:
A generator yielding the query result in tabular format, one line at a time.
- resume_connector(pipeline_name: str, table_name: str, connector_name: str)[source]
Resume the specified connector.
Connectors allow feldera to fetch data from a source or write data to a sink. This method allows users to RESUME / START a specific INPUT connector. All connectors are RUNNING by default.
Refer to the connector documentation for more information: <https://docs.feldera.com/connectors/#input-connector-orchestration>
- Parameters:
pipeline_name – The name of the pipeline.
table_name – The name of the table associated with this connector.
connector_name – The name of the connector.
- Raises:
FelderaAPIError – If the connector cannot be found, or if the pipeline is not running.
- start_pipeline(pipeline_name: str, wait: bool = True, timeout_s: float | None = 300)[source]
- Parameters:
pipeline_name – The name of the pipeline to start
wait – Set True to wait for the pipeline to start. True by default
timeout_s – The amount of time in seconds to wait for the pipeline to start. 300 seconds by default.
- start_transaction(pipeline_name: str) int [source]
Start a new transaction.
Transaction ID.
- Parameters:
pipeline_name – The name of the pipeline.
- stop_pipeline(pipeline_name: str, force: bool, wait: bool = True, timeout_s: float | None = 300)[source]
Stop a pipeline
- Parameters:
pipeline_name – The name of the pipeline to stop
force – Set True to immediately scale compute resources to zero. Set False to automatically checkpoint before stopping.
wait – Set True to wait for the pipeline to stop. True by default
timeout_s – The amount of time in seconds to wait for the pipeline to stop. Default is 300 seconds.
- sync_checkpoint(pipeline_name: str) str [source]
Triggers a checkpoint synchronization for the specified pipeline. Check the status by calling pipeline_sync_checkpoint_status.
- Parameters:
pipeline_name – Name of the pipeline whose checkpoint should be synchronized.
- sync_checkpoint_status(pipeline_name: str) dict [source]
Gets the checkpoint sync status of the pipeline
- Parameters:
pipeline_name – The name of the pipeline to check the checkpoint synchronization status of.
- wait_for_token(pipeline_name: str, token: str, timeout_s: float | None = 600)[source]
Blocks until all records represented by this completion token have been processed.
- Parameters:
pipeline_name – The name of the pipeline
token – The token to check for completion
timeout_s – The amount of time in seconds to wait for the pipeline to process these records. Default 600s
feldera.rest.feldera_config module
- class feldera.rest.feldera_config.FelderaConfig(cfg: dict)[source]
Bases:
object
General configuration of the current Feldera instance.
- class feldera.rest.feldera_config.FelderaEdition(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]
Bases:
Enum
The compilation profile to use when compiling the program. Represents the Feldera edition between Enterprise and Open source.
- ENTERPRISE = 'Enterprise'
The Enterprise version of Feldera.
- OPEN_SOURCE = 'Open source'
The open source version of Feldera.
feldera.rest.pipeline module
feldera.rest.sql_table module
feldera.rest.sql_view module
Module contents
This is the lower level REST client for Feldera.
This is a thin wrapper around the Feldera REST API.
It is recommended to use the higher level abstractions in the feldera package, instead of using the REST client directly.