feldera package

feldera.pipeline_builder module

class feldera.pipeline_builder.PipelineBuilder(client: ~feldera.rest.feldera_client.FelderaClient, name: str, sql: str, udf_rust: str = '', udf_toml: str = '', description: str = '', compilation_profile: ~feldera.enums.CompilationProfile = CompilationProfile.OPTIMIZED, runtime_config: ~feldera.runtime_config.RuntimeConfig = <feldera.runtime_config.RuntimeConfig object>)[source]

Bases: object

A builder for creating a Feldera Pipeline.

Parameters:
  • client – The FelderaClient instance

  • name – The name of the pipeline

  • description – The description of the pipeline

  • sql – The SQL code of the pipeline

  • udf_rust – Rust code for UDFs

  • udf_toml – Rust dependencies required by UDFs (in the TOML format)

  • compilation_profile – The CompilationProfile to use

  • runtime_config – The RuntimeConfig to use. Enables configuring the runtime behavior of the pipeline such as: fault tolerance, storage and Resources

create() Pipeline[source]

Create the pipeline if it does not exist.

Returns:

The created pipeline

create_or_replace() Pipeline[source]

Creates a pipeline if it does not exist and replaces it if it exists.

If the pipeline exists and is running, it will be stopped and replaced.

feldera.pipeline module

class feldera.pipeline.Pipeline(client: FelderaClient)[source]

Bases: object

checkpoint(wait: bool = False, timeout_s=300) int[source]

Checkpoints this pipeline.

Parameters:
  • wait – If true, will block until the checkpoint completes.

  • timeout_s – The maximum time (in seconds) to wait for the checkpoint to complete.

Raises:

FelderaAPIError – If enterprise features are not enabled.

checkpoint_status(seq: int) CheckpointStatus[source]

Checks the status of the given checkpoint.

Parameters:

seq – The checkpoint sequence number.

clear_storage()[source]

Clears the storage of the pipeline if it is currently in use. This action cannot be canceled, and will delete all the pipeline storage.

created_at() datetime[source]

Return the creation time of the pipeline.

delete(clear_storage: bool = False)[source]

Deletes the pipeline.

The pipeline must be stopped, and the storage cleared before it can be deleted.

Parameters:

clear_storage – True if the storage should be cleared before deletion. False by default

Raises:

FelderaAPIError – If the pipeline is not in STOPPED state or the storage is still bound.

deployment_config() Mapping[str, Any][source]

Return the deployment config of the pipeline.

deployment_desired_status() PipelineStatus[source]

Return the desired deployment status of the pipeline. This is the next state that the pipeline should transition to.

deployment_error() Mapping[str, Any][source]

Return the deployment error of the pipeline. Returns an empty string if there is no error.

deployment_location() str[source]

Return the deployment location of the pipeline. Deployment location is the location where the pipeline can be reached at runtime (a TCP port number or a URI).

deployment_status_since() datetime[source]

Return the timestamp when the current deployment status of the pipeline was set.

description() str[source]

Return the description of the pipeline.

errors() List[Mapping[str, Any]][source]

Returns a list of all errors in this pipeline.

execute(query: str)[source]

Executes an ad-hoc SQL query on the current pipeline, discarding its result. Unlike the query() method which returns a generator for retrieving query results lazily, this method processes the query eagerly and fully before returning.

This method is suitable for SQL operations like INSERT and DELETE, where the user needs confirmation of successful query execution, but does not require the query result. If the query fails, an exception will be raised.

Important:

If you try to INSERT or DELETE data from a table while the pipeline is paused, it will block until the pipeline is resumed.

Parameters:

query – The SQL query to be executed.

Raises:
foreach_chunk(view_name: str, callback: Callable[[DataFrame, int], None])[source]

Run the given callback on each chunk of the output of the specified view.

You must call this method before starting the pipeline to operate on the entire output. You can call this method after the pipeline has started, but you will only get the output from that point onwards.

Parameters:
  • view_name – The name of the view.

  • callback

    The callback to run on each chunk. The callback should take two arguments:

    • chunk -> The chunk as a pandas DataFrame

    • seq_no -> The sequence number. The sequence number is a monotonically increasing integer that starts from 0. Note that the sequence number is unique for each chunk, but not necessarily contiguous.

Please note that the callback is run in a separate thread, so it should be thread-safe. Please note that the callback should not block for a long time, as by default, backpressure is enabled and will block the pipeline.

Note

  • The callback must be thread-safe as it will be run in a separate thread.

static get(name: str, client: FelderaClient) Pipeline[source]

Get the pipeline if it exists.

Parameters:
  • name – The name of the pipeline.

  • client – The FelderaClient instance.

id() str[source]

Return the ID of the pipeline.

input_json(table_name: str, data: Dict | list, update_format: str = 'raw', force: bool = False)[source]

Push this JSON data to the specified table of the pipeline.

The pipeline must either be in RUNNING or PAUSED states to push data. An error will be raised if the pipeline is in any other state.

Parameters:
  • table_name – The name of the table to push data into.

  • data – The JSON encoded data to be pushed to the pipeline. The data should be in the form: {‘col1’: ‘val1’, ‘col2’: ‘val2’} or [{‘col1’: ‘val1’, ‘col2’: ‘val2’}, {‘col1’: ‘val1’, ‘col2’: ‘val2’}]

  • update_format – The update format of the JSON data to be pushed to the pipeline. Must be one of: “raw”, “insert_delete”. https://docs.feldera.com/formats/json#the-insertdelete-format

  • forceTrue to push data even if the pipeline is paused. False by default.

Raises:
  • ValueError – If the update format is invalid.

  • FelderaAPIError – If the pipeline is not in a valid state to push data.

  • RuntimeError – If the pipeline is paused and force is not set to True.

input_pandas(table_name: str, df: DataFrame, force: bool = False)[source]

Push all rows in a pandas DataFrame to the pipeline.

The pipeline must either be in RUNNING or PAUSED states to push data. An error will be raised if the pipeline is in any other state.

The dataframe must have the same columns as the table in the pipeline.

Parameters:
  • table_name – The name of the table to insert data into.

  • df – The pandas DataFrame to be pushed to the pipeline.

  • forceTrue to push data even if the pipeline is paused. False by default.

Raises:
  • ValueError – If the table does not exist in the pipeline.

  • RuntimeError – If the pipeline is not in a valid state to push data.

  • RuntimeError – If the pipeline is paused and force is not set to True.

listen(view_name: str) OutputHandler[source]

Follow the change stream (i.e., the output) of the provided view. Returns an output handler to read the changes.

When the pipeline is stopped, these listeners are dropped.

You must call this method before starting the pipeline to get the entire output of the view. If this method is called once the pipeline has started, you will only get the output from that point onwards.

Parameters:

view_name – The name of the view to listen to.

property name: str

Return the name of the pipeline.

pause(timeout_s: float | None = None)[source]

Pause the pipeline.

The pipeline can only transition to the PAUSED state from the RUNNING state. If the pipeline is already paused, it will remain in the PAUSED state.

Parameters:

timeout_s – The maximum time (in seconds) to wait for the pipeline to pause.

pause_connector(table_name: str, connector_name: str)[source]

Pause the specified input connector.

Connectors allow feldera to fetch data from a source or write data to a sink. This method allows users to PAUSE a specific INPUT connector. All connectors are RUNNING by default.

Refer to the connector documentation for more information:

https://docs.feldera.com/connectors/#input-connector-orchestration

Parameters:
  • table_name – The name of the table that the connector is attached to.

  • connector_name – The name of the connector to pause.

Raises:

FelderaAPIError – If the connector is not found, or if the pipeline is not running.

program_binary_url() str[source]

Return the program binary URL of the pipeline. This is the URL where the compiled program binary can be downloaded from.

program_code() str[source]

Return the program SQL code of the pipeline.

program_config() Mapping[str, Any][source]

Return the program config of the pipeline.

program_error() Mapping[str, Any][source]

Return the program error of the pipeline. If there are no errors, the exit_code field inside both sql_compilation and rust_compilation will be 0.

program_info() Mapping[str, Any][source]

Return the program info of the pipeline. This is the output returned by the SQL compiler, including: the list of input and output connectors, the generated Rust code for the pipeline, and the SQL program schema.

program_status() ProgramStatus[source]

Return the program status of the pipeline.

Program status is the status of compilation of this SQL program. We first compile the SQL program to Rust code, and then compile the Rust code to a binary.

program_status_since() datetime[source]

Return the timestamp when the current program status was set.

program_version() int[source]

Return the program version of the pipeline.

query(query: str) Generator[Mapping[str, Any], None, None][source]

Executes an ad-hoc SQL query on this pipeline and returns a generator that yields the rows of the result as Python dictionaries. For INSERT and DELETE queries, consider using execute() instead. All floating-point numbers are deserialized as Decimal objects to avoid precision loss.

Note:

You can only SELECT from materialized tables and views.

Important:

This method is lazy. It returns a generator and is not evaluated until you consume the result.

Parameters:

query – The SQL query to be executed.

Returns:

A generator that yields the rows of the result as Python dictionaries.

Raises:
query_parquet(query: str, path: str)[source]

Executes an ad-hoc SQL query on this pipeline and saves the result to the specified path as a parquet file. If the extension isn’t parquet, it will be automatically appended to path.

Note:

You can only SELECT from materialized tables and views.

Parameters:
  • query – The SQL query to be executed.

  • path – The path of the parquet file.

Raises:
query_tabular(query: str) Generator[str, None, None][source]

Executes a SQL query on this pipeline and returns the result as a formatted string.

Note:

You can only SELECT from materialized tables and views.

Important:

This method is lazy. It returns a generator and is not evaluated until you consume the result.

Parameters:

query – The SQL query to be executed.

Returns:

A generator that yields a string representing the query result in a human-readable, tabular format.

Raises:
refresh()[source]

Calls the backend to get the updated, latest version of the pipeline.

Raises:

FelderaConnectionError – If there is an issue connecting to the backend.

restart(timeout_s: float | None = None)[source]

Restarts the pipeline.

This method forcibly STOPS the pipeline regardless of its current state and then starts it again. No checkpoints are made when stopping the pipeline.

Parameters:

timeout_s – The maximum time (in seconds) to wait for the pipeline to restart.

resume(timeout_s: float | None = None)[source]

Resumes the pipeline from the PAUSED state. If the pipeline is already running, it will remain in the RUNNING state.

Parameters:

timeout_s – The maximum time (in seconds) to wait for the pipeline to resume.

resume_connector(table_name: str, connector_name: str)[source]

Resume the specified connector.

Connectors allow feldera to fetch data from a source or write data to a sink. This method allows users to RESUME / START a specific INPUT connector. All connectors are RUNNING by default.

Refer to the connector documentation for more information:

https://docs.feldera.com/connectors/#input-connector-orchestration

Parameters:
  • table_name – The name of the table that the connector is attached to.

  • connector_name – The name of the connector to resume.

Raises:

FelderaAPIError – If the connector is not found, or if the pipeline is not running.

runtime_config() RuntimeConfig[source]

Return the runtime config of the pipeline.

set_runtime_config(runtime_config: RuntimeConfig)[source]

Updates the runtime config of the pipeline. The pipeline must be stopped and, in addition, changing some pipeline configuration requires storage to be cleared.

For example, to set ‘min_batch_size_records’ on a pipeline:

runtime_config = pipeline.runtime_config() runtime_config.min_batch_size_records = 500 pipeline.set_runtime_config(runtime_config)

start(timeout_s: float | None = None)[source]

Starts this pipeline.

  • The pipeline must be in STOPPED state to start.

  • If the pipeline is in any other state, an error will be raised.

  • If the pipeline is in PAUSED state, use .meth:resume instead.

Parameters:

timeout_s – The maximum time (in seconds) to wait for the pipeline to start.

Raises:

RuntimeError – If the pipeline is not in STOPPED state.

stats() PipelineStatistics[source]

Gets the pipeline metrics and performance counters.

status() PipelineStatus[source]

Return the current status of the pipeline.

stop(force: bool, timeout_s: float | None = None)[source]

Stops the pipeline.

Stops the pipeline regardless of its current state.

Parameters:
  • force – Set True to immediately scale compute resources to zero. Set False to automatically checkpoint before stopping.

  • timeout_s – The maximum time (in seconds) to wait for the pipeline to stop.

storage_status() StorageStatus[source]

Return the storage status of the pipeline.

sync_checkpoint(wait: bool = False, timeout_s=300) str[source]

Syncs this checkpoint to object store.

Parameters:
  • wait – If true, will block until the checkpoint sync opeartion completes.

  • timeout_s – The maximum time (in seconds) to wait for the checkpoint to complete syncing.

Raises:

FelderaAPIError – If no checkpoints have been made.

sync_checkpoint_status(uuid: str) CheckpointStatus[source]

Checks the status of the given checkpoint sync operation. If the checkpoint is currently being synchronized, returns CheckpointStatus.Unknown.

Parameters:

uuid – The checkpoint uuid.

tables() List[SQLTable][source]

Return the tables of the pipeline.

udf_rust() str[source]

Return the Rust code for UDFs.

udf_toml() str[source]

Return the Rust dependencies required by UDFs (in the TOML format).

version() int[source]

Return the version of the pipeline.

views() List[SQLView][source]

Return the views of the pipeline.

wait_for_completion(force_stop: bool = False, timeout_s: float | None = None)[source]

Block until the pipeline has completed processing all input records.

This method blocks until (1) all input connectors attached to the pipeline have finished reading their input data sources and issued end-of-input notifications to the pipeline, and (2) all inputs received from these connectors have been fully processed and corresponding outputs have been sent out through the output connectors.

This method will block indefinitely if at least one of the input connectors attached to the pipeline is a streaming connector, such as Kafka, that does not issue the end-of-input notification.

Parameters:
  • force_stop – If True, the pipeline will be forcibly stopped after completion. False by default. No checkpoints will be made.

  • timeout_s – Optional. The maximum time (in seconds) to wait for the pipeline to complete. The default is None, which means wait indefinitely.

Raises:

RuntimeError – If the pipeline returns unknown metrics.

wait_for_idle(idle_interval_s: float = 5.0, timeout_s: float = 600.0, poll_interval_s: float = 0.2)[source]

Wait for the pipeline to become idle and then returns.

Idle is defined as a sufficiently long interval in which the number of input and processed records reported by the pipeline do not change, and they equal each other (thus, all input records present at the pipeline have been processed).

Parameters:
  • idle_interval_s – Idle interval duration (default is 5.0 seconds).

  • timeout_s – Timeout waiting for idle (default is 600.0 seconds).

  • poll_interval_s – Polling interval, should be set substantially smaller than the idle interval (default is 0.2 seconds).

Raises:
  • ValueError – If idle interval is larger than timeout, poll interval is larger than timeout, or poll interval is larger than idle interval.

  • RuntimeError – If the metrics are missing or the timeout was reached.

feldera.enums module

class feldera.enums.BuildMode(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

CREATE = 1
GET = 2
GET_OR_CREATE = 3
class feldera.enums.CheckpointStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Failure = 2
InProgress = 3
Success = 1
Unknown = 4
get_error() str | None[source]

Returns the error, if any.

class feldera.enums.CompilationProfile(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

The compilation profile to use when compiling the program.

DEV = 'dev'

The development compilation profile.

OPTIMIZED = 'optimized'

The optimized compilation profile, the default for this API.

SERVER_DEFAULT = None

The compiler server default compilation profile.

UNOPTIMIZED = 'unoptimized'

The unoptimized compilation profile.

class feldera.enums.FaultToleranceModel(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

The fault tolerance model.

AtLeastOnce = 1

Each record is output at least once. Crashes may duplicate output, but no input or output is dropped.

ExactlyOnce = 2

Each record is output exactly once. Crashes do not drop or duplicate input or output.

static from_str(value)[source]
class feldera.enums.PipelineStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Represents the state that this pipeline is currently in.

              Stopped ◄─────────── Stopping ◄───── All states can transition
                 │                    ▲            to Stopping by either:
/start or /pause │                    │            (1) user calling /stop?force=true, or;
                 ▼                    │            (2) pipeline encountering a fatal
          ⌛Provisioning          Suspending            resource or runtime error,
                 │                    ▲                having the system call /stop?force=true
                 ▼                    │ /stop          effectively
          ⌛Initializing ─────────────┤  ?force=false
                 │                    │
       ┌─────────┼────────────────────┴─────┐
       │         ▼                          │
       │       Paused  ◄──────► Unavailable │
       │        │   ▲                ▲      │
       │ /start │   │  /pause        │      │
       │        ▼   │                │      │
       │       Running ◄─────────────┘      │
       └────────────────────────────────────┘
INITIALIZING = 3

The pipeline is initializing its internal state and connectors.

The pipeline remains in this state until:

  1. Initialization succeeds, transitioning to PAUSED.

  2. Initialization fails or times out, transitioning to STOPPING.

  3. The user suspends the pipeline via /suspend, transitioning to SUSPENDING.

  4. The user stops the pipeline via /stop, transitioning to STOPPING.

NOT_FOUND = 0

The pipeline has not been created yet.

PAUSED = 4

The pipeline is initialized but data processing is paused.

The pipeline remains in this state until:

  1. The user starts it via /start, transitioning to RUNNING.

  2. A runtime error occurs, transitioning to STOPPING.

  3. The user suspends it via /suspend, transitioning to SUSPENDING.

  4. The user stops it via /stop, transitioning to STOPPING.

PROVISIONING = 2

Compute (and optionally storage) resources needed for running the pipeline are being provisioned.

The pipeline remains in this state until:

  1. Resources are provisioned successfully, transitioning to INITIALIZING.

  2. Provisioning fails or times out, transitioning to STOPPING.

  3. The user cancels the pipeline via /stop, transitioning to STOPPING.

RUNNING = 5

The pipeline is processing data.

The pipeline remains in this state until:

  1. The user pauses it via /pause, transitioning to PAUSED.

  2. A runtime error occurs, transitioning to STOPPING.

  3. The user suspends it via /suspend, transitioning to SUSPENDING.

  4. The user stops it via /stop, transitioning to STOPPING.

STOPPED = 1

The pipeline has not (yet) been started or has been stopped either manually by the user or automatically by the system due to a resource or runtime error.

The pipeline remains in this state until:

  1. The user starts it via /start or /pause, transitioning to PROVISIONING.

  2. Early start fails (e.g., compilation failure), transitioning to STOPPING.

STOPPING = 8

The pipeline’s compute resources are being scaled down to zero.

The pipeline remains in this state until deallocation completes, transitioning to STOPPED.

SUSPENDING = 7

The pipeline is being suspended to storage.

The pipeline remains in this state until:

  1. Suspension succeeds, transitioning to STOPPING.

  2. A runtime error occurs, transitioning to STOPPING.

UNAVAILABLE = 6

The pipeline was initialized at least once but is currently unreachable or not ready.

The pipeline remains in this state until:

  1. A successful status check transitions it back to PAUSED or RUNNING.

  2. A runtime error occurs, transitioning to STOPPING.

  3. The user suspends it via /suspend, transitioning to SUSPENDING.

  4. The user stops it via /stop, transitioning to STOPPING.

Note: While in this state, /start or /pause express desired state but are only applied once the pipeline becomes reachable.

static from_str(value)[source]
class feldera.enums.ProgramStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

CompilingRust = 4
CompilingSql = 2
Pending = 1
RustError = 7
SqlCompiled = 3
SqlError = 6
Success = 5
SystemError = 8
static from_value(value)[source]
get_error() dict | None[source]

Returns the compilation error, if any.

class feldera.enums.StorageStatus(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Represents the current storage usage status of the pipeline.

CLEARED = 0

The pipeline has not been started before, or the user has cleared storage.

In this state, the pipeline has no storage resources bound to it.

CLEARING = 2

The pipeline is in the process of becoming unbound from its storage resources.

If storage resources are configured to be deleted upon clearing, their deletion occurs before transitioning to CLEARED. Otherwise, no actual work is required, and the transition happens immediately.

If storage is not deleted during clearing, the responsibility to manage or delete those resources lies with the user.

INUSE = 1

The pipeline was (attempted to be) started before, transitioning from STOPPED to PROVISIONING, which caused the storage status to become INUSE.

Being in the INUSE state restricts certain edits while the pipeline is STOPPED.

The pipeline remains in this state until the user invokes /clear, transitioning it to CLEARING.

static from_str(value)[source]

feldera.output_handler module

class feldera.output_handler.OutputHandler(client: FelderaClient, pipeline_name: str, view_name: str, queue: Queue | None)[source]

Bases: object

start()[source]

Starts the output handler in a separate thread

to_dict(clear_buffer: bool = True)[source]

Returns the output of the pipeline as a list of python dictionaries

Parameters:

clear_buffer – Whether to clear the buffer after getting the output.

to_pandas(clear_buffer: bool = True)[source]

Returns the output of the pipeline as a pandas DataFrame

Parameters:

clear_buffer – Whether to clear the buffer after getting the output.

feldera.runtime_config module

class feldera.runtime_config.Resources(config: Mapping[str, Any] | None = None, cpu_cores_max: int | None = None, cpu_cores_min: int | None = None, memory_mb_max: int | None = None, memory_mb_min: int | None = None, storage_class: str | None = None, storage_mb_max: int | None = None)[source]

Bases: object

Class used to specify the resource configuration for a pipeline.

Parameters:
  • config – A dictionary containing all the configuration values.

  • cpu_cores_max – The maximum number of CPU cores to reserve for an instance of the pipeline.

  • cpu_cores_min – The minimum number of CPU cores to reserve for an instance of the pipeline.

  • memory_mb_max – The maximum memory in Megabytes to reserve for an instance of the pipeline.

  • memory_mb_min – The minimum memory in Megabytes to reserve for an instance of the pipeline.

  • storage_class – The storage class to use for the pipeline. The class determines storage performance such as IOPS and throughput.

  • storage_mb_max – The storage in Megabytes to reserve for an instance of the pipeline.

class feldera.runtime_config.RuntimeConfig(workers: int | None = None, storage: Storage | bool | None = None, tracing: bool | None = False, tracing_endpoint_jaeger: str | None = '', cpu_profiler: bool = True, max_buffering_delay_usecs: int = 0, min_batch_size_records: int = 0, clock_resolution_usecs: int | None = None, provisioning_timeout_secs: int | None = None, resources: Resources | None = None, runtime_version: str | None = None, fault_tolerance_model: FaultToleranceModel | None = None, checkpoint_interval_secs: int | None = None)[source]

Bases: object

Runtime configuration class to define the configuration for a pipeline. To create runtime config from a dictionary, use RuntimeConfig.from_dict().

Documentation:

https://docs.feldera.com/pipelines/configuration/#runtime-configuration

static default() RuntimeConfig[source]
classmethod from_dict(d: Mapping[str, Any])[source]

Create a RuntimeConfig object from a dictionary.

to_dict() dict[source]
class feldera.runtime_config.Storage(config: Mapping[str, Any] | None = None, min_storage_bytes: int | None = None)[source]

Bases: object

Storage configuration for a pipeline.

Parameters:

min_storage_bytes – The minimum estimated number of bytes in a batch of data to write it to storage.

Subpackages