feldera package
feldera.pipeline_builder module
- class feldera.pipeline_builder.PipelineBuilder(client: ~feldera.rest.feldera_client.FelderaClient, name: str, sql: str, description: str = '', compilation_profile: ~feldera.enums.CompilationProfile = CompilationProfile.OPTIMIZED, runtime_config: ~feldera.runtime_config.RuntimeConfig = <feldera.runtime_config.RuntimeConfig object>)[source]
Bases:
object
A builder for creating a Feldera Pipeline.
- Parameters:
client – The .FelderaClient instance
name – The name of the pipeline
description – The description of the pipeline
sql – The SQL code of the pipeline
compilation_profile – The compilation profile to use
runtime_config – The runtime config to use
feldera.pipeline module
- class feldera.pipeline.Pipeline(name: str, client: FelderaClient)[source]
Bases:
object
- foreach_chunk(view_name: str, callback: Callable[[DataFrame, int], None])[source]
Run the given callback on each chunk of the output of the specified view.
- Parameters:
view_name – The name of the view.
callback –
The callback to run on each chunk. The callback should take two arguments:
chunk -> The chunk as a pandas DataFrame
seq_no -> The sequence number. The sequence number is a monotonically increasing integer that starts from 0. Note that the sequence number is unique for each chunk, but not necessarily contiguous.
Please note that the callback is run in a separate thread, so it should be thread-safe. Please note that the callback should not block for a long time, as by default, backpressure is enabled and will block the pipeline.
Note
The callback must be thread-safe as it will be run in a separate thread.
- static get(name: str, client: FelderaClient) Pipeline [source]
Get the pipeline if it exists.
- Parameters:
name – The name of the pipeline.
client – The FelderaClient instance.
- input_json(table_name: str, data: Dict | list, update_format: str = 'raw', force: bool = False)[source]
Push this JSON data to the specified table of the pipeline.
- Parameters:
table_name – The name of the table to push data into.
data – The JSON encoded data to be pushed to the pipeline. The data should be in the form: {‘col1’: ‘val1’, ‘col2’: ‘val2’} or [{‘col1’: ‘val1’, ‘col2’: ‘val2’}, {‘col1’: ‘val1’, ‘col2’: ‘val2’}]
update_format – The update format of the JSON data to be pushed to the pipeline. Must be one of: “raw”, “insert_delete”. <https://docs.feldera.com/formats/json#the-insertdelete-format>
force – True to push data even if the pipeline is paused. False by default.
- input_pandas(table_name: str, df: DataFrame, force: bool = False)[source]
Push all rows in a pandas DataFrame to the pipeline.
- Parameters:
table_name – The name of the table to insert data into.
df – The pandas DataFrame to be pushed to the pipeline.
force – True to push data even if the pipeline is paused. False by default.
- listen(view_name: str) OutputHandler [source]
Listen to the output of the provided view so that it is available in the notebook / python code. When the pipeline is shutdown, these listeners are dropped.
- Parameters:
view_name – The name of the view to listen to.
- start()[source]
Starts this pipeline.
- Raises:
RuntimeError – If the pipeline returns unknown metrics.
- status() PipelineStatus [source]
Return the current status of the pipeline.
- wait_for_completion(shutdown: bool = False)[source]
Block until the pipeline has completed processing all input records.
This method blocks until (1) all input connectors attached to the pipeline have finished reading their input data sources and issued end-of-input notifications to the pipeline, and (2) all inputs received from these connectors have been fully processed and corresponding outputs have been sent out through the output connectors.
This method will block indefinitely if at least one of the input connectors attached to the pipeline is a streaming connector, such as Kafka, that does not issue the end-of-input notification.
- Parameters:
shutdown – If True, the pipeline will be shutdown after completion. False by default.
- Raises:
RuntimeError – If the pipeline returns unknown metrics.
- wait_for_idle(idle_interval_s: float = 5.0, timeout_s: float = 600.0, poll_interval_s: float = 0.2)[source]
Wait for the pipeline to become idle and then returns.
Idle is defined as a sufficiently long interval in which the number of input and processed records reported by the pipeline do not change, and they equal each other (thus, all input records present at the pipeline have been processed).
- Parameters:
idle_interval_s – Idle interval duration (default is 5.0 seconds).
timeout_s – Timeout waiting for idle (default is 600.0 seconds).
poll_interval_s – Polling interval, should be set substantially smaller than the idle interval (default is 0.2 seconds).
- Raises:
ValueError – If idle interval is larger than timeout, poll interval is larger than timeout, or poll interval is larger than idle interval.
RuntimeError – If the metrics are missing or the timeout was reached.
feldera.enums module
- class feldera.enums.BuildMode(value)[source]
Bases:
Enum
An enumeration.
- CREATE = 1
- GET = 2
- GET_OR_CREATE = 3
- class feldera.enums.CompilationProfile(value)[source]
Bases:
Enum
The compilation profile to use when compiling the program.
- DEV = 'dev'
The development compilation profile.
- OPTIMIZED = 'optimized'
The optimized compilation profile, the default for this API.
- SERVER_DEFAULT = None
The compiler server default compilation profile.
- UNOPTIMIZED = 'unoptimized'
The unoptimized compilation profile.
- class feldera.enums.PipelineStatus(value)[source]
Bases:
Enum
Represents the state that this pipeline is currently in.
Shutdown ◄────┐ │ │ /deploy │ │ │ ⌛ShuttingDown ▼ ▲ ⌛Provisioning │ │ │ Provisioned │ ▼ │/shutdown ⌛Initializing │ │ │ ┌────────┴─────────┴─┐ │ ▼ │ │ Paused │ │ │ ▲ │ │/start│ │/pause │ │ ▼ │ │ │ Running │ └──────────┬─────────┘ │ ▼ Failed
- FAILED = 8
The pipeline remains in this state until the users acknowledge the failure by issuing a call to shutdown the pipeline; transitions to the PipelineStatus.SHUTDOWN state.
- INITIALIZING = 4
The pipeline is initializing its internal state and connectors.
This state is part of the pipeline’s deployment process. In this state, the pipeline’s HTTP server is up and running, but its query engine and input and output connectors are still initializing.
The pipeline remains in this state until:
Initialization completes successfully; the pipeline transitions to the PipelineStatus.PAUSED state.
Initialization fails; transitions to the PipelineStatus.FAILED state.
A pre-defined timeout has passed. The runner performs forced shutdown of the pipeline; returns to the PipelineStatus.SHUTDOWN state.
The user cancels the pipeline by invoking the /shutdown endpoint. The manager performs forced shutdown of the pipeline; returns to the PipelineStatus.SHUTDOWN state.
- NOT_FOUND = 1
The pipeline has not been created yet.
- PAUSED = 5
The pipeline is fully initialized, but data processing has been paused.
The pipeline remains in this state until:
The user starts the pipeline by invoking the /start endpoint. The manager passes the request to the pipeline; transitions to the PipelineStatus.RUNNING state.
The user cancels the pipeline by invoking the /shutdown endpoint. The manager passes the shutdown request to the pipeline to perform a graceful shutdown; transitions to the PipelineStatus.SHUTTING_DOWN state.
An unexpected runtime error renders the pipeline PipelineStatus.FAILED.
- PROVISIONING = 3
The runner triggered a deployment of the pipeline and is waiting for the pipeline HTTP server to come up.
In this state, the runner provisions a runtime for the pipeline, starts the pipeline within this runtime and waits for it to start accepting HTTP requests.
The user is unable to communicate with the pipeline during this time. The pipeline remains in this state until:
Its HTTP server is up and running; the pipeline transitions to the PipelineStatus.INITIALIZING state.
A pre-defined timeout has passed. The runner performs forced shutdown of the pipeline; returns to the PipelineStatus.SHUTDOWN state.
The user cancels the pipeline by invoking the /shutdown endpoint. The manager performs forced shutdown of the pipeline, returns to the PipelineStatus.SHUTDOWN state.
- RUNNING = 6
The pipeline is processing data.
The pipeline remains in this state until:
The user pauses the pipeline by invoking the /pause endpoint. The manager passes the request to the pipeline; transitions to the PipelineStatus.PAUSED state.
The user cancels the pipeline by invoking the /shutdown endpoint. The runner passes the shutdown request to the pipeline to perform a graceful shutdown; transitions to the PipelineStatus.SHUTTING_DOWN state.
An unexpected runtime error renders the pipeline PipelineStatus.FAILED.
- SHUTDOWN = 2
Pipeline has not been started or has been shut down.
The pipeline remains in this state until the user triggers a deployment by invoking the /deploy endpoint.
- SHUTTING_DOWN = 7
Graceful shutdown in progress.
In this state, the pipeline finishes any ongoing data processing, produces final outputs, shuts down input/output connectors and terminates.
The pipeline remains in this state until:
Shutdown completes successfully; transitions to the PipelineStatus.SHUTDOWN state.
A pre-defined timeout has passed. The manager performs forced shutdown of the pipeline; returns to the PipelineStatus.SHUTDOWN state.
feldera.output_handler module
- class feldera.output_handler.OutputHandler(client: FelderaClient, pipeline_name: str, view_name: str, queue: Queue | None)[source]
Bases:
object
feldera.runtime_config module
- class feldera.runtime_config.Resources(config: Mapping[str, Any] | None = None, cpu_cores_max: int | None = None, cpu_cores_min: int | None = None, memory_mb_max: int | None = None, memory_mb_min: int | None = None, storage_class: str | None = None, storage_mb_max: int | None = None)[source]
Bases:
object
Class used to specify the resource configuration for a pipeline.
- Parameters:
config – A dictionary containing all the configuration values.
cpu_cores_max – The maximum number of CPU cores to reserve for an instance of the pipeline.
cpu_cores_min – The minimum number of CPU cores to reserve for an instance of the pipeline.
memory_mb_max – The maximum memory in Megabytes to reserve for an instance of the pipeline.
memory_mb_min – The minimum memory in Megabytes to reserve for an instance of the pipeline.
storage_class – The storage class to use for the pipeline. The class determines storage performance such as IOPS and throughput.
storage_mb_max – The storage in Megabytes to reserve for an instance of the pipeline.
- class feldera.runtime_config.RuntimeConfig(workers: int | None = None, storage: bool | None = False, tracing: bool | None = False, tracing_endpoint_jaeger: str | None = '', cpu_profiler: bool = True, max_buffering_delay_usecs: int = 0, min_batch_size_records: int = 0, min_storage_bytes: int | None = None, clock_resolution_usecs: int | None = None, resources: Resources | None = None)[source]
Bases:
object
Runtime configuration class to define the configuration for a pipeline.
Subpackages
- feldera.rest package
- Submodules
- feldera.rest.config module
- feldera.rest.errors module
- feldera.rest.feldera_client module
FelderaClient
FelderaClient.create_or_update_pipeline()
FelderaClient.create_pipeline()
FelderaClient.delete_pipeline()
FelderaClient.get_pipeline()
FelderaClient.get_pipeline_stats()
FelderaClient.get_runtime_config()
FelderaClient.listen_to_pipeline()
FelderaClient.patch_pipeline()
FelderaClient.pause_pipeline()
FelderaClient.pipelines()
FelderaClient.push_to_pipeline()
FelderaClient.shutdown_pipeline()
FelderaClient.start_pipeline()
- feldera.rest.pipeline module
- feldera.rest.sql_table module
- feldera.rest.sql_view module
- Module contents