Upsert Pipeline

Fully update a pipeline if it already exists, otherwise create a new pipeline.

Path Parameters
`pipeline_name` string — REQUIRED Unique pipeline name

Request Body — REQUIRED

description string

name string — REQUIRED

program_code string — REQUIRED

program_config object

Program configuration.

cache boolean

If true (default), when a prior compilation with the same checksum already exists, the output of that (i.e., binary) is used. Set false to always trigger a new compilation, which might take longer and as well can result in overriding an existing binary.

profile string

Possible values: [dev, unoptimized, optimized, optimized_symbols]

Enumeration of possible compilation profiles that can be passed to the Rust compiler as an argument via cargo build --profile <>. A compilation profile affects among other things the compilation speed (how long till the program is ready to be run) and runtime speed (the performance while running).

runtime_version string

Override runtime version of the pipeline being executed.

Warning: This setting is experimental and may change in the future. Requires the platform to run with the unstable feature runtime_version enabled. Should only be used for testing purposes, and requires network access.

A runtime version can be specified in the form of a version or SHA taken from the feldera/feldera repository main branch.

Examples: v0.96.0 or f4dcac0989ca0fda7d2eb93602a49d007cb3b0ae

A platform of version 0.x.y may be capable of running future and past runtimes with versions >=0.x.y and <=0.x.y until breaking API changes happen, the exact bounds for each platform version are unspecified until we reach a stable version. Compatibility is only guaranteed if platform and runtime version are exact matches.

Note that any enterprise features are currently considered to be part of the platform.

If not set (null), the runtime version will be the same as the platform version.

runtime_config object

Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.

checkpoint_during_suspend boolean

Deprecated: setting this true or false does not have an effect anymore.

clock_resolution_usecs int64

Real-time clock resolution in microseconds.

This parameter controls the execution of queries that use the NOW() function. The output of such queries depends on the real-time clock and can change over time without any external inputs. If the query uses NOW(), the pipeline will update the clock value and trigger incremental recomputation at most each clock_resolution_usecs microseconds. If the query does not use NOW(), then clock value updates are suppressed and the pipeline ignores this setting.

It is set to 1 second (1,000,000 microseconds) by default.

cpu_profiler boolean

Enable CPU profiler.

The default value is true.

dev_tweaks object

Optional settings for tweaking Feldera internals.

The available key-value pairs change from one version of Feldera to another, so users should not depend on particular settings being available, or on their behavior.

fault_tolerance object

Fault-tolerance configuration.

The default [FtConfig] (via [FtConfig::default]) disables fault tolerance, which is the configuration that one gets if [RuntimeConfig] omits fault tolerance configuration.

The default value for [FtConfig::model] enables fault tolerance, as Some(FtModel::default()). This is the configuration that one gets if [RuntimeConfig] includes a fault tolerance configuration but does not specify a particular model.

checkpoint_interval_secs int64

Interval between automatic checkpoints, in seconds.

The default is 60 seconds. Values less than 1 or greater than 3600 will be forced into that range.

model

hosts integer

Number of DBSP hosts.

The worker threads are evenly divided among the hosts. For single-host deployments, this should be 1 (the default).

Multihost pipelines are an enterprise-only preview feature.

http_workers int64

Sets the number of available runtime threads for the http server.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case the pipeline HTTP API operations are a bottleneck.

If not specified, the default is set to workers.

init_containers

Specification of additional (sidecar) containers.

io_workers int64

Sets the number of available runtime threads for async IO tasks.

This affects some networking and file I/O operations especially adapters and ad-hoc queries.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case ingress, egress or ad-hoc queries are a bottleneck.

If not specified, the default is set to workers.

logging string

Log filtering directives.

If set to a valid tracing-subscriber filter, this controls the log messages emitted by the pipeline process. Otherwise, or if the filter has invalid syntax, messages at "info" severity and higher are written to the log and all others are discarded.

max_buffering_delay_usecs int64

Maximal delay in microseconds to wait for min_batch_size_records to get buffered by the controller, defaults to 0.

max_parallel_connector_init int64

The maximum number of connectors initialized in parallel during pipeline startup.

At startup, the pipeline must initialize all of its input and output connectors. Depending on the number and types of connectors, this can take a long time. To accelerate the process, multiple connectors are initialized concurrently. This option controls the maximum number of connectors that can be initialized in parallel.

The default is 10.

min_batch_size_records int64

Minimal input batch size.

The controller delays pushing input records to the circuit until at least min_batch_size_records records have been received (total across all endpoints) or max_buffering_delay_usecs microseconds have passed since at least one input records has been buffered. Defaults to 0.

pin_cpus integer[]

Optionally, a list of CPU numbers for CPUs to which the pipeline may pin its worker threads. Specify at least twice as many CPU numbers as workers. CPUs are generally numbered starting from 0. The pipeline might not be able to honor CPU pinning requests.

CPU pinning can make pipelines run faster and perform more consistently, as long as different pipelines running on the same machine are pinned to different CPUs.

pipeline_template_configmap object

Configuration for supplying a custom pipeline StatefulSet template via a Kubernetes ConfigMap.

Operators can provide a custom StatefulSet YAML that the Kubernetes runner will use when creating pipeline StatefulSets for a pipeline. The custom template must be stored as the value of a key in a ConfigMap in the same namespace as the pipeline; set name to the ConfigMap name and key to the entry that contains the template.

Recommendations and requirements:

Start from the default template and modify it as needed. The default template is present in ConfigMap named as <release-name>-pipeline-template, with key pipelineTemplate in the release namespace and should be used as a reference.
The template must contain a valid Kubernetes StatefulSet manifest in YAML form. The runner substitutes variables in the template before parsing; therefore the final YAML must be syntactically valid.
The runner performs simple string substitution for the following placeholders. Please ensure these placeholders are placed at appropriate location for their semantics:
{id}: pipeline Kubernetes name (used for object names and labels)
{namespace}: Kubernetes namespace where the pipeline runs
{pipeline_executor_image}: container image used to run the pipeline executor
{binary_ref}: program binary reference passed as an argument
{program_info_ref}: program info reference passed as an argument
{pipeline_storage_path}: mount path for persistent pipeline storage
{storage_class_name}: storage class name to use for PVCs (if applicable)
{deployment_id}: UUID identifying the deployment instance
{deployment_initial}: initial desired runtime status (e.g., provisioning)
{bootstrap_policy}: bootstrap policy value when applicable

key string

Key in the ConfigMap containing the pipeline template.

If not set, defaults to pipelineTemplate.

name string — REQUIRED

Name of the ConfigMap containing the pipeline template.

provisioning_timeout_secs int64

Timeout in seconds for the Provisioning phase of the pipeline. Setting this value will override the default of the runner.

resources object

cpu_cores_max double

The maximum number of CPU cores to reserve for an instance of this pipeline

cpu_cores_min double

The minimum number of CPU cores to reserve for an instance of this pipeline

memory_mb_max int64

The maximum memory in Megabytes to reserve for an instance of this pipeline

memory_mb_min int64

The minimum memory in Megabytes to reserve for an instance of this pipeline

namespace string

Kubernetes namespace to use for an instance of this pipeline. The namespace determines the scope of names for resources created for the pipeline. If not set, the pipeline will be deployed in the same namespace as the control-plane.

service_account_name string

Kubernetes service account name to use for an instance of this pipeline. The account determines permissions and access controls.

storage_class string

Storage class to use for an instance of this pipeline. The class determines storage performance such as IOPS and throughput.

storage_mb_max int64

The total storage in Megabytes to reserve for an instance of this pipeline

storage object

Storage configuration for a pipeline.

backend

Backend storage configuration.

cache_mib integer

The maximum size of the in-memory storage cache, in MiB.

If set, the specified cache size is spread across all the foreground and background threads. If unset, each foreground or background thread cache is limited to 256 MiB.

compression string

Possible values: [default, none, snappy]

Storage compression algorithm.

min_step_storage_bytes integer

For a batch of data passed through the pipeline during a single step, the minimum estimated number of bytes to write it to storage.

This is provided for debugging and fine-tuning and should ordinarily be left unset. A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. usize::MAX, the default, effectively disables storage for such batches. If it is set to another value, it should ordinarily be greater than or equal to min_storage_bytes.

min_storage_bytes integer

For a batch of data maintained as part of a persistent index during a pipeline run, the minimum estimated number of bytes to write it to storage.

This is provided for debugging and fine-tuning and should ordinarily be left unset.

A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. usize::MAX would effectively disable storage for such batches. The default is 10,048,576 (10 MiB).

tracing boolean

Enable pipeline tracing.

tracing_endpoint_jaeger string

Jaeger tracing endpoint to send tracing information to.

workers int32

Number of DBSP worker threads.

Each DBSP "foreground" worker thread is paired with a "background" thread for LSM merging, making the total number of threads twice the specified number.

The typical sweet spot for the number of workers is between 4 and 16. Each worker increases overall memory consumption for data structures used during a step.

udf_rust string

udf_toml string

Responses

200

Pipeline successfully updated

Schema — OPTIONAL

created_at date-time

deployment_desired_status string

Possible values: [Stopped, Unavailable, Standby, Paused, Running, Suspended]

deployment_desired_status_since date-time

deployment_error object — OPTIONAL

Information returned by REST API endpoints on error.

details

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.

deployment_id uuid — OPTIONAL

deployment_initial string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Paused, Running, Suspended]

deployment_resources_desired_status string

Possible values: [Stopped, Provisioned]

deployment_resources_desired_status_since date-time

deployment_resources_status string

Possible values: [Stopped, Provisioning, Provisioned, Stopping]

Pipeline resources status.

/start (early start failed)
┌───────────────────┐
│                   ▼
Stopped ◄────────── Stopping
/start │                   ▲
│                   │ /stop?force=true
│                   │ OR: timeout (from Provisioning)
▼                   │ OR: fatal runtime or resource error
⌛Provisioning ────────────│ OR: runtime status is Suspended
│                   │
│                   │
▼                   │
Provisioned ─────────────┘

Desired and actual status

We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the desired status, which represents what the user would like the pipeline to do, and the current status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards.

There are two desired statuses:

Provisioned (set by invoking /start)
Stopped (set by invoking /stop?force=true)

The user can monitor the current status of the pipeline via the GET /v0/pipelines/{name} endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the /start endpoint, and then polls the GET /v0/pipelines/{name} endpoint to monitor the actual status of the pipeline until its deployment_resources_status attribute changes to Provisioned indicating that the pipeline has been successfully provisioned, or Stopped with deployment_error being set.

deployment_resources_status_details — OPTIONAL

deployment_resources_status_since date-time

deployment_runtime_desired_status string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Paused, Running, Suspended]

deployment_runtime_desired_status_since date-time — OPTIONAL

deployment_runtime_status string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Initializing, AwaitingApproval, Bootstrapping, Replaying, Paused, Running, Suspended]

Runtime status of the pipeline.

Of the statuses, only Unavailable is determined by the runner. All other statuses are determined by the pipeline and taken over by the runner.

deployment_runtime_status_details — OPTIONAL

deployment_runtime_status_since date-time — OPTIONAL

deployment_status string

Possible values: [Stopped, Provisioning, Unavailable, Coordination, Standby, AwaitingApproval, Initializing, Bootstrapping, Replaying, Paused, Running, Suspended, Stopping]

deployment_status_since date-time

description string

id uuid

Pipeline identifier.

name string

platform_version string

program_code string

program_config object

Program configuration.

cache boolean — OPTIONAL

profile string — OPTIONAL

Possible values: [dev, unoptimized, optimized, optimized_symbols]

runtime_version string — OPTIONAL

Override runtime version of the pipeline being executed.

A runtime version can be specified in the form of a version or SHA taken from the feldera/feldera repository main branch.

Examples: v0.96.0 or f4dcac0989ca0fda7d2eb93602a49d007cb3b0ae

Note that any enterprise features are currently considered to be part of the platform.

If not set (null), the runtime version will be the same as the platform version.

program_error object

Log, warning and error information about the program compilation.

rust_compilation object — OPTIONAL

Rust compilation information.

exit_code int32

Exit code of the cargo compilation command.

stderr string

Output printed to stderr by the cargo compilation command.

stdout string

Output printed to stdout by the cargo compilation command.

sql_compilation object — OPTIONAL

SQL compilation information.

exit_code int32

Exit code of the SQL compiler.

messages object[]

Messages (warnings and errors) generated by the SQL compiler.

end_column integer

end_line_number integer

error_type string

message string

snippet string — OPTIONAL

start_column integer

start_line_number integer

warning boolean

system_error string — OPTIONAL

System error that occurred.

Set Some(...) upon transition to SystemError
Set None upon transition to Pending

program_info object — OPTIONAL

Program information is the result of the SQL compilation.

input_connectors object

Input connectors derived from the schema.

output_connectors object

Output connectors derived from the schema.

schema object

A struct containing the tables (inputs) and views for a program.

Parse from the JSON data-type of the DDL generated by the SQL compiler.

inputs object[]

case_sensitive boolean

name string

fields object[]

case_sensitive boolean

name string

columntype (circular)

default string — OPTIONAL

lateness string — OPTIONAL

unused boolean

watermark string — OPTIONAL

materialized boolean — OPTIONAL

properties object — OPTIONAL

outputs object[]

case_sensitive boolean

name string

fields object[]

case_sensitive boolean

name string

columntype (circular)

default string — OPTIONAL

lateness string — OPTIONAL

unused boolean

watermark string — OPTIONAL

materialized boolean — OPTIONAL

properties object — OPTIONAL

udf_stubs string

Generated user defined function (UDF) stubs Rust code: stubs.rs

program_status string

Possible values: [Pending, CompilingSql, SqlCompiled, CompilingRust, Success, SqlError, RustError, SystemError]

Program compilation status.

program_status_since date-time

program_version int64

Version number.

refresh_version int64

Version number.

runtime_config object

Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.

checkpoint_during_suspend boolean — OPTIONAL

Deprecated: setting this true or false does not have an effect anymore.

clock_resolution_usecs int64 — OPTIONAL

Real-time clock resolution in microseconds.

It is set to 1 second (1,000,000 microseconds) by default.

cpu_profiler boolean — OPTIONAL

Enable CPU profiler.

The default value is true.

dev_tweaks object — OPTIONAL

Optional settings for tweaking Feldera internals.

The available key-value pairs change from one version of Feldera to another, so users should not depend on particular settings being available, or on their behavior.

fault_tolerance object — OPTIONAL

Fault-tolerance configuration.

The default [FtConfig] (via [FtConfig::default]) disables fault tolerance, which is the configuration that one gets if [RuntimeConfig] omits fault tolerance configuration.

checkpoint_interval_secs int64 — OPTIONAL

Interval between automatic checkpoints, in seconds.

The default is 60 seconds. Values less than 1 or greater than 3600 will be forced into that range.

model — OPTIONAL

hosts integer — OPTIONAL

Number of DBSP hosts.

The worker threads are evenly divided among the hosts. For single-host deployments, this should be 1 (the default).

Multihost pipelines are an enterprise-only preview feature.

http_workers int64 — OPTIONAL

Sets the number of available runtime threads for the http server.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case the pipeline HTTP API operations are a bottleneck.

If not specified, the default is set to workers.

init_containers — OPTIONAL

Specification of additional (sidecar) containers.

io_workers int64 — OPTIONAL

Sets the number of available runtime threads for async IO tasks.

This affects some networking and file I/O operations especially adapters and ad-hoc queries.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case ingress, egress or ad-hoc queries are a bottleneck.

If not specified, the default is set to workers.

logging string — OPTIONAL

Log filtering directives.

max_buffering_delay_usecs int64 — OPTIONAL

Maximal delay in microseconds to wait for min_batch_size_records to get buffered by the controller, defaults to 0.

max_parallel_connector_init int64 — OPTIONAL

The maximum number of connectors initialized in parallel during pipeline startup.

The default is 10.

min_batch_size_records int64 — OPTIONAL

Minimal input batch size.

pin_cpus integer[] — OPTIONAL

CPU pinning can make pipelines run faster and perform more consistently, as long as different pipelines running on the same machine are pinned to different CPUs.

pipeline_template_configmap object — OPTIONAL

Configuration for supplying a custom pipeline StatefulSet template via a Kubernetes ConfigMap.

Recommendations and requirements:

Start from the default template and modify it as needed. The default template is present in ConfigMap named as <release-name>-pipeline-template, with key pipelineTemplate in the release namespace and should be used as a reference.
The template must contain a valid Kubernetes StatefulSet manifest in YAML form. The runner substitutes variables in the template before parsing; therefore the final YAML must be syntactically valid.
The runner performs simple string substitution for the following placeholders. Please ensure these placeholders are placed at appropriate location for their semantics:
{id}: pipeline Kubernetes name (used for object names and labels)
{namespace}: Kubernetes namespace where the pipeline runs
{pipeline_executor_image}: container image used to run the pipeline executor
{binary_ref}: program binary reference passed as an argument
{program_info_ref}: program info reference passed as an argument
{pipeline_storage_path}: mount path for persistent pipeline storage
{storage_class_name}: storage class name to use for PVCs (if applicable)
{deployment_id}: UUID identifying the deployment instance
{deployment_initial}: initial desired runtime status (e.g., provisioning)
{bootstrap_policy}: bootstrap policy value when applicable

key string — OPTIONAL

Key in the ConfigMap containing the pipeline template.

If not set, defaults to pipelineTemplate.

name string

Name of the ConfigMap containing the pipeline template.

provisioning_timeout_secs int64 — OPTIONAL

Timeout in seconds for the Provisioning phase of the pipeline. Setting this value will override the default of the runner.

resources object — OPTIONAL

cpu_cores_max double — OPTIONAL

The maximum number of CPU cores to reserve for an instance of this pipeline

cpu_cores_min double — OPTIONAL

The minimum number of CPU cores to reserve for an instance of this pipeline

memory_mb_max int64 — OPTIONAL

The maximum memory in Megabytes to reserve for an instance of this pipeline

memory_mb_min int64 — OPTIONAL

The minimum memory in Megabytes to reserve for an instance of this pipeline

namespace string — OPTIONAL

service_account_name string — OPTIONAL

Kubernetes service account name to use for an instance of this pipeline. The account determines permissions and access controls.

storage_class string — OPTIONAL

Storage class to use for an instance of this pipeline. The class determines storage performance such as IOPS and throughput.

storage_mb_max int64 — OPTIONAL

The total storage in Megabytes to reserve for an instance of this pipeline

storage object — OPTIONAL

Storage configuration for a pipeline.

backend — OPTIONAL

Backend storage configuration.

cache_mib integer — OPTIONAL

The maximum size of the in-memory storage cache, in MiB.

If set, the specified cache size is spread across all the foreground and background threads. If unset, each foreground or background thread cache is limited to 256 MiB.

compression string — OPTIONAL

Possible values: [default, none, snappy]

Storage compression algorithm.

min_step_storage_bytes integer — OPTIONAL

For a batch of data passed through the pipeline during a single step, the minimum estimated number of bytes to write it to storage.

min_storage_bytes integer — OPTIONAL

For a batch of data maintained as part of a persistent index during a pipeline run, the minimum estimated number of bytes to write it to storage.

This is provided for debugging and fine-tuning and should ordinarily be left unset.

A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. usize::MAX would effectively disable storage for such batches. The default is 10,048,576 (10 MiB).

tracing boolean — OPTIONAL

Enable pipeline tracing.

tracing_endpoint_jaeger string — OPTIONAL

Jaeger tracing endpoint to send tracing information to.

workers int32 — OPTIONAL

Number of DBSP worker threads.

Each DBSP "foreground" worker thread is paired with a "background" thread for LSM merging, making the total number of threads twice the specified number.

The typical sweet spot for the number of workers is between 4 and 16. Each worker increases overall memory consumption for data structures used during a step.

storage_status string

Possible values: [Cleared, InUse, Clearing]

Storage status.

The storage status can only transition when the resources status is Stopped.

Cleared ───┐
▲       │
/clear │       │
│       │
Clearing   │
▲       │
│       │
InUse ◄───┘

udf_rust string

udf_toml string

version int64

Version number.

201

Pipeline successfully created

Schema — OPTIONAL

created_at date-time

deployment_desired_status string

Possible values: [Stopped, Unavailable, Standby, Paused, Running, Suspended]

deployment_desired_status_since date-time

deployment_error object — OPTIONAL

Information returned by REST API endpoints on error.

details

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.

deployment_id uuid — OPTIONAL

deployment_initial string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Paused, Running, Suspended]

deployment_resources_desired_status string

Possible values: [Stopped, Provisioned]

deployment_resources_desired_status_since date-time

deployment_resources_status string

Possible values: [Stopped, Provisioning, Provisioned, Stopping]

Pipeline resources status.

/start (early start failed)
┌───────────────────┐
│                   ▼
Stopped ◄────────── Stopping
/start │                   ▲
│                   │ /stop?force=true
│                   │ OR: timeout (from Provisioning)
▼                   │ OR: fatal runtime or resource error
⌛Provisioning ────────────│ OR: runtime status is Suspended
│                   │
│                   │
▼                   │
Provisioned ─────────────┘

Desired and actual status

There are two desired statuses:

Provisioned (set by invoking /start)
Stopped (set by invoking /stop?force=true)

deployment_resources_status_details — OPTIONAL

deployment_resources_status_since date-time

deployment_runtime_desired_status string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Paused, Running, Suspended]

deployment_runtime_desired_status_since date-time — OPTIONAL

deployment_runtime_status string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Initializing, AwaitingApproval, Bootstrapping, Replaying, Paused, Running, Suspended]

Runtime status of the pipeline.

Of the statuses, only Unavailable is determined by the runner. All other statuses are determined by the pipeline and taken over by the runner.

deployment_runtime_status_details — OPTIONAL

deployment_runtime_status_since date-time — OPTIONAL

deployment_status string

deployment_status_since date-time

description string

id uuid

Pipeline identifier.

name string

platform_version string

program_code string

program_config object

Program configuration.

cache boolean — OPTIONAL

profile string — OPTIONAL

Possible values: [dev, unoptimized, optimized, optimized_symbols]

runtime_version string — OPTIONAL

Override runtime version of the pipeline being executed.

A runtime version can be specified in the form of a version or SHA taken from the feldera/feldera repository main branch.

Examples: v0.96.0 or f4dcac0989ca0fda7d2eb93602a49d007cb3b0ae

Note that any enterprise features are currently considered to be part of the platform.

If not set (null), the runtime version will be the same as the platform version.

program_error object

Log, warning and error information about the program compilation.

rust_compilation object — OPTIONAL

Rust compilation information.

exit_code int32

Exit code of the cargo compilation command.

stderr string

Output printed to stderr by the cargo compilation command.

stdout string

Output printed to stdout by the cargo compilation command.

sql_compilation object — OPTIONAL

SQL compilation information.

exit_code int32

Exit code of the SQL compiler.

messages object[]

Messages (warnings and errors) generated by the SQL compiler.

end_column integer

end_line_number integer

error_type string

message string

snippet string — OPTIONAL

start_column integer

start_line_number integer

warning boolean

system_error string — OPTIONAL

System error that occurred.

Set Some(...) upon transition to SystemError
Set None upon transition to Pending

program_info object — OPTIONAL

Program information is the result of the SQL compilation.

input_connectors object

Input connectors derived from the schema.

output_connectors object

Output connectors derived from the schema.

schema object

A struct containing the tables (inputs) and views for a program.

Parse from the JSON data-type of the DDL generated by the SQL compiler.

inputs object[]

case_sensitive boolean

name string

fields object[]

case_sensitive boolean

name string

columntype (circular)

default string — OPTIONAL

lateness string — OPTIONAL

unused boolean

watermark string — OPTIONAL

materialized boolean — OPTIONAL

properties object — OPTIONAL

outputs object[]

case_sensitive boolean

name string

fields object[]

case_sensitive boolean

name string

columntype (circular)

default string — OPTIONAL

lateness string — OPTIONAL

unused boolean

watermark string — OPTIONAL

materialized boolean — OPTIONAL

properties object — OPTIONAL

udf_stubs string

Generated user defined function (UDF) stubs Rust code: stubs.rs

program_status string

Possible values: [Pending, CompilingSql, SqlCompiled, CompilingRust, Success, SqlError, RustError, SystemError]

Program compilation status.

program_status_since date-time

program_version int64

Version number.

refresh_version int64

Version number.

runtime_config object

Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.

checkpoint_during_suspend boolean — OPTIONAL

Deprecated: setting this true or false does not have an effect anymore.

clock_resolution_usecs int64 — OPTIONAL

Real-time clock resolution in microseconds.

It is set to 1 second (1,000,000 microseconds) by default.

cpu_profiler boolean — OPTIONAL

Enable CPU profiler.

The default value is true.

dev_tweaks object — OPTIONAL

Optional settings for tweaking Feldera internals.

The available key-value pairs change from one version of Feldera to another, so users should not depend on particular settings being available, or on their behavior.

fault_tolerance object — OPTIONAL

Fault-tolerance configuration.

The default [FtConfig] (via [FtConfig::default]) disables fault tolerance, which is the configuration that one gets if [RuntimeConfig] omits fault tolerance configuration.

checkpoint_interval_secs int64 — OPTIONAL

Interval between automatic checkpoints, in seconds.

The default is 60 seconds. Values less than 1 or greater than 3600 will be forced into that range.

model — OPTIONAL

hosts integer — OPTIONAL

Number of DBSP hosts.

The worker threads are evenly divided among the hosts. For single-host deployments, this should be 1 (the default).

Multihost pipelines are an enterprise-only preview feature.

http_workers int64 — OPTIONAL

Sets the number of available runtime threads for the http server.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case the pipeline HTTP API operations are a bottleneck.

If not specified, the default is set to workers.

init_containers — OPTIONAL

Specification of additional (sidecar) containers.

io_workers int64 — OPTIONAL

Sets the number of available runtime threads for async IO tasks.

This affects some networking and file I/O operations especially adapters and ad-hoc queries.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case ingress, egress or ad-hoc queries are a bottleneck.

If not specified, the default is set to workers.

logging string — OPTIONAL

Log filtering directives.

max_buffering_delay_usecs int64 — OPTIONAL

Maximal delay in microseconds to wait for min_batch_size_records to get buffered by the controller, defaults to 0.

max_parallel_connector_init int64 — OPTIONAL

The maximum number of connectors initialized in parallel during pipeline startup.

The default is 10.

min_batch_size_records int64 — OPTIONAL

Minimal input batch size.

pin_cpus integer[] — OPTIONAL

CPU pinning can make pipelines run faster and perform more consistently, as long as different pipelines running on the same machine are pinned to different CPUs.

pipeline_template_configmap object — OPTIONAL

Configuration for supplying a custom pipeline StatefulSet template via a Kubernetes ConfigMap.

Recommendations and requirements:

Start from the default template and modify it as needed. The default template is present in ConfigMap named as <release-name>-pipeline-template, with key pipelineTemplate in the release namespace and should be used as a reference.
The template must contain a valid Kubernetes StatefulSet manifest in YAML form. The runner substitutes variables in the template before parsing; therefore the final YAML must be syntactically valid.
The runner performs simple string substitution for the following placeholders. Please ensure these placeholders are placed at appropriate location for their semantics:
{id}: pipeline Kubernetes name (used for object names and labels)
{namespace}: Kubernetes namespace where the pipeline runs
{pipeline_executor_image}: container image used to run the pipeline executor
{binary_ref}: program binary reference passed as an argument
{program_info_ref}: program info reference passed as an argument
{pipeline_storage_path}: mount path for persistent pipeline storage
{storage_class_name}: storage class name to use for PVCs (if applicable)
{deployment_id}: UUID identifying the deployment instance
{deployment_initial}: initial desired runtime status (e.g., provisioning)
{bootstrap_policy}: bootstrap policy value when applicable

key string — OPTIONAL

Key in the ConfigMap containing the pipeline template.

If not set, defaults to pipelineTemplate.

name string

Name of the ConfigMap containing the pipeline template.

provisioning_timeout_secs int64 — OPTIONAL

Timeout in seconds for the Provisioning phase of the pipeline. Setting this value will override the default of the runner.

resources object — OPTIONAL

cpu_cores_max double — OPTIONAL

The maximum number of CPU cores to reserve for an instance of this pipeline

cpu_cores_min double — OPTIONAL

The minimum number of CPU cores to reserve for an instance of this pipeline

memory_mb_max int64 — OPTIONAL

The maximum memory in Megabytes to reserve for an instance of this pipeline

memory_mb_min int64 — OPTIONAL

The minimum memory in Megabytes to reserve for an instance of this pipeline

namespace string — OPTIONAL

service_account_name string — OPTIONAL

Kubernetes service account name to use for an instance of this pipeline. The account determines permissions and access controls.

storage_class string — OPTIONAL

Storage class to use for an instance of this pipeline. The class determines storage performance such as IOPS and throughput.

storage_mb_max int64 — OPTIONAL

The total storage in Megabytes to reserve for an instance of this pipeline

storage object — OPTIONAL

Storage configuration for a pipeline.

backend — OPTIONAL

Backend storage configuration.

cache_mib integer — OPTIONAL

The maximum size of the in-memory storage cache, in MiB.

If set, the specified cache size is spread across all the foreground and background threads. If unset, each foreground or background thread cache is limited to 256 MiB.

compression string — OPTIONAL

Possible values: [default, none, snappy]

Storage compression algorithm.

min_step_storage_bytes integer — OPTIONAL

For a batch of data passed through the pipeline during a single step, the minimum estimated number of bytes to write it to storage.

min_storage_bytes integer — OPTIONAL

For a batch of data maintained as part of a persistent index during a pipeline run, the minimum estimated number of bytes to write it to storage.

This is provided for debugging and fine-tuning and should ordinarily be left unset.

A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. usize::MAX would effectively disable storage for such batches. The default is 10,048,576 (10 MiB).

tracing boolean — OPTIONAL

Enable pipeline tracing.

tracing_endpoint_jaeger string — OPTIONAL

Jaeger tracing endpoint to send tracing information to.

workers int32 — OPTIONAL

Number of DBSP worker threads.

Each DBSP "foreground" worker thread is paired with a "background" thread for LSM merging, making the total number of threads twice the specified number.

The typical sweet spot for the number of workers is between 4 and 16. Each worker increases overall memory consumption for data structures used during a step.

storage_status string

Possible values: [Cleared, InUse, Clearing]

Storage status.

The storage status can only transition when the resources status is Stopped.

Cleared ───┐
▲       │
/clear │       │
│       │
Clearing   │
▲       │
│       │
InUse ◄───┘

udf_rust string

udf_toml string

version int64

Version number.

400

Schema — OPTIONAL
`details` Detailed error metadata. The contents of this field is determined by `error_code`.
`error_code` string Error code is a string that specifies this error type.
`message` string Human-readable error message.

409

Cannot rename pipeline as the new name already exists

Schema — OPTIONAL
`details` Detailed error metadata. The contents of this field is determined by `error_code`.
`error_code` string Error code is a string that specifies this error type.
`message` string Human-readable error message.

500

Schema — OPTIONAL
`details` Detailed error metadata. The contents of this field is determined by `error_code`.
`error_code` string Error code is a string that specifies this error type.
`message` string Human-readable error message.

Desired and actual status​

Desired and actual status​

Desired and actual status

Desired and actual status