Create Pipeline

Required role: write or higher.

Create a new pipeline with the provided configuration.

Request Body — REQUIRED

description string

Human-readable description of the pipeline. Default is an empty string.

tags string[]

Free-form labels used to organize, group, and filter pipelines. Default is no tags (empty vector).

deployment_config object

The part of a deployment config that can change while the pipeline runs.

resources object

CPU and memory, the only fields that can change while the pipeline runs.

cpu_cores_max double

cpu_cores_min double

memory_max string

The maximum memory, such as "16Gi". memory_mb_max, a number of megabytes, is a deprecated alias.

memory_min string

The minimum memory, such as "4Gi". memory_mb_min, a number of megabytes, is a deprecated alias.

name string — REQUIRED

program_code string — REQUIRED

program_config object

Program configuration.

cache boolean

If true (default), when a prior compilation with the same checksum already exists, the output of that (i.e., binary) is used. Set false to always trigger a new compilation, which might take longer and as well can result in overriding an existing binary.

profile string

Possible values: [dev, unoptimized, optimized, optimized_symbols]

Enumeration of possible compilation profiles that can be passed to the Rust compiler as an argument via cargo build --profile <>. A compilation profile affects among other things the compilation speed (how long till the program is ready to be run) and runtime speed (the performance while running).

runtime_version string

Override runtime version of the pipeline being executed.

Warning: This setting is experimental and may change in the future. Requires the platform to run with the unstable feature runtime_version enabled. Should only be used for testing purposes, and requires network access.

A runtime version can be specified in the form of a version or SHA taken from the feldera/feldera repository main branch.

Examples: v0.96.0 or f4dcac0989ca0fda7d2eb93602a49d007cb3b0ae

The value gen2 names an engine rather than a runtime commit: the SQL compiler captures the circuit IR and the runner launches the Gen-2 engine, so no pipeline binary is compiled. gen2:<sha> selects the engine build of that commit, where the runner supports it.

A platform of version 0.x.y may be capable of running future and past runtimes with versions >=0.x.y and <=0.x.y until breaking API changes happen, the exact bounds for each platform version are unspecified until we reach a stable version. Compatibility is only guaranteed if platform and runtime version are exact matches.

Note that any enterprise features are currently considered to be part of the platform.

If not set (null), the runtime version will be the same as the platform version.

use_platform_compiler boolean

Use the platform SQL compiler when a non-platform runtime_version is specified.

Warning: This setting is experimental and may change in the future. Requires the platform to run with the unstable feature runtime_version enabled.

When false (default), the SQL compiler matching the runtime_version is downloaded and used. When true, the platform's SQL compiler is used instead.

Setting this to true avoids downloading the runtime-version-specific SQL compiler JAR (e.g., when network access is unavailable or slow), at the cost of potentially using a mismatched SQL compiler. The Rust runtime sources are still checked out and compiled from the requested runtime_version.

Has no effect when runtime_version is not set or the platform does not have the unstable feature runtime_version enabled.

runtime_config object

Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.

checkpoint_during_suspend boolean

Deprecated: setting this true or false does not have an effect anymore.

clock_resolution string

Real-time clock resolution, for example 1s.

This parameter controls the execution of queries that use the NOW() function. The output of such queries depends on the real-time clock and can change over time without any external inputs. If the query uses NOW(), the pipeline will update the clock value and trigger incremental recomputation at most once per clock_resolution. If the query does not use NOW(), then clock value updates are suppressed and the pipeline ignores this setting.

Unset or null selects the default, DEFAULT_CLOCK_RESOLUTION.

clock_timezone_offset string

Fixed timezone offset for the SQL NOW() clock.

An ISO-8601 UTC offset, for example "+05:30" or "-08:00", that the clock connector adds to every NOW() value it emits, so NOW() returns local time in that fixed timezone instead of UTC.

The offset is baked into the pipeline's checkpointed state and cannot be changed when the pipeline resumes from a checkpoint: the value from the checkpoint stays in effect, and a differing new value is ignored with a warning in the pipeline log.

cpu_profiler boolean

Enable CPU profiler.

The default value is true.

datafusion_memory string

DataFusion memory pool size, such as "1Gi", shared by the ad-hoc query engine and the Delta Lake / Iceberg connectors.

Carved out of max_rss (falling back to resources.memory_max); the remainder goes to the DBSP circuit, so the two do not double-book RAM.

Unset: defaults to 5% of the effective budget, capped at 2 GB. Pipelines that don't run heavy ad-hoc / Delta / Iceberg workloads can leave this unset.

Set this explicitly for ad-hoc queries, or Delta / Iceberg scans, over data too large for the default share. A pool that cannot hold the query surfaces as ResourcesExhausted on that query alone; the pipeline keeps running.

No pool limit applied if no overall budget is configured.

Before release 0.367.0 this setting was datafusion_memory_mb, a number of megabytes. That name is deprecated but still read.

See documentation on the pipeline's memory usage for more details.

dev_tweaks object

Optional settings for tweaking Feldera internals.

These settings reflect experiments that may come and go and change from version to version. Users should not consider them to be stable.

accumulator_merge_threshold_batches int32

Merge threshold of an accumulator's spine: how many batches each level above level 1 waits for before it merges them.

Zero restores the built-in minimum. A spine that garbage-collects old records always uses the built-in minimum.

adaptive_joins boolean

Enable adaptive joins.

Adaptive joins dynamically change their partitioning policy to avoid skew.

Adaptive joins are disabled by default.

balancer_balance_tax double

Factor that discourages the use of the Balance policy in a perfectly balanced collection.

Assuming a perfectly balanced key distribution, the Balance policy is slightly less efficient than Shard, since it requires computing the hash of the entire key/value pair. This factor discourages the use of this policy if the skew is <balancer_balance_tax.

The default value is 1.1.

balancer_key_distribution_refresh_threshold double

The balancer threshold for checking for an improved partitioning policy for a stream.

Finding a good partitioning policy for a circuit involves solving an optimization problem, which can be relatively expensive. Instead of doing this on every step, the balancer only checks for an improved partitioning policy if the key distribution of a stream has changed significantly since the current solution was computed. Specifically, it only kicks in when the size of at least one shard of at least one stream in the cluster has changed by more than this threshold.

The default value is 0.1.

balancer_min_absolute_improvement_threshold int64

The minimum absolute improvement threshold for the balancer.

The join balancer is a component that dynamically chooses an optimal partitioning policy for adaptive join operators. This parameter prevents the join balancer from making changes to the partitioning policy if the improvement is not significant, since the overhead of such rebalancing, especially when performed frequently, can exceed the benefits.

A rebalancing is considered significant if the absolute estimated improvement across the collections whose partitioning policy the rebalancing changes is at least this threshold. The cost model used by the balancer is based on the number of records in the largest partition of a collection.

A rebalancing is applied if both this threshold and balancer_min_relative_improvement_threshold are met.

The default value is 10,000.

balancer_min_relative_improvement_threshold double

The minimum relative improvement threshold for the join balancer.

A rebalancing is considered significant if the relative estimated improvement across the collections whose partitioning policy the rebalancing changes is at least this threshold. Collections that keep their policy are excluded, since they cost the same either way.

A rebalancing is applied if both this threshold and balancer_min_absolute_improvement_threshold are met.

The default value is 1.2.

bloom_false_positive_rate double

False-positive rate for Bloom filters on batches on storage.

Deprecated: use storage.bloom_false_positive_rate instead.

buffer_cache_allocation_strategy string

Possible values: [shared_per_worker_pair, per_thread, global]

Controls how caches are shared across a foreground/background worker pair.

buffer_cache_strategy string

Possible values: [s3_fifo, lru]

Selects which eviction strategy backs a cache instance.

buffer_max_buckets integer

Override the number of buckets/shards used by sharded buffer caches.

This only applies when buffer_cache_strategy = "s3_fifo". Values are rounded up to the next power of two because the current implementation shards by hash(key) & (n - 1).

disable_auto_transaction boolean

Don't automatically start a transaction for every step.

eager_evict boolean

Evict eagerly from buffer caches as files get deleted.

This is an optimization that drops files from the cache as soon as they are deleted.

It has unknown (no?) performance benefits from what I can tell.

Historically it made sense to do this for two reasons: a) we know with 100% guarantee that the file won't ever be read again. b) we could do this in O(logn) time with the LRU cache. This is no longer true for s3-fifo where it is O(n).

If the eviction is expensive, (many small objects in the cache) this can cause a regression.

New default disables this behavior by making it false.

If this doesn't cause regression we will remove this option in the future.

enable_roaring boolean

Whether file-backed batches may use roaring membership filters when the key type supports them.

exchange_channel_capacity string

Maximum size of the queued but unacknowledged exchange messages per pair of remote host and message type.

A sender that pushes past this budget waits for the receiver to acknowledge earlier messages before it queues more. There are three message types, so a host buffers up to three times this many bytes for each of the other hosts, plus any single message that exceeds the budget on its own.

The default is "10M". Before release 0.367.0 this setting was exchange_channel_capacity_bytes, a number of bytes.

fbuf_slab_size_per_class string

Target size of the cache retained in each FBuf slab size class.

The default is "16Mi". Before release 0.367.0 this setting was fbuf_slab_bytes_per_class, a number of bytes.

fetch_distinct boolean

Whether to asynchronously fetch keys needed for the distinct operator from storage. Asynchronous fetching should be faster for high-latency storage, such as object storage, but it could use excessive amounts of memory if the number of keys fetched is very large.

fetch_join boolean

Whether to asynchronously fetch keys needed for the join operator from storage. Asynchronous fetching should be faster for high-latency storage, such as object storage, but it could use excessive amounts of memory if the number of keys fetched is very large.

integral_merge_threshold_batches int32

Merge threshold of an integral's spine: how many batches each level above level 1 waits for before it merges them.

The default is ten. Zero restores the built-in minimum. A spine that garbage-collects old records always uses the built-in minimum.

layer_file_read_ahead_blocks int64

How many data blocks a layer-file cursor that declared a sequential walk reads ahead of its position.

Each block a cursor steps into that the buffer cache does not hold is a device round trip the worker waits out, and a key column is a strided subset of its file that the kernel's readahead never serves. Reading ahead this many blocks keeps that many round trips in flight. Zero disables it. The default is 8.

lazy_input_map_key_block_size string

Minimum size of the key blocks in the batches a lazy input map's accumulator writes for itself. A power of two, at least 4096 bytes.

The map resolves a transaction by walking the accumulated updates in key order without reading values, one storage request per key block, so larger blocks mean fewer requests. Only the accumulator's own batches (merge outputs and spills) take this size; the batches that reach the integral keep it until its merger rewrites them at the default size. The default is "32Ki"; "8Ki" is the file writer's default. Before release 0.367.0 this setting was lazy_input_map_key_block_bytes, a number of bytes.

lazy_input_map_keys_per_step int64

How many keys a lazy input map resolves against its integral in one step.

The lazy upsert operator may scan a large number of keys without producing any output. Therefore bounding its step by the number of outputs is not sufficient. This setting additionally bounds it by its input.

The default is 100,000.

max_level0_batch_size_records int32

Maximum batch size in records for level 0 merges.

merger string

Possible values: [push_merger, list_merger]

Which merger to use.

merger_threads int32

The number of merger threads.

The default value is equal to the number of worker threads.

negative_weight_multiplier int32

Additional bias the merger assigns to records with negative weights (retractions) to promote them to higher levels of the LSM tree sooner.

Reasonable values for this parameter are in the range [0, 10].

The default value is 0, which means that retractions are not given any additional bias.

now_http_driven boolean

Drive NOW() from an external HTTP endpoint instead of wall clock.

When true, the clock connector emits one initial tick (using now_offset if set, otherwise wall clock) and then holds that value. Subsequent calls to POST /clock/advance move NOW() forward by the requested delta. Negative deltas are rejected; the clock is forward-only.

now_offset date-time

Override the timestamp returned by SQL NOW() at pipeline start.

When set, the clock connector anchors NOW() to this RFC 3339 timestamp the first time the pipeline starts and advances at wall-clock cadence from there: NOW() = now_offset + (wall_clock - wall_clock_at_start).

Any RFC 3339 timestamp parseable by chrono::DateTime<Utc> is accepted (years 0001 through 9999), in the past or future relative to wall clock.

This is a testing knob for queries that depend on NOW().

On resume the clock continues from the last journaled NOW(); now_offset's value is honored only on a fresh start:

Initial run	Resume from checkpoint	Post-replay `NOW()`
no offset	no offset	wall clock (unchanged)
offset	offset	wall-clock pace from the last journaled value; the new offset value is ignored
offset	no offset	jumps to wall clock (explicit opt-out of the anchor)
no offset	offset	wall-clock pace from the last journaled value; the new offset value is ignored

optimize_input_during_commit boolean

Optimize input operators during transaction commit.

splitter_chunk_size_records int64

Controls the maximal number of records output by splitter operators (joins, distinct, aggregation, rolling window and group operators) at each step.

The default value is 10,000 records.

stack_overflow_backtrace boolean

Attempt to print a stack trace on stack overflow.

To be used for debugging only; do not enable in production.

storage_max string

If set, the maximum amount of storage for the POSIX backend to allow to be in use before failing all writes with StorageFull. This is useful for testing on top of storage that does not implement its own quota mechanism.

Before release 0.367.0 this setting was storage_mb_max, a number of mebibytes.

streaming_exchange boolean

Enable streaming exchange.

top_level_negative_weight_fraction double

The share of the records at a spine's highest level with negative weights above which the level merges its batches.

A spine keeps most of its records at its highest level. When many of them are retractions, merging the level's batches cancels them against the records they retract, which shrinks the spine's storage.

A fraction from 0 through 1; the default is 0.2. At 1 the level never merges for this reason.

env object

Environment variables for the pipeline process.

These are key-value pairs injected into the pipeline process environment. Some variable names are reserved by the platform and cannot be overridden (for example RUST_LOG, and variables in the FELDERA_, KUBERNETES_, and TOKIO_ namespaces).

fault_tolerance object

Fault-tolerance configuration.

The default [FtConfig] (via [FtConfig::default]) disables fault tolerance, which is the configuration that one gets if [RuntimeConfig] omits fault tolerance configuration.

The default value for [FtConfig::model] enables fault tolerance, as Some(FtModel::default()). This is the configuration that one gets if [RuntimeConfig] includes a fault tolerance configuration but does not specify a particular model.

checkpoint_interval string

Interval between automatic checkpoints, for example 60s.

Unset selects the default, DEFAULT_CHECKPOINT_INTERVAL; null disables periodic checkpointing. Values less than 1 second or greater than 1 hour will be forced into that range.

model

hosts integer

Number of DBSP hosts.

The worker threads are evenly divided among the hosts. For single-host deployments, this should be 1 (the default).

Multihost pipelines are an enterprise-only preview feature.

http_workers int64

Sets the number of available runtime threads for the http server.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case the pipeline HTTP API operations are a bottleneck.

If not specified, the default is set to workers.

init_containers

Specification of additional (sidecar) containers.

io_workers int64

Sets the number of available runtime threads for async IO tasks.

This affects some networking and file I/O operations especially adapters and ad-hoc queries.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case ingress, egress or ad-hoc queries are a bottleneck.

If not specified, the default is set to workers.

logging string

Log filtering directives.

If set to a valid tracing-subscriber filter, this controls the log messages emitted by the pipeline process. Otherwise, or if the filter has invalid syntax, messages at "info" severity and higher are written to the log and all others are discarded.

max_buffering_delay string

Maximal delay to wait for min_batch_size_records to get buffered by the controller, for example 10ms. Defaults to no delay.

max_parallel_connector_init int64

The maximum number of connectors initialized in parallel during pipeline startup.

At startup, the pipeline must initialize all of its input and output connectors. Depending on the number and types of connectors, this can take a long time. To accelerate the process, multiple connectors are initialized concurrently. This option controls the maximum number of connectors that can be initialized in parallel.

The default is 10.

max_rss string

The maximum amount of memory that the pipeline is allowed to use on each host, such as "16G" or "16Gi".

Setting this property activates memory pressure monitoring and backpressure mechanisms. The pipeline will track the amount of remaining memory and report the memory pressure level via the memory_pressure metric.

As the memory pressure increases, the system will apply increasing backpressure to push state cached in memory to storage, preventing the pipeline from running out of memory at the cost of some performance degradation.

It is strongly recommended to set this property to prevent the pipeline from running out of memory. The setting should not exceed the memory limit of the pipeline instance.

When max_rss is not specified but resources.memory_max is set, the latter is used as the effective memory cap for the pipeline.

The pipeline caps this limit at the memory available to it, which is the host's memory or the container's memory limit, whichever is smaller. It rechecks the available memory every second.

Before release 0.367.0 this setting was max_rss_mb, a number of megabytes. That name is deprecated but still read.

See documentation on the pipeline's memory usage for more details.

min_batch_size_records int64

Minimal input batch size.

The controller delays pushing input records to the circuit until at least min_batch_size_records records have been received (total across all endpoints) or max_buffering_delay has passed since the first input record was buffered. Defaults to 0.

pin_cpus integer[]

Optionally, a list of CPU numbers for CPUs to which the pipeline may pin its worker threads. Specify at least twice as many CPU numbers as workers. CPUs are generally numbered starting from 0. The pipeline might not be able to honor CPU pinning requests.

CPU pinning can make pipelines run faster and perform more consistently, as long as different pipelines running on the same machine are pinned to different CPUs.

pipeline_template_configmap object

Configuration for supplying a custom pipeline StatefulSet template via a Kubernetes ConfigMap.

Operators can provide a custom StatefulSet YAML that the Kubernetes runner will use when creating pipeline StatefulSets for a pipeline. The custom template must be stored as the value of a key in a ConfigMap in the same namespace as the pipeline; set name to the ConfigMap name and key to the entry that contains the template.

Recommendations and requirements:

Start from the default template and modify it as needed. The default template is present in ConfigMap named as <release-name>-pipeline-template, with key pipelineTemplate in the release namespace and should be used as a reference.
The template must contain a valid Kubernetes StatefulSet manifest in YAML form. The runner substitutes variables in the template before parsing; therefore the final YAML must be syntactically valid.
The runner performs simple string substitution for the following placeholders. Please ensure these placeholders are placed at appropriate location for their semantics:
{id}: pipeline Kubernetes name (used for object names and labels)
{namespace}: Kubernetes namespace where the pipeline runs
{pipeline_executor_image}: container image used to run the pipeline executor
{binary_ref}: program binary reference passed as an argument
{program_info_ref}: program info reference passed as an argument
{pipeline_storage_path}: mount path for persistent pipeline storage
{storage_class_name}: storage class name to use for PVCs (if applicable)
{deployment_id}: UUID identifying the deployment instance
{deployment_initial}: initial desired runtime status (e.g., provisioning)
{bootstrap_policy}: bootstrap policy value when applicable

key string

Key in the ConfigMap containing the pipeline template.

If not set, defaults to pipelineTemplate.

name string — REQUIRED

Name of the ConfigMap containing the pipeline template.

provisioning_timeout string

Timeout for the Provisioning phase of the pipeline, for example 5m. Setting this value will override the default of the runner.

resources object

autoscaling object

Autoscaling policy for an instance of a pipeline.

storage object

Storage autoscaling policy: expand storage from storage_min toward storage_max as it fills up.

scale_factor double

Expansion multiplier. Defaults to 2.0.

scale_threshold double

Usage fraction that triggers expansion. Defaults to 0.8.

scale_threshold_available string

Expand storage when available space falls below this size, such as "10Gi", even if usage is still under scale_threshold. Either condition triggers expansion. Unset by default.

Before release 0.367.0 this setting was scale_threshold_available_mb, a number of megabytes. That name is deprecated but still read.

cpu_cores_max double

The maximum number of CPU cores to reserve for an instance of this pipeline

cpu_cores_min double

The minimum number of CPU cores to reserve for an instance of this pipeline

memory_max string

The maximum memory to reserve for an instance of this pipeline, such as "16Gi".

Before release 0.367.0 this setting was memory_mb_max, a number of megabytes. That name is deprecated but still read.

memory_min string

The minimum memory to reserve for an instance of this pipeline, such as "4Gi".

Before release 0.367.0 this setting was memory_mb_min, a number of megabytes. That name is deprecated but still read.

namespace string

Kubernetes namespace to use for an instance of this pipeline. The namespace determines the scope of names for resources created for the pipeline. If not set, the pipeline will be deployed in the same namespace as the control-plane.

service_account_name string

Kubernetes service account name to use for an instance of this pipeline. The account determines permissions and access controls.

storage_class string

Storage class to use for an instance of this pipeline. The class determines storage performance such as IOPS and throughput.

storage_max string

The maximum storage for an instance of this pipeline, such as "400Gi".

Without storage_min, this much storage is reserved up front.

With storage_min, storage starts there and autoscaling expands it up to this limit.

Before release 0.367.0 this setting was storage_mb_max, a number of megabytes. That name is deprecated but still read.

storage_min string

The initial storage to reserve for an instance of this pipeline, such as "100Gi". Set together with storage_max to enable storage autoscaling.

Before release 0.367.0 this setting was storage_mb_min, a number of megabytes. That name is deprecated but still read.

volume_attributes_class string

Kubernetes volume attributes class to use for the storage of an instance of this pipeline. The class sets mutable storage parameters such as IOPS and throughput. Requires Kubernetes 1.34 or later (or the VolumeAttributesClass feature gate) and CSI driver support. A change takes effect on the next start, also for existing storage.

storage object

Storage configuration for a pipeline.

backend

Backend storage configuration.

bloom_false_positive_rate double

False-positive rate for Bloom filters on batches on storage, as a fraction f, where 0 < f < 1.

The false-positive rate trades off between the amount of memory used by Bloom filters and how frequently storage needs to be searched for keys that are not actually present. Typical false-positive rates and their corresponding memory costs are:

0.1: 4.8 bits per key
0.01: 9.6 bits per key
0.001: 14.4 bits per key
0.0001: 19.2 bits per key (default)

Values outside the valid range, such as 0.0, disable Bloom filters.

cache_size string

The maximum size of the in-memory storage cache, such as "4Gi".

If set, the specified cache size is spread across all the foreground and background threads. If unset, each foreground or background thread cache is limited to 256 MiB.

Before release 0.367.0 this setting was cache_mib, a number of mebibytes. That name is deprecated but still read.

compression string

Possible values: [default, none, snappy, lz4, zstd]

Storage compression algorithm.

compression_level int32

Compression level, for compression algorithms that have one.

Only StorageCompression::Zstd uses this. Higher levels compress harder and more slowly; decompression speed is largely unaffected, so the tradeoff is write throughput against stored size. Zstd accepts roughly 1 to 22, and a value outside the range its build supports is clamped into it.

This is provided for fine-tuning and should ordinarily be left unset, which selects the algorithm's own default.

min_spill_size string

For a batch of data maintained as part of a persistent index during a pipeline run, the minimum estimated size to write it to storage, such as "10Mi".

This is provided for debugging and fine-tuning and should ordinarily be left unset.

A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. A very large value effectively disables storage for such batches. The default is "10Mi".

Before release 0.367.0 this setting was min_storage_bytes, a number of bytes. That name is deprecated but still read.

min_step_spill_size string

For a batch of data passed through the pipeline during a single step, the minimum estimated size to write it to storage, such as "64Mi".

This is provided for debugging and fine-tuning and should ordinarily be left unset. A value of 0 writes every batch that holds anything to storage; an empty batch stays in memory, since it has nothing to spill and a layer file is not free. Nonzero values provide a threshold. Unset, the default, effectively disables storage for such batches. If it is set, it should ordinarily be greater than or equal to min_spill_size.

Before release 0.367.0 this setting was min_step_storage_bytes, a number of bytes. That name is deprecated but still read.

tracing boolean

Enable pipeline tracing.

tracing_endpoint_jaeger string

Jaeger tracing endpoint to send tracing information to.

workers int32

Number of DBSP worker threads.

Each DBSP "foreground" worker thread is paired with a "background" thread for LSM merging, making the total number of threads twice the specified number.

The typical sweet spot for the number of workers is between 4 and 16. Each worker increases overall memory consumption for data structures used during a step.

udf_rust string

udf_toml string

Responses

201

Pipeline successfully created

Schema — OPTIONAL

description string — OPTIONAL

Human-readable description of the pipeline. Default is an empty string.

tags string[] — OPTIONAL

Free-form labels used to organize, group, and filter pipelines. Default is no tags (empty vector).

created_at date-time

deployment_config object — OPTIONAL

The part of a deployment config that can change while the pipeline runs.

resources object — OPTIONAL

CPU and memory, the only fields that can change while the pipeline runs.

cpu_cores_max double — OPTIONAL

cpu_cores_min double — OPTIONAL

memory_max string — OPTIONAL

The maximum memory, such as "16Gi". memory_mb_max, a number of megabytes, is a deprecated alias.

memory_min string — OPTIONAL

The minimum memory, such as "4Gi". memory_mb_min, a number of megabytes, is a deprecated alias.

deployment_desired_status string

Possible values: [Stopped, Unavailable, Standby, Paused, Running, Suspended]

deployment_desired_status_since date-time

deployment_error object — OPTIONAL

Information returned by REST API endpoints on error.

details

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.

deployment_id uuid — OPTIONAL

deployment_initial string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Paused, Running, Suspended]

deployment_resources_desired_status string

Possible values: [Stopped, Provisioned]

deployment_resources_desired_status_since date-time

deployment_resources_status string

Possible values: [Stopped, Provisioning, Provisioned, Stopping]

Pipeline resources status.

/start (early start or provision check failed)
┌───┐
│   ▼
Stopped ◄────────── Stopping
/start │                   ▲
│                   │ /stop?force=true
│                   │ OR: timeout (from Provisioning)
▼                   │ OR: fatal runtime or resource error
⌛Provisioning ────────────│ OR: runtime status is Suspended
│                   │
│                   │
▼                   │
Provisioned ─────────────┘

Desired and actual status

We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the desired status, which represents what the user would like the pipeline to do, and the current status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards.

There are two desired statuses:

Provisioned (set by invoking /start)
Stopped (set by invoking /stop?force=true)

The user can monitor the current status of the pipeline via the GET /v0/pipelines/{name} endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the /start endpoint, and then polls the GET /v0/pipelines/{name} endpoint to monitor the actual status of the pipeline until its deployment_resources_status attribute changes to Provisioned indicating that the pipeline has been successfully provisioned, or Stopped with deployment_error being set.

deployment_resources_status_details — OPTIONAL

deployment_resources_status_since date-time

deployment_runtime_desired_status string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Paused, Running, Suspended]

deployment_runtime_desired_status_since date-time — OPTIONAL

deployment_runtime_status string — OPTIONAL

Possible values: [Unavailable, Coordination, Standby, Initializing, AwaitingApproval, Bootstrapping, Replaying, Paused, Running, Suspended, ConcurrentBootstrapping, Synchronizing]

Runtime status of the pipeline.

Of the statuses, only Unavailable is determined by the runner. All other statuses are determined by the pipeline and taken over by the runner.

deployment_runtime_status_details object — OPTIONAL

Details about the current runtime status. The fields in this struct should all be optional and set only by a runtime status when they are known. Otherwise, they can just be set None.

approval_diff — OPTIONAL

The diff which is awaiting approval.

Specifically useful for: AwaitingApproval.

connector_stats object — OPTIONAL

Statistics across all connectors.

num_errors int64

Total number of errors across all connectors.

num_transport_errors from all input connectors
num_parse_errors from all input connectors
num_encode_errors from all output connectors
num_transport_errors from all output connectors

reason string — OPTIONAL

Free form text giving an explanation why it is currently in this runtime status.

Specifically useful for: Unavailable, Initializing.

deployment_runtime_status_since date-time — OPTIONAL

deployment_status string

Possible values: [Stopped, Provisioning, Unavailable, Coordination, Standby, AwaitingApproval, Initializing, Bootstrapping, ConcurrentBootstrapping, Synchronizing, Replaying, Paused, Running, Suspended, Stopping]

deployment_status_since date-time

id uuid

Pipeline identifier.

name string

platform_version string

program_code string

program_config object

Program configuration.

cache boolean — OPTIONAL

profile string — OPTIONAL

Possible values: [dev, unoptimized, optimized, optimized_symbols]

runtime_version string — OPTIONAL

Override runtime version of the pipeline being executed.

A runtime version can be specified in the form of a version or SHA taken from the feldera/feldera repository main branch.

Examples: v0.96.0 or f4dcac0989ca0fda7d2eb93602a49d007cb3b0ae

Note that any enterprise features are currently considered to be part of the platform.

If not set (null), the runtime version will be the same as the platform version.

use_platform_compiler boolean — OPTIONAL

Use the platform SQL compiler when a non-platform runtime_version is specified.

Warning: This setting is experimental and may change in the future. Requires the platform to run with the unstable feature runtime_version enabled.

When false (default), the SQL compiler matching the runtime_version is downloaded and used. When true, the platform's SQL compiler is used instead.

Has no effect when runtime_version is not set or the platform does not have the unstable feature runtime_version enabled.

program_error object

Log, warning and error information about the program compilation.

rust_compilation object — OPTIONAL

Rust compilation information.

exit_code int32

Exit code of the cargo compilation command.

messages object[] — OPTIONAL

Parsed rustc diagnostics. Older stored JSON omits this field.

end_column integer

end_line_number integer

error_type string

file string — OPTIONAL

message string

rendered string — OPTIONAL

start_column integer

start_line_number integer

warning boolean

stderr string

Output printed to stderr by the cargo compilation command.

stdout string

Output printed to stdout by the cargo compilation command.

sql_compilation object — OPTIONAL

SQL compilation information.

exit_code int32

Exit code of the SQL compiler.

messages object[]

Messages (warnings and errors) generated by the SQL compiler.

end_column integer

end_line_number integer

error_type string

message string

snippet string — OPTIONAL

start_column integer

start_line_number integer

warning boolean

system_error string — OPTIONAL

System error that occurred.

Set Some(...) upon transition to SystemError
Set None upon transition to Pending

program_info object — OPTIONAL

Program information is the result of the SQL compilation.

input_connectors object

Input connectors derived from the schema.

output_connectors object

Output connectors derived from the schema.

schema object

A struct containing the tables (inputs) and views for a program.

Parse from the JSON data-type of the DDL generated by the SQL compiler.

inputs object[]

case_sensitive boolean

name string

fields object[]

case_sensitive boolean

name string

columntype (circular)

default string — OPTIONAL

lateness string — OPTIONAL

unused boolean

materialized boolean — OPTIONAL

primary_key string[] — OPTIONAL

properties object — OPTIONAL

outputs object[]

case_sensitive boolean

name string

fields object[]

case_sensitive boolean

name string

columntype (circular)

default string — OPTIONAL

lateness string — OPTIONAL

unused boolean

materialized boolean — OPTIONAL

primary_key string[] — OPTIONAL

properties object — OPTIONAL

udf_stubs string

Generated user defined function (UDF) stubs Rust code: stubs.rs

program_status string

Possible values: [Pending, CompilingSql, SqlCompiled, CompilingRust, Success, SqlError, RustError, SystemError]

Program compilation status.

program_status_since date-time

program_version int64

Version number.

refresh_version int64

Version number.

runtime_config object

Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.

checkpoint_during_suspend boolean — OPTIONAL

Deprecated: setting this true or false does not have an effect anymore.

clock_resolution string — OPTIONAL

Real-time clock resolution, for example 1s.

Unset or null selects the default, DEFAULT_CLOCK_RESOLUTION.

clock_timezone_offset string — OPTIONAL

Fixed timezone offset for the SQL NOW() clock.

An ISO-8601 UTC offset, for example "+05:30" or "-08:00", that the clock connector adds to every NOW() value it emits, so NOW() returns local time in that fixed timezone instead of UTC.

cpu_profiler boolean — OPTIONAL

Enable CPU profiler.

The default value is true.

datafusion_memory string — OPTIONAL

DataFusion memory pool size, such as "1Gi", shared by the ad-hoc query engine and the Delta Lake / Iceberg connectors.

Carved out of max_rss (falling back to resources.memory_max); the remainder goes to the DBSP circuit, so the two do not double-book RAM.

Unset: defaults to 5% of the effective budget, capped at 2 GB. Pipelines that don't run heavy ad-hoc / Delta / Iceberg workloads can leave this unset.

No pool limit applied if no overall budget is configured.

Before release 0.367.0 this setting was datafusion_memory_mb, a number of megabytes. That name is deprecated but still read.

See documentation on the pipeline's memory usage for more details.

dev_tweaks object — OPTIONAL

Optional settings for tweaking Feldera internals.

These settings reflect experiments that may come and go and change from version to version. Users should not consider them to be stable.

accumulator_merge_threshold_batches int32 — OPTIONAL

Merge threshold of an accumulator's spine: how many batches each level above level 1 waits for before it merges them.

Zero restores the built-in minimum. A spine that garbage-collects old records always uses the built-in minimum.

adaptive_joins boolean — OPTIONAL

Enable adaptive joins.

Adaptive joins dynamically change their partitioning policy to avoid skew.

Adaptive joins are disabled by default.

balancer_balance_tax double — OPTIONAL

Factor that discourages the use of the Balance policy in a perfectly balanced collection.

The default value is 1.1.

balancer_key_distribution_refresh_threshold double — OPTIONAL

The balancer threshold for checking for an improved partitioning policy for a stream.

The default value is 0.1.

balancer_min_absolute_improvement_threshold int64 — OPTIONAL

The minimum absolute improvement threshold for the balancer.

A rebalancing is applied if both this threshold and balancer_min_relative_improvement_threshold are met.

The default value is 10,000.

balancer_min_relative_improvement_threshold double — OPTIONAL

The minimum relative improvement threshold for the join balancer.

A rebalancing is applied if both this threshold and balancer_min_absolute_improvement_threshold are met.

The default value is 1.2.

bloom_false_positive_rate double — OPTIONAL

False-positive rate for Bloom filters on batches on storage.

Deprecated: use storage.bloom_false_positive_rate instead.

buffer_cache_allocation_strategy string — OPTIONAL

Possible values: [shared_per_worker_pair, per_thread, global]

Controls how caches are shared across a foreground/background worker pair.

buffer_cache_strategy string — OPTIONAL

Possible values: [s3_fifo, lru]

Selects which eviction strategy backs a cache instance.

buffer_max_buckets integer — OPTIONAL

Override the number of buckets/shards used by sharded buffer caches.

This only applies when buffer_cache_strategy = "s3_fifo". Values are rounded up to the next power of two because the current implementation shards by hash(key) & (n - 1).

disable_auto_transaction boolean — OPTIONAL

Don't automatically start a transaction for every step.

eager_evict boolean — OPTIONAL

Evict eagerly from buffer caches as files get deleted.

This is an optimization that drops files from the cache as soon as they are deleted.

It has unknown (no?) performance benefits from what I can tell.

If the eviction is expensive, (many small objects in the cache) this can cause a regression.

New default disables this behavior by making it false.

If this doesn't cause regression we will remove this option in the future.

enable_roaring boolean — OPTIONAL

Whether file-backed batches may use roaring membership filters when the key type supports them.

exchange_channel_capacity string — OPTIONAL

Maximum size of the queued but unacknowledged exchange messages per pair of remote host and message type.

The default is "10M". Before release 0.367.0 this setting was exchange_channel_capacity_bytes, a number of bytes.

fbuf_slab_size_per_class string — OPTIONAL

Target size of the cache retained in each FBuf slab size class.

The default is "16Mi". Before release 0.367.0 this setting was fbuf_slab_bytes_per_class, a number of bytes.

fetch_distinct boolean — OPTIONAL

fetch_join boolean — OPTIONAL

integral_merge_threshold_batches int32 — OPTIONAL

Merge threshold of an integral's spine: how many batches each level above level 1 waits for before it merges them.

The default is ten. Zero restores the built-in minimum. A spine that garbage-collects old records always uses the built-in minimum.

layer_file_read_ahead_blocks int64 — OPTIONAL

How many data blocks a layer-file cursor that declared a sequential walk reads ahead of its position.

lazy_input_map_key_block_size string — OPTIONAL

Minimum size of the key blocks in the batches a lazy input map's accumulator writes for itself. A power of two, at least 4096 bytes.

lazy_input_map_keys_per_step int64 — OPTIONAL

How many keys a lazy input map resolves against its integral in one step.

The default is 100,000.

max_level0_batch_size_records int32 — OPTIONAL

Maximum batch size in records for level 0 merges.

merger string — OPTIONAL

Possible values: [push_merger, list_merger]

Which merger to use.

merger_threads int32 — OPTIONAL

The number of merger threads.

The default value is equal to the number of worker threads.

negative_weight_multiplier int32 — OPTIONAL

Additional bias the merger assigns to records with negative weights (retractions) to promote them to higher levels of the LSM tree sooner.

Reasonable values for this parameter are in the range [0, 10].

The default value is 0, which means that retractions are not given any additional bias.

now_http_driven boolean — OPTIONAL

Drive NOW() from an external HTTP endpoint instead of wall clock.

now_offset date-time — OPTIONAL

Override the timestamp returned by SQL NOW() at pipeline start.

Any RFC 3339 timestamp parseable by chrono::DateTime<Utc> is accepted (years 0001 through 9999), in the past or future relative to wall clock.

This is a testing knob for queries that depend on NOW().

On resume the clock continues from the last journaled NOW(); now_offset's value is honored only on a fresh start:

Initial run	Resume from checkpoint	Post-replay `NOW()`
no offset	no offset	wall clock (unchanged)
offset	offset	wall-clock pace from the last journaled value; the new offset value is ignored
offset	no offset	jumps to wall clock (explicit opt-out of the anchor)
no offset	offset	wall-clock pace from the last journaled value; the new offset value is ignored

optimize_input_during_commit boolean — OPTIONAL

Optimize input operators during transaction commit.

splitter_chunk_size_records int64 — OPTIONAL

Controls the maximal number of records output by splitter operators (joins, distinct, aggregation, rolling window and group operators) at each step.

The default value is 10,000 records.

stack_overflow_backtrace boolean — OPTIONAL

Attempt to print a stack trace on stack overflow.

To be used for debugging only; do not enable in production.

storage_max string — OPTIONAL

Before release 0.367.0 this setting was storage_mb_max, a number of mebibytes.

streaming_exchange boolean — OPTIONAL

Enable streaming exchange.

top_level_negative_weight_fraction double — OPTIONAL

The share of the records at a spine's highest level with negative weights above which the level merges its batches.

A fraction from 0 through 1; the default is 0.2. At 1 the level never merges for this reason.

env object — OPTIONAL

Environment variables for the pipeline process.

fault_tolerance object — OPTIONAL

Fault-tolerance configuration.

The default [FtConfig] (via [FtConfig::default]) disables fault tolerance, which is the configuration that one gets if [RuntimeConfig] omits fault tolerance configuration.

checkpoint_interval string — OPTIONAL

Interval between automatic checkpoints, for example 60s.

Unset selects the default, DEFAULT_CHECKPOINT_INTERVAL; null disables periodic checkpointing. Values less than 1 second or greater than 1 hour will be forced into that range.

model — OPTIONAL

hosts integer — OPTIONAL

Number of DBSP hosts.

The worker threads are evenly divided among the hosts. For single-host deployments, this should be 1 (the default).

Multihost pipelines are an enterprise-only preview feature.

http_workers int64 — OPTIONAL

Sets the number of available runtime threads for the http server.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case the pipeline HTTP API operations are a bottleneck.

If not specified, the default is set to workers.

init_containers — OPTIONAL

Specification of additional (sidecar) containers.

io_workers int64 — OPTIONAL

Sets the number of available runtime threads for async IO tasks.

This affects some networking and file I/O operations especially adapters and ad-hoc queries.

In most cases, this does not need to be set explicitly and the default is sufficient. Can be increased in case ingress, egress or ad-hoc queries are a bottleneck.

If not specified, the default is set to workers.

logging string — OPTIONAL

Log filtering directives.

max_buffering_delay string — OPTIONAL

Maximal delay to wait for min_batch_size_records to get buffered by the controller, for example 10ms. Defaults to no delay.

max_parallel_connector_init int64 — OPTIONAL

The maximum number of connectors initialized in parallel during pipeline startup.

The default is 10.

max_rss string — OPTIONAL

The maximum amount of memory that the pipeline is allowed to use on each host, such as "16G" or "16Gi".

It is strongly recommended to set this property to prevent the pipeline from running out of memory. The setting should not exceed the memory limit of the pipeline instance.

When max_rss is not specified but resources.memory_max is set, the latter is used as the effective memory cap for the pipeline.

The pipeline caps this limit at the memory available to it, which is the host's memory or the container's memory limit, whichever is smaller. It rechecks the available memory every second.

Before release 0.367.0 this setting was max_rss_mb, a number of megabytes. That name is deprecated but still read.

See documentation on the pipeline's memory usage for more details.

min_batch_size_records int64 — OPTIONAL

Minimal input batch size.

pin_cpus integer[] — OPTIONAL

CPU pinning can make pipelines run faster and perform more consistently, as long as different pipelines running on the same machine are pinned to different CPUs.

pipeline_template_configmap object — OPTIONAL

Configuration for supplying a custom pipeline StatefulSet template via a Kubernetes ConfigMap.

Recommendations and requirements:

Start from the default template and modify it as needed. The default template is present in ConfigMap named as <release-name>-pipeline-template, with key pipelineTemplate in the release namespace and should be used as a reference.
The template must contain a valid Kubernetes StatefulSet manifest in YAML form. The runner substitutes variables in the template before parsing; therefore the final YAML must be syntactically valid.
The runner performs simple string substitution for the following placeholders. Please ensure these placeholders are placed at appropriate location for their semantics:
{id}: pipeline Kubernetes name (used for object names and labels)
{namespace}: Kubernetes namespace where the pipeline runs
{pipeline_executor_image}: container image used to run the pipeline executor
{binary_ref}: program binary reference passed as an argument
{program_info_ref}: program info reference passed as an argument
{pipeline_storage_path}: mount path for persistent pipeline storage
{storage_class_name}: storage class name to use for PVCs (if applicable)
{deployment_id}: UUID identifying the deployment instance
{deployment_initial}: initial desired runtime status (e.g., provisioning)
{bootstrap_policy}: bootstrap policy value when applicable

key string — OPTIONAL

Key in the ConfigMap containing the pipeline template.

If not set, defaults to pipelineTemplate.

name string

Name of the ConfigMap containing the pipeline template.

provisioning_timeout string — OPTIONAL

Timeout for the Provisioning phase of the pipeline, for example 5m. Setting this value will override the default of the runner.

resources object — OPTIONAL

autoscaling object — OPTIONAL

Autoscaling policy for an instance of a pipeline.

storage object — OPTIONAL

Storage autoscaling policy: expand storage from storage_min toward storage_max as it fills up.

scale_factor double — OPTIONAL

Expansion multiplier. Defaults to 2.0.

scale_threshold double — OPTIONAL

Usage fraction that triggers expansion. Defaults to 0.8.

scale_threshold_available string — OPTIONAL

Expand storage when available space falls below this size, such as "10Gi", even if usage is still under scale_threshold. Either condition triggers expansion. Unset by default.

Before release 0.367.0 this setting was scale_threshold_available_mb, a number of megabytes. That name is deprecated but still read.

cpu_cores_max double — OPTIONAL

The maximum number of CPU cores to reserve for an instance of this pipeline

cpu_cores_min double — OPTIONAL

The minimum number of CPU cores to reserve for an instance of this pipeline

memory_max string — OPTIONAL

The maximum memory to reserve for an instance of this pipeline, such as "16Gi".

Before release 0.367.0 this setting was memory_mb_max, a number of megabytes. That name is deprecated but still read.

memory_min string — OPTIONAL

The minimum memory to reserve for an instance of this pipeline, such as "4Gi".

Before release 0.367.0 this setting was memory_mb_min, a number of megabytes. That name is deprecated but still read.

namespace string — OPTIONAL

service_account_name string — OPTIONAL

Kubernetes service account name to use for an instance of this pipeline. The account determines permissions and access controls.

storage_class string — OPTIONAL

Storage class to use for an instance of this pipeline. The class determines storage performance such as IOPS and throughput.

storage_max string — OPTIONAL

The maximum storage for an instance of this pipeline, such as "400Gi".

Without storage_min, this much storage is reserved up front.

With storage_min, storage starts there and autoscaling expands it up to this limit.

Before release 0.367.0 this setting was storage_mb_max, a number of megabytes. That name is deprecated but still read.

storage_min string — OPTIONAL

The initial storage to reserve for an instance of this pipeline, such as "100Gi". Set together with storage_max to enable storage autoscaling.

Before release 0.367.0 this setting was storage_mb_min, a number of megabytes. That name is deprecated but still read.

volume_attributes_class string — OPTIONAL

storage object — OPTIONAL

Storage configuration for a pipeline.

backend — OPTIONAL

Backend storage configuration.

bloom_false_positive_rate double — OPTIONAL

False-positive rate for Bloom filters on batches on storage, as a fraction f, where 0 < f < 1.

0.1: 4.8 bits per key
0.01: 9.6 bits per key
0.001: 14.4 bits per key
0.0001: 19.2 bits per key (default)

Values outside the valid range, such as 0.0, disable Bloom filters.

cache_size string — OPTIONAL

The maximum size of the in-memory storage cache, such as "4Gi".

If set, the specified cache size is spread across all the foreground and background threads. If unset, each foreground or background thread cache is limited to 256 MiB.

Before release 0.367.0 this setting was cache_mib, a number of mebibytes. That name is deprecated but still read.

compression string — OPTIONAL

Possible values: [default, none, snappy, lz4, zstd]

Storage compression algorithm.

compression_level int32 — OPTIONAL

Compression level, for compression algorithms that have one.

This is provided for fine-tuning and should ordinarily be left unset, which selects the algorithm's own default.

min_spill_size string — OPTIONAL

For a batch of data maintained as part of a persistent index during a pipeline run, the minimum estimated size to write it to storage, such as "10Mi".

This is provided for debugging and fine-tuning and should ordinarily be left unset.

A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. A very large value effectively disables storage for such batches. The default is "10Mi".

Before release 0.367.0 this setting was min_storage_bytes, a number of bytes. That name is deprecated but still read.

min_step_spill_size string — OPTIONAL

For a batch of data passed through the pipeline during a single step, the minimum estimated size to write it to storage, such as "64Mi".

Before release 0.367.0 this setting was min_step_storage_bytes, a number of bytes. That name is deprecated but still read.

tracing boolean — OPTIONAL

Enable pipeline tracing.

tracing_endpoint_jaeger string — OPTIONAL

Jaeger tracing endpoint to send tracing information to.

workers int32 — OPTIONAL

Number of DBSP worker threads.

Each DBSP "foreground" worker thread is paired with a "background" thread for LSM merging, making the total number of threads twice the specified number.

The typical sweet spot for the number of workers is between 4 and 16. Each worker increases overall memory consumption for data structures used during a step.

storage_status string

Possible values: [Cleared, InUse, Clearing]

Storage status.

The storage status can only transition when the resources status is Stopped.

Cleared ───┐
▲       │
/clear │       │
│       │
Clearing   │
▲       │
│       │
InUse ◄───┘

storage_status_details object — OPTIONAL

Details about pipeline storage, which are returned as part of the regular runtime status polling by the runner.

checkpoints object[]

Present checkpoints.

fingerprint uint64

Fingerprint of the circuit at the time of the checkpoint.

identifier string — OPTIONAL

An optional name for the checkpoint.

processed_records int64 — OPTIONAL

Total number of records processed.

size int64 — OPTIONAL

Total size of the checkpoint files in bytes.

steps int64 — OPTIONAL

Total number of steps made.

uuid uuid

A unique identifier for the given checkpoint.

This is used to identify the checkpoint in the file-system hierarchy.

udf_rust string

udf_toml string

version int64

Version number.

400

Schema — OPTIONAL
`details` Detailed error metadata. The contents of this field is determined by `error_code`.
`error_code` string Error code is a string that specifies this error type.
`message` string Human-readable error message.

409

Cannot create pipeline as the name already exists

Schema — OPTIONAL
`details` Detailed error metadata. The contents of this field is determined by `error_code`.
`error_code` string Error code is a string that specifies this error type.
`message` string Human-readable error message.

500

Schema — OPTIONAL
`details` Detailed error metadata. The contents of this field is determined by `error_code`.
`error_code` string Error code is a string that specifies this error type.
`message` string Human-readable error message.

Desired and actual status​

Desired and actual status