200
Pipeline successfully updated | Schema — OPTIONAL |
|---|
created_at date-time | deployment_desired_status stringPossible values: [Stopped, Unavailable, Standby, Paused, Running, Suspended] | deployment_desired_status_since date-time | deployment_error object — OPTIONALInformation returned by REST API endpoints on error. details objectDetailed error metadata.
The contents of this field is determined by error_code. | error_code stringError code is a string that specifies this error type. | message stringHuman-readable error message. |
| deployment_id uuid — OPTIONAL | deployment_initial string — OPTIONALPossible values: [Unavailable, Standby, Paused, Running, Suspended] | deployment_resources_desired_status stringPossible values: [Stopped, Provisioned] | deployment_resources_desired_status_since date-time | deployment_resources_status stringPossible values: [Stopped, Provisioning, Provisioned, Stopping] Pipeline resources status. /start (early start failed)
┌───────────────────┐
│ ▼
Stopped ◄────────── Stopping
/start │ ▲
│ │ /stop?force=true
│ │ OR: timeout (from Provisioning)
▼ │ OR: fatal runtime or resource error
⌛Provisioning ────────────│ OR: runtime status is Suspended
│ │
│ │
▼ │
Provisioned ─────────────┘
Desired and actual statusWe use the desired state model to manage the lifecycle of a pipeline. In this model, the
pipeline has two status attributes associated with it: the desired status, which represents
what the user would like the pipeline to do, and the current status, which represents the
actual (last observed) status of the pipeline. The pipeline runner service continuously monitors
the desired status field to decide where to steer the pipeline towards. There are two desired statuses:
Provisioned (set by invoking /start)Stopped (set by invoking /stop?force=true) The user can monitor the current status of the pipeline via the GET /v0/pipelines/{name}
endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the
/start endpoint, and then polls the GET /v0/pipelines/{name} endpoint to monitor the actual
status of the pipeline until its deployment_resources_status attribute changes to
Provisioned indicating that the pipeline has been successfully provisioned, or Stopped with
deployment_error being set. | deployment_resources_status_since date-time | deployment_runtime_desired_status string — OPTIONALPossible values: [Unavailable, Standby, Paused, Running, Suspended] | deployment_runtime_desired_status_since date-time — OPTIONAL | deployment_runtime_status string — OPTIONALPossible values: [Unavailable, Standby, Initializing, AwaitingApproval, Bootstrapping, Replaying, Paused, Running, Suspended] Runtime status of the pipeline. Of the statuses, only Unavailable is determined by the runner. All other statuses are
determined by the pipeline and taken over by the runner. | deployment_runtime_status_since date-time — OPTIONAL | deployment_status stringPossible values: [Stopped, Provisioning, Unavailable, Standby, AwaitingApproval, Initializing, Bootstrapping, Replaying, Paused, Running, Suspended, Stopping] | deployment_status_since date-time | description string | id uuid | name string | platform_version string | program_code string | program_config objectcache boolean — OPTIONALIf true (default), when a prior compilation with the same checksum
already exists, the output of that (i.e., binary) is used.
Set false to always trigger a new compilation, which might take longer
and as well can result in overriding an existing binary. | profile string — OPTIONALPossible values: [dev, unoptimized, optimized, optimized_symbols] Enumeration of possible compilation profiles that can be passed to the Rust compiler
as an argument via cargo build --profile <>. A compilation profile affects among
other things the compilation speed (how long till the program is ready to be run)
and runtime speed (the performance while running). | runtime_version string — OPTIONALOverride runtime version of the pipeline being executed. Warning: This setting is experimental and may change in the future.
Requires the platform to run with the unstable feature runtime_version
enabled. Should only be used for testing purposes, and requires
network access. A runtime version can be specified in the form of a version
or SHA taken from the feldera/feldera repository main branch. Examples: v0.96.0 or f4dcac0989ca0fda7d2eb93602a49d007cb3b0ae A platform of version 0.x.y may be capable of running future and past
runtimes with versions >=0.x.y and <=0.x.y until breaking API changes happen,
the exact bounds for each platform version are unspecified until we reach a
stable version. Compatibility is only guaranteed if platform and runtime version
are exact matches. Note that any enterprise features are currently considered to be part of
the platform. If not set (null), the runtime version will be the same as the platform version. |
| program_error objectLog, warning and error information about the program compilation. rust_compilation object — OPTIONALRust compilation information. exit_code int32Exit code of the cargo compilation command. | stderr stringOutput printed to stderr by the cargo compilation command. | stdout stringOutput printed to stdout by the cargo compilation command. |
| sql_compilation object — OPTIONALSQL compilation information. exit_code int32Exit code of the SQL compiler. | messages object[]Messages (warnings and errors) generated by the SQL compiler. end_column integer | end_line_number integer | error_type string | message string | snippet string — OPTIONAL | start_column integer | start_line_number integer | warning boolean |
|
| system_error string — OPTIONALSystem error that occurred.
- Set
Some(...) upon transition to SystemError
- Set
None upon transition to Pending
|
| program_info object — OPTIONALProgram information is the result of the SQL compilation. input_connectors objectInput connectors derived from the schema. | output_connectors objectOutput connectors derived from the schema. | schema objectA struct containing the tables (inputs) and views for a program. Parse from the JSON data-type of the DDL generated by the SQL compiler. inputs object[]case_sensitive boolean | name string | fields object[]case_sensitive boolean | name string | columntype (circular) | default string — OPTIONAL | lateness string — OPTIONAL | unused boolean | watermark string — OPTIONAL |
| materialized boolean — OPTIONAL | properties object — OPTIONAL |
| outputs object[]case_sensitive boolean | name string | fields object[]case_sensitive boolean | name string | columntype (circular) | default string — OPTIONAL | lateness string — OPTIONAL | unused boolean | watermark string — OPTIONAL |
| materialized boolean — OPTIONAL | properties object — OPTIONAL |
|
| udf_stubs stringGenerated user defined function (UDF) stubs Rust code: stubs.rs |
| program_status stringPossible values: [Pending, CompilingSql, SqlCompiled, CompilingRust, Success, SqlError, RustError, SystemError] Program compilation status. | program_status_since date-time | program_version int64 | refresh_version int64 | runtime_config objectGlobal pipeline configuration settings. This is the publicly
exposed type for users to configure pipelines. checkpoint_during_suspend boolean — OPTIONALDeprecated: setting this true or false does not have an effect anymore. | clock_resolution_usecs int64 — OPTIONALReal-time clock resolution in microseconds. This parameter controls the execution of queries that use the NOW() function. The output of such
queries depends on the real-time clock and can change over time without any external
inputs. If the query uses NOW(), the pipeline will update the clock value and trigger incremental
recomputation at most each clock_resolution_usecs microseconds. If the query does not use
NOW(), then clock value updates are suppressed and the pipeline ignores this setting. It is set to 1 second (1,000,000 microseconds) by default. | cpu_profiler boolean — OPTIONALEnable CPU profiler. The default value is true. | dev_tweaks object — OPTIONALOptional settings for tweaking Feldera internals. The available key-value pairs change from one version of Feldera to
another, so users should not depend on particular settings being
available, or on their behavior. | fault_tolerance object — OPTIONALFault-tolerance configuration. The default [FtConfig] (via [FtConfig::default]) disables fault tolerance,
which is the configuration that one gets if [RuntimeConfig] omits fault
tolerance configuration. The default value for [FtConfig::model] enables fault tolerance, as
Some(FtModel::default()). This is the configuration that one gets if
[RuntimeConfig] includes a fault tolerance configuration but does not
specify a particular model. checkpoint_interval_secs int64 — OPTIONALInterval between automatic checkpoints, in seconds. The default is 60 seconds. Values less than 1 or greater than 3600 will
be forced into that range. | model — OPTIONAL |
| http_workers int64 — OPTIONALSets the number of available runtime threads for the http server. In most cases, this does not need to be set explicitly and
the default is sufficient. Can be increased in case the
pipeline HTTP API operations are a bottleneck. If not specified, the default is set to workers. | init_containers — OPTIONALSpecification of additional (sidecar) containers. | io_workers int64 — OPTIONALSets the number of available runtime threads for async IO tasks. This affects some networking and file I/O operations
especially adapters and ad-hoc queries. In most cases, this does not need to be set explicitly and
the default is sufficient. Can be increased in case
ingress, egress or ad-hoc queries are a bottleneck. If not specified, the default is set to workers. | logging string — OPTIONALLog filtering directives. If set to a valid tracing-subscriber filter, this controls the log
messages emitted by the pipeline process. Otherwise, or if the filter
has invalid syntax, messages at "info" severity and higher are written
to the log and all others are discarded. | max_buffering_delay_usecs int64 — OPTIONALMaximal delay in microseconds to wait for min_batch_size_records to
get buffered by the controller, defaults to 0. | max_parallel_connector_init int64 — OPTIONALThe maximum number of connectors initialized in parallel during pipeline
startup. At startup, the pipeline must initialize all of its input and output connectors.
Depending on the number and types of connectors, this can take a long time.
To accelerate the process, multiple connectors are initialized concurrently.
This option controls the maximum number of connectors that can be initialized
in parallel. The default is 10. | min_batch_size_records int64 — OPTIONALMinimal input batch size. The controller delays pushing input records to the circuit until at
least min_batch_size_records records have been received (total
across all endpoints) or max_buffering_delay_usecs microseconds
have passed since at least one input records has been buffered.
Defaults to 0. | pin_cpus integer[] — OPTIONALOptionally, a list of CPU numbers for CPUs to which the pipeline may pin
its worker threads. Specify at least twice as many CPU numbers as
workers. CPUs are generally numbered starting from 0. The pipeline
might not be able to honor CPU pinning requests. CPU pinning can make pipelines run faster and perform more consistently,
as long as different pipelines running on the same machine are pinned to
different CPUs. | provisioning_timeout_secs int64 — OPTIONALTimeout in seconds for the Provisioning phase of the pipeline.
Setting this value will override the default of the runner. | resources object — OPTIONALcpu_cores_max double — OPTIONALThe maximum number of CPU cores to reserve
for an instance of this pipeline | cpu_cores_min double — OPTIONALThe minimum number of CPU cores to reserve
for an instance of this pipeline | memory_mb_max int64 — OPTIONALThe maximum memory in Megabytes to reserve
for an instance of this pipeline | memory_mb_min int64 — OPTIONALThe minimum memory in Megabytes to reserve
for an instance of this pipeline | namespace string — OPTIONALKubernetes namespace to use for an instance of this pipeline.
The namespace determines the scope of names for resources created
for the pipeline.
If not set, the pipeline will be deployed in the same namespace
as the control-plane. | service_account_name string — OPTIONALKubernetes service account name to use for an instance of this pipeline.
The account determines permissions and access controls. | storage_class string — OPTIONALStorage class to use for an instance of this pipeline.
The class determines storage performance such as IOPS and throughput. | storage_mb_max int64 — OPTIONALThe total storage in Megabytes to reserve
for an instance of this pipeline |
| storage object — OPTIONALStorage configuration for a pipeline. backend — OPTIONALBackend storage configuration. | cache_mib integer — OPTIONALThe maximum size of the in-memory storage cache, in MiB. If set, the specified cache size is spread across all the foreground and
background threads. If unset, each foreground or background thread cache
is limited to 256 MiB. | compression string — OPTIONALPossible values: [default, none, snappy] Storage compression algorithm. | min_step_storage_bytes integer — OPTIONALFor a batch of data passed through the pipeline during a single step,
the minimum estimated number of bytes to write it to storage. This is provided for debugging and fine-tuning and should ordinarily be
left unset. A value of 0 will write even empty batches to storage, and
nonzero values provide a threshold. usize::MAX, the default,
effectively disables storage for such batches. If it is set to another
value, it should ordinarily be greater than or equal to
min_storage_bytes. | min_storage_bytes integer — OPTIONALFor a batch of data maintained as part of a persistent index during a
pipeline run, the minimum estimated number of bytes to write it to
storage. This is provided for debugging and fine-tuning and should ordinarily be
left unset. A value of 0 will write even empty batches to storage, and nonzero
values provide a threshold. usize::MAX would effectively disable
storage for such batches. The default is 1,048,576 (1 MiB). |
| tracing boolean — OPTIONAL | tracing_endpoint_jaeger string — OPTIONALJaeger tracing endpoint to send tracing information to. | workers int32 — OPTIONALNumber of DBSP worker threads. Each DBSP "foreground" worker thread is paired with a "background"
thread for LSM merging, making the total number of threads twice the
specified number. The typical sweet spot for the number of workers is between 4 and 16.
Each worker increases overall memory consumption for data structures
used during a step. |
| storage_status stringPossible values: [Cleared, InUse, Clearing] Storage status. The storage status can only transition when the resources status is Stopped. Cleared ───┐
▲ │
/clear │ │
│ │
Clearing │
▲ │
│ │
InUse ◄───┘
| udf_rust string | udf_toml string | version int64 |
|