Feldera API
With Feldera, users create data pipelines out of SQL programs. A SQL program comprises tables and views, and includes as well the definition of input and output connectors for each respectively. A connector defines a data source or data sink to feed input data into tables or receive output data computed by the views respectively.
Pipeline
The API is centered around the pipeline, which most importantly consists out of the SQL program, but also has accompanying metadata and configuration parameters (e.g., compilation profile, number of workers, etc.).
- A pipeline is identified and referred to by a user-provided unique name.
- The pipeline program is asynchronously compiled when the pipeline is first created or its program code or configuration is updated.
- Running the pipeline (deployment) is only possible once the program is compiled
- A pipeline cannot be updated while it is running
Concurrency
Both the pipeline and its program have an associated version. A version is a monotonically increasing number. Anytime the core fields (name, description, runtime_config, program_code, program_config) are modified, the pipeline version is incremented. Anytime the program core fields (program_code, program_config) are modified, the program version is incremented. The program version is used internally by the compiler to know when to recompile.