Feldera API
With Feldera, users create data pipelines out of SQL programs. A SQL program comprises tables and views, and includes as well the definition of input and output connectors for each respectively. A connector defines a data source or data sink to feed input data into tables or receive output data computed by the views respectively.
Pipeline
The API is centered around the pipeline, which most importantly consists out of the SQL program, but also has accompanying metadata and configuration parameters (e.g., compilation profile, number of workers, etc.).
- A pipeline is identified and referred to by its user-provided unique name.
- The pipeline program is asynchronously compiled when the pipeline is first created or when its program is subsequently updated.
- Pipeline deployment is only possible once the program is successfully compiled.
- A pipeline cannot be updated while it is deployed.
Concurrency
Each pipeline has a version, which is incremented each time its core fields are updated. The version is monotonically increasing. There is additionally a program version which covers only the program-related core fields, and is used by the compiler to discern when to recompile.