Skip to main content
Version: 0.169.0

Feldera API

With Feldera, users create data pipelines out of SQL programs. A SQL program comprises tables and views, and includes as well the definition of input and output connectors for each respectively. A connector defines a data source or data sink to feed input data into tables or receive output data computed by the views respectively.

Pipeline​

The API is centered around the pipeline, which most importantly consists out of the SQL program, but also has accompanying metadata and configuration parameters (e.g., compilation profile, number of workers, etc.).

  • A pipeline is identified and referred to by its user-provided unique name.
  • The pipeline program is asynchronously compiled when the pipeline is first created or when its program is subsequently updated.
  • Pipeline deployment start is only able to proceed to provisioning once the program is successfully compiled.
  • A pipeline cannot be updated while it is deployed.

Concurrency​

Each pipeline has a version, which is incremented each time its core fields are updated. The version is monotonically increasing. There is additionally a program version which covers only the program-related core fields, and is used by the compiler to discern when to recompile.

Client request handling​

Request outcome expectations​

The outcome of a request is that it either fails (e.g., DNS lookup failed) without any response (no status code nor body), or it succeeds and gets back a response status code and body.

In case of a response, usually it is the Feldera endpoint that generated it:

  • If it is success (2xx), it will return whichever body belongs to the success response.
  • Otherwise, if it is an error (4xx, 5xx), it will return a Feldera error response JSON body which will have an application-level error_code.

However, there are two notable exceptions when the response is not generated by the Feldera endpoint:

  • If the HTTP server, to which the endpoint belongs, encountered an issue, it might return 4xx (e.g., for an unknown endpoint) or 5xx error codes by itself (e.g., when it is initializing).
  • If the Feldera API server is behind a (reverse) proxy, the proxy can return error codes by itself, for example BAD GATEWAY (502) or GATEWAY TIMEOUT (504).

As such, it is not guaranteed that the (4xx, 5xx) will have a Feldera error response JSON body in these latter cases.

Error handling and retrying​

The error type returned by the client should distinguish between the error responses generated by Feldera endpoints themselves (which have a Feldera error response body) and those that are generated by other sources.

In order for a client operation (e.g., pipeline.resume()) to be robust (i.e., not fail due to a single HTTP request not succeeding) the client should use a retry mechanism if the operation is idempotent. The retry mechanism must however have a time limit, after which it times out. This guarantees that the client operation is eventually responsive, which enables the script it is a part of to not hang indefinitely on Feldera operations and instead be able to decide by itself whether and how to proceed. If no response is returned, the mechanism should generally retry. When a response is returned, the decision whether to retry can generally depend on the status code: especially the status codes 502, 503 and 504 should be considered as transient errors. Finer grained retry decisions should be made by taking into account the application-level error_code if the response body was indeed a Feldera error response body.

Feldera client errors (4xx)​

Client behavior: clients should generally return with an error when they get back a 4xx status code, as it usually means the request will likely not succeed even if it is sent again. Certain requests might make use of a timed retry mechanism when the client error is transient without requiring any user intervention to overcome, for instance a transaction already being in progress leading to a temporary CONFLICT (409) error.

  • BAD REQUEST (400): invalid user request (general).

    • Example: the new pipeline name example1@~ contains invalid characters.

  • UNAUTHORIZED (401): the user is not authorized to issue the request.

    • Example: an invalid API key is provided.

  • NOT FOUND (404): a resource required to exist in order to process the request was not found.

    • Example: a pipeline named example does not exist when trying to update it.

  • CONFLICT (409): there is a conflict between the request and a relevant resource.

    • Example: a pipeline named example already exists.
    • Example: another transaction is already in process.

Feldera server errors (5xx)​

  • INTERNAL SERVER ERROR (500): the server is unexpectedly unable to process the request (general).

    • Example: unable to reach the database.
    • Client behavior: immediately return with an error.

  • NOT IMPLEMENTED (501): the server does not implement functionality required to process the request.

    • Example: making a request to an enterprise-only endpoint in the OSS edition.
    • Client behavior: immediately return with an error.

  • SERVICE UNAVAILABLE (503): the server is not (yet) able to process the request.

    • Example: pausing a pipeline which is still provisioning.
    • Client behavior: depending on the type of request, client may use a timed retry mechanism.

Feldera error response body​

When the Feldera API returns an HTTP error status code (4xx, 5xx), the body will contain the following JSON object:

{
  "message": "Human-readable explanation.",
  "error_code": "CodeSpecifyingError",
  "details": {

  }
}

It contains the following fields:

  • message (string): human-readable explanation of the error that occurred and potentially hinting what can be done about it.
  • error_code (string): application-level code about the error that occurred, written in CamelCase. For example: UnknownPipelineName, DuplicateName, PauseWhileNotProvisioned, ... .
  • details (object): JSON object corresponding to the error_code with fields that provide details relevant to it. For example: if a name is unknown, a field with the unknown name in question.