Development workflow
Develop from Devcontainer
A simple way to start interacting with the Feldera source code is via the supplied Devcontainer. It will prepare an environment which will let you build and run all components of Feldera out-of-the-box.
Spin up the Devcontainer, e.g. by opening the Feldera repository in VS Code with Dev Containers extension enabled.
On your first launch you might need to change the ownership of the files:
sudo chown -R user /workspaces/feldera
Build the SQL Compiler:
cd ./sql-to-dbsp-compiler && ./build.sh && cd ..
Build and start the Pipeline Manager (that also serves the Feldera Web Console):
RUST_LOG=info RUST_BACKTRACE=1 cargo run --bin pipeline-manager --features pg-embed -- --api-server-working-directory ~/.feldera -d postgres-embed --dev-mode --bind-address 0.0.0.0 --sql-compiler-home ./sql-to-dbsp-compiler --dbsp-override-path . --compiler-working-directory ~/.feldera --runner-working-directory ~/.feldera
Here,
~/.feldera
is the directory that will host compilation artifacts of SQL Compiler that Pipeline Manager will then use. It will be created if it doesn't exist, and you can use another name for it - just replace corresponding arguments in the above command.
--dbsp-override-path .
should be the path of the Feldera repository root - so update the argument if you are running from a different directory.
You should now be able to access the Web Console at http://localhost:8080/, connected to your local Pipeline Manager instance!
You can also open Web Console in dev mode to be able to see your changes to it live:
cd web-console && bun install && bun run dev
The Web Console in dev mode is available at http://localhost:3000/
Now you can proceed with the demo.
Authenticated mode
In Authenticated mode, you need to login via the Web Console using one of the supported OAuth providers (e.g. AWS Cognito). The Pipeline Manager will require Bearer authorization header for protected requests.
Start the Pipeline Manager in authenticated mode, substituting values from your environment:
AUTH_CLIENT_ID=<client-id> AUTH_ISSUER=<issuer> <see below for additional environment variables> \
cargo run --bin pipeline-manager --features pg-embed -- --auth-provider=aws-cognito
AWS Cognito
First, setup an AWS Cognito user pool, configure an app client and enable the
Amplify-based hosted
UI.
When setting up the hosted UI, setup a redirect URI as
<feldera-api-url>/auth/aws/
(note the trailing slash). For example, when
running Feldera on localhost:8080, the redirect URI should be
http://localhost:8080/auth/aws/
. In this process, you will also set up a
Cognito domain
, which will be something like
https://<domain-name>.auth.us-east-1.amazoncognito.com
. This domain forms the
base of the login and logout
URL
which you will need below.
Additional variables for AWS Cognito:
- AWS_COGNITO_LOGIN_URL: URL to Cognito Hosted UI login, omitting query parameters
redirect_uri
andstate
- AWS_COGNITO_LOGOUT_URL: URL to Cognito Hosted UI logout, omitting query parameters
redirect_uri
andstate
Example:
AUTH_CLIENT_ID=xxxxxxxxxxxxxxxxxxxxxxxxxx AUTH_ISSUER=https://cognito-idp.us-east-1.amazonaws.com/us-east-1_xxxxxxxxx AWS_COGNITO_LOGIN_URL="https://itest-pool.auth.us-east-1.amazoncognito.com/login\?client_id=xxxxxxxxxxxxxxxxxxxxxxxxxx&response_type=token&scope=email+openid" AWS_COGNITO_LOGOUT_URL="https://itest-pool.auth.us-east-1.amazoncognito.com/logout\?client_id=xxxxxxxxxxxxxxxxxxxxxxxxxx&response_type=token&scope=email+openid" RUST_LOG=debug,tokio_postgres=info cargo run --bin=pipeline-manager --features pg-embed -- --dev-mode --auth-provider aws-cognito
Google Identity Platform
Additional variables for Google Identity Platform: none
Example:
AUTH_CLIENT_ID=xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com AUTH_ISSUER="https://accounts.google.com" RUST_LOG=debug,tokio_postgres=info cargo run --bin=pipeline-manager --features pg-embed -- --dev-mode --auth-provider google-identity
Run benchmarks
You can run bundled benchmarks in devcontainer.
NEXMark benchmark
Example running one of nexmark queries:
cd benchmark
./run-nexmark.sh -L sql --query 9 --partitions 1 --events 10M --kafka-broker redpanda:9092