Skip to main content

Real-time pipelines

A batch job computes all of its results again on each run, so the results are old and the compute cost increases with the data. A Feldera pipeline runs the same SQL incrementally. When the input changes, Feldera updates only the results that the change affects, from the raw tables to the final views.

SourcesLakehouse, Kafka, CDCFeldera pipelinebronze → silver → goldDestinationsLakehouse, Kafka, databaseschangeschanges
Feldera backfills history from the lakehouse, then continues on the change stream.

Why Feldera​

  • The SQL you already run. Feldera evaluates full SQL incrementally, including joins, aggregates, window functions, subqueries and recursion.
  • History and live data in one pipeline. Load a snapshot from Delta Lake or Iceberg, then continue on changes from Kafka, Debezium or PostgreSQL CDC.
  • Correct results. Each view always equals the result of its query on all of the input. With fault tolerance, a pipeline continues from where it stopped after a crash.

Guides and demos​

GuideFrom batch jobs to real-time pipelinesConvert a Spark SQL batch job into a Feldera pipeline, then run it on historical and live data.GuideReal-time medallion architectureKeep bronze, silver and gold layers current in one pipeline. Includes a comparison with Spark and Databricks.GuideTime series analysisLateness, final values, append-only tables and temporal filters for event data.DemoDebezium PostgreSQL sourceReplicate a Postgres table into Feldera through Debezium and Kafka, in JSON and Avro.DemoDebezium MySQL sourceReplicate a MySQL database into Feldera through Debezium and Kafka.DemoJDBC sinkWrite the output of a view to PostgreSQL through the Debezium and Confluent JDBC sinks.