Checkpoint Sync to Object Store

Feldera can synchronize pipeline checkpoints to an S3-compatible object store and restore them at startup. This enables disaster recovery scenarios where local storage may be lost, and supports running standby pipelines that can take over from a failed primary within seconds.

Key capabilities:

• Automatic sync — Periodically push local checkpoints to object store.
• Standby pipelines — A secondary pipeline continuously pulls checkpoints and activates on demand.
• Pipeline seeding — Bootstrap a new pipeline from another pipeline's checkpoints via a read-only bucket.

Getting started

Checkpoint sync uses the file storage backend with rclone under the hood to interact with S3-compatible object stores.

Here is a minimal configuration to sync checkpoints to S3:

"storage": {
  "backend": {
    "name": "file",
    "config": {
      "sync": {
        "bucket": "BUCKET_NAME/DIRECTORY_NAME",
        "provider": "AWS",
        "access_key": "ACCESS_KEY",
        "secret_key": "SECRET_KEY",
        "start_from_checkpoint": "latest",
        "push_interval": 120
      }
    }
  }
}

This configuration restores from the latest checkpoint on startup (downloading from S3 if not available locally) and pushes new checkpoints to S3 every 120 seconds.

Bucket ownership

The configured bucket is a read/write checkpoint location owned by one pipeline. This location can be either an entire S3 bucket or a prefix inside a bucket, for example mybucket/checkpoints/pipeline-a. Do not configure two different pipelines with the same bucket location.

On the first checkpoint sync to a location without an ownership file, Feldera writes an owner.json file at the root of the configured bucket location. The file records the system-generated pipeline name that owns the checkpoint location. Before every push, Feldera reads owner.json again and refuses to write checkpoint data if the file names a different pipeline.

If the ownership file is changed manually, the current owner can lose write access to the location and future checkpoint syncs will fail until the configuration or ownership file is corrected. Pulling checkpoints is less strict: if a pipeline pulls from a location owned by another pipeline, Feldera logs a warning but does not fail the pull.

Standby mode

Pipelines can start in standby mode by passing initial=standby to the start pipeline endpoint. A standby pipeline does not process data. Instead, it continuously pulls the latest checkpoint from object store at the interval specified by pull_interval, staying ready for immediate activation.

To activate a standby pipeline:

curl -X POST 'https://{FELDERA_HOST}/v0/pipelines/{PIPELINE_NAME}/activate'

important

A pipeline that was previously activated and still has its storage intact will auto-activate from the latest local checkpoint. This prevents unintentional standby behavior if the pipeline is rescheduled or restarted.

Standby mode is useful for keeping a backup pipeline ready to take over if the primary fails. The following table illustrates a typical failover scenario:

Time	Pipeline A (Primary)	Pipeline B (Standby)
Step 1	Start	Standby Start
Step 2	Processing	Standby
Step 3	Checkpoint 1	Standby
Step 4	Sync 1 to S3	Standby
Step 5	Processing	Pulls Checkpoint 1
Step 6	Checkpoint 2	Standby
Step 7	Sync 2 to S3	Standby
Step 8	Processing	Pulls Checkpoint 2
Step 9	Failed	Standby
Step 10		Activate
Step 11		Running From Checkpoint 2

Seeding from an existing pipeline

read_bucket lets you seed a new pipeline from a read-only checkpoint source without giving it write access to the source.

This is the supported way to share checkpoint data between pipelines. Each pipeline still writes to its own bucket; read_bucket points at another pipeline's checkpoint location and is never modified by the reader.

For example, suppose pipeline A pushes checkpoints to bucket-a/pipeline-a and you want a new pipeline B to start from A's latest state without writing back to A's bucket:

"sync": {
  "bucket": "bucket-b/pipeline-b",
  "read_bucket": "bucket-a/pipeline-a",
  "provider": "AWS",
  "access_key": "ACCESS_KEY",
  "secret_key": "SECRET_KEY",
  "start_from_checkpoint": "latest"
}

On its first start B pulls from bucket-a/pipeline-a (because bucket-b/pipeline-b is empty). All subsequent checkpoints are pushed to bucket-b/pipeline-b. A's bucket is never modified.

read_bucket requires only read permissions on the source bucket (ListBucket and GetObject). See S3 permissions.

Configuration reference

sync configuration fields

Field	Type	Default	Description
endpoint	string		The S3-compatible object store endpoint (e.g., http://localhost:9000 for MinIO).
bucket *	string		The bucket name and optional prefix to store checkpoints (e.g., mybucket/checkpoints). This is the pipeline's read/write checkpoint location and must be unique to this pipeline. See Bucket ownership.
read_bucket	string		A read-only fallback bucket used to seed the pipeline when bucket has no checkpoint. Uses the same connection settings as bucket (provider, access_key, secret_key, endpoint, region). The pipeline never writes to read_bucket. Must point to a different location than bucket. See Seeding from an existing pipeline.
region	string	us-east-1	The region of the bucket. Leave empty for MinIO. If provider is AWS, and no region is specified, us-east-1 is used.
provider *	string		The S3 provider identifier. Must match rclone's list. Case-sensitive. Use "Other" if unsure.
access_key	string		S3 access key. Not required if using environment-based auth (e.g., IRSA).
secret_key	string		S3 secret key. Not required if using environment-based auth.
start_from_checkpoint	string		Checkpoint UUID to resume from, or latest to restore from the latest checkpoint.
fail_if_no_checkpoint	boolean	false	When true, the pipeline fails to start if no checkpoint is found in any source (local storage, bucket, or read_bucket). When false, the pipeline starts from scratch instead.
standby	boolean	false	Deprecated. Use initial=standby when starting the pipeline instead. See Standby mode.
pull_interval	integer(u64)	10	Interval (in seconds) between fetch attempts for the latest checkpoint while in standby.
push_interval	integer(u64)		Interval (in seconds) between automatic syncs of a local checkpoint to object store, measured from the completion of the previous sync. Disabled by default. See Automatic checkpoint synchronization.
transfers	integer (u8)	20	Number of concurrent file transfers.
checkers	integer (u8)	20	Number of parallel checkers for verification.
ignore_checksum	boolean	false	Skip checksum verification after transfer and only check the file size. May improve throughput.
multi_thread_streams	integer (u8)	10	Number of streams for multi-threaded downloads.
multi_thread_cutoff	string	100M	File size threshold to enable multi-threaded downloads (e.g., 100M, 1G). Supported suffixes: k, M, G, T.
upload_concurrency	integer (u8)	10	Number of concurrent chunks to upload during multipart uploads.
flags	array[string]		Extra flags to pass to rclone. Incorrect or conflicting flags may break behavior. See rclone flags and S3 flags.
retention_min_count	integer (u32)	10	The minimum number of checkpoints to retain in object store. No checkpoints will be deleted if the total count is below this threshold.
retention_min_age	integer (u32)	30	The minimum age (in days) a checkpoint must reach before it becomes eligible for deletion. All younger checkpoints will be preserved.

*Fields marked with an asterisk are required.

Checkpoint resolution priority

When start_from_checkpoint is set, Feldera resolves the checkpoint to restore from using the following priority order:

1. Local storage — if a matching checkpoint (or, for latest, any local checkpoint) exists on the local file system, it is used immediately; no download is performed. This makes restarts fast when the pipeline's local state is still intact.
2. bucket — if no matching checkpoint is found locally, Feldera downloads it from the primary read/write bucket.
3. read_bucket — if the checkpoint is also absent from bucket, Feldera falls back to read_bucket (when configured).
4. Start fresh or fail — if no checkpoint is found in any source, behavior is controlled by fail_if_no_checkpoint.

S3 permissions

The following minimum permissions are required on the bucket being written to:

• ListBucket
• DeleteObject
• GetObject
• PutObject
• PutObjectACL

Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::USER_SID:user/USER_NAME"
            },
            "Action": [
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME/*",
                "arn:aws:s3:::BUCKET_NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

For more details, refer to rclone S3 permissions.

These permissions also cover the owner.json ownership file stored at the root of the configured bucket location.

IRSA

To use IRSA (IAM Roles for Service Accounts), omit the access_key and secret_key fields. This loads credentials from the environment.

Automatic checkpoint synchronization

Feldera can automatically synchronize checkpoints to the configured object store at regular intervals by setting the push_interval field in the sync configuration.

When enabled, Feldera periodically pushes the latest local checkpoint to the object store every push_interval seconds after the previous sync operation completes.

Automatic sync is disabled by default.

The status of the most recent automatic sync operation can be queried with:

curl 'http://localhost/v0/pipelines/{PIPELINE_NAME}/checkpoint/sync_status' | jq '.periodic'

It is recommended to set push_interval greater than fault_tolerance.checkpoint_interval_secs to avoid syncing more frequently than checkpoints are created.

Trigger conditions

An automatic checkpoint synchronization is triggered only when all of the following conditions are met:

• The configured push_interval has elapsed.
• No checkpoint sync is currently in progress.
• Checkpoint sync has not been manually requested.
• A valid checkpoint exists.
• The checkpoint has not already been synced.

Triggering a checkpoint sync

A sync operation can be triggered by making a POST request to:

curl -X POST 'http://localhost/v0/pipelines/{PIPELINE_NAME}/checkpoint/sync'

This initiates the sync and returns the UUID of the checkpoint being synced, along with the UUID of the pipeline process that accepted the request:

{
    "checkpoint_uuid": "019779b4-8760-75f2-bdf0-71b825e63610",
    "incarnation_uuid": "019779b4-8760-75f2-bdf0-71b825e63611"
}

Pass incarnation_uuid to sync_status while waiting for the sync, as described under detecting a pipeline restart.

Requesting a sync for a checkpoint that is already syncing joins the sync in progress rather than starting a second one. The response is 202 Accepted either way, so treat it as "a sync for this checkpoint is under way", not as proof that a fresh one started; watch running in sync status to follow it.

Checking sync status

The status of the sync operation can be checked with:

curl 'http://localhost/v0/pipelines/{PIPELINE_NAME}/checkpoint/sync_status'

Response fields

Field	Type	Description
success	uuid | null	UUID of the most recently successful manually triggered checkpoint sync (POST /checkpoint/sync).
failure	object | null	Details of the most recently failed manually triggered checkpoint sync. Contains uuid and error fields.
periodic	uuid | null	UUID of the most recently successful automatic periodic checkpoint sync (configured via push_interval).
running	uuid[]	UUIDs of the checkpoint syncs running right now. Empty when none is in progress.

success and periodic track different sync mechanisms:

• success is updated only by manual syncs triggered via POST /checkpoint/sync.
• periodic is updated only by automatic syncs configured via push_interval.

success, failure and periodic are sticky: each keeps naming the last sync to reach that outcome, long after it finished. running is the field that says what is happening now, so a UUID that appears in both running and failure is a sync in progress that failed on an earlier attempt.

If success and failure would otherwise name the same UUID, then the most recent result is kept and the other is set to null.

running lists only manual syncs. Automatic periodic syncs do not pass through the endpoint that records it, so an in-flight periodic sync leaves running empty; watch periodic for its result instead.

A pipeline older than the release that introduced running omits the field entirely. Absent means "this pipeline cannot report progress", which is not the same as the empty list.

Response examples

No syncs yet:

{ "success": null, "failure": null, "periodic": null, "running": [] }

Manual sync in progress:

{
    "success": null,
    "failure": null,
    "periodic": null,
    "running": ["019779b4-8760-75f2-bdf0-71b825e63610"]
}

Successful manual sync:

{
    "success": "019779b4-8760-75f2-bdf0-71b825e63610",
    "failure": null,
    "periodic": null,
    "running": []
}

Failed manual sync:

{
    "success": null,
    "failure": {
        "uuid": "019779c1-8317-7a71-bd78-7b971f4a3c43",
        "error": "Error pushing checkpoint to object store: ... SignatureDoesNotMatch ..."
    },
    "periodic": null,
    "running": []
}

Automatic periodic sync only (no manual syncs):

{
    "success": null,
    "failure": null,
    "periodic": "019779c1-8317-7a71-bd78-7b971f4a3c43",
    "running": []
}

Both manual and automatic syncs:

{
    "success": "019779b4-8760-75f2-bdf0-71b825e63610",
    "failure": null,
    "periodic": "019779c1-8317-7a71-bd78-7b971f4a3c43",
    "running": []
}

Detecting a pipeline restart

If a pipeline restarts while a sync is in flight (e.g. because its pod was evicted), the request dies with it. The client can detect this by passing the incarnation_uuid returned by POST /checkpoint/sync back to sync_status:

curl 'http://localhost/v0/pipelines/{PIPELINE_NAME}/checkpoint/sync_status?incarnation_uuid=019779b4-8760-75f2-bdf0-71b825e63611'

If the pipeline has restarted since the sync was requested, the incarnation no longer matches and the request fails with HTTP 400 and error code IncarnationUuidMismatch instead of returning an ambiguous status:

{
    "message": "Incarnation UUID mismatch (019779b4-8760-75f2-bdf0-71b825e63611 in request, expected 019779c1-8317-7a71-bd78-7b971f4a3c99)",
    "error_code": "IncarnationUuidMismatch",
    "details": {
        "requested": "019779b4-8760-75f2-bdf0-71b825e63611",
        "expected": "019779c1-8317-7a71-bd78-7b971f4a3c99"
    }
}

Trigger a new sync in response; the old one no longer exists. The parameter is optional, so a client that omits it keeps the previous behavior and cannot tell a restart apart from a sync in progress.

The same parameter and error apply to POST /checkpoint and GET /checkpoint_status, which track local checkpoints the same way.

Pipeline.sync_checkpoint(wait=True) in the Python SDK does this for you: it detects the restart and retries with a fresh sync. Pipeline.checkpoint(wait=True) and fda pipeline checkpoint do the same for local checkpoints.

To poll status yourself instead, use sync_checkpoint_response(), or checkpoint_response() for a local checkpoint. Only these return the incarnation_uuid that the later status check needs; sync_checkpoint() and checkpoint() return just the UUID or sequence number.

Multihost pipelines

In a multihost pipeline, each host pod maintains its own subdirectory inside the shared bucket (host0/, host1/, …). LSM tree reference files are stored at the bucket root and shared across all hosts, since workers on different hosts may reference the same files.

Remote layout

storage-bucket/
├── owner.json                    # Pipeline that owns this checkpoint location
├── w0-[UUID].feldera             # LSM tree reference file (shared across hosts)
├── w1-[UUID].feldera             # LSM tree reference file (shared across hosts)
├── dependencies/                 # Per-checkpoint dependency manifests
│   ├── [UUID-host0-ckpt1].json
│   ├── [UUID-host1-ckpt1].json
│   └── ...
├── host0/
│   ├── checkpoints.feldera      # Checkpoint catalog for host 0
│   ├── [UUID-host0-ckpt1].zip   # Host 0 checkpoint data
│   └── ...
└── host1/
    ├── checkpoints.feldera      # Checkpoint catalog for host 1
    ├── [UUID-host1-ckpt1].zip   # Host 1 checkpoint data
    └── ...

How sync works

The coordinator manages checkpoint sync for multihost pipelines. Before triggering a sync, the coordinator selects the same logical step across all host pods, ensuring their remote catalogs always represent a consistent snapshot. Each pod then syncs its own subdirectory independently.

important

The remote bucket must contain checkpoints written by a pipeline with the same number of hosts. Restoring into a pipeline with a different host count will cause the pipeline to fail.

Limitations

• start_from_checkpoint supports only "latest" for multihost pipelines. Restoring from a specific checkpoint UUID is not supported.
• Garbage collection for multihost pipelines is not yet implemented. Checkpoints accumulate in the bucket until GC support is added.

Buckets with server side encryption

If the bucket has server side encryption enabled, set the flag --s3-server-side-encryption in the flags field.

Example:

"sync": {
  "bucket": "BUCKET_NAME/DIRECTORY_NAME",
  "provider": "AWS",
  "start_from_checkpoint": "latest",
  "flags": ["--s3-server-side-encryption", "aws:kms"]
}

Performance

Sync performance varies based on configuration and environment. The following average speeds were observed in testing:

Storage Type	Avg Upload Speed	Avg Download Speed	Avg Download Speed (Ignore Checksum)
GP3	650 MiB/s	650 MiB/s	850 MiB/s
GP2	125 MiB/s	125 MiB/s	250 MiB/s
NVMe	1.5 GiB/s	2.2 GiB/s	2.3 GiB/s

• GP3 was configured with throughput of 1000 MB/s, and IOPS of 10,000.
• GP2 has a max throughput of 250 MB/s.
• Both GP2 and GP3 can briefly hit maximum download speeds of around 1 GiB/s, but performance drops off quickly.
• NVMe speeds were tested on an i4i.4xlarge instance type.
• Performance may improve by tuning sync parameters such as transfers, checkers, upload_concurrency, etc.