S2 output connector

Experimental feature

S2 support is an experimental feature of Feldera.

Feldera can append encoded SQL view updates to an S2 stream with the s2_output connector.

Configuration

Property	Type	Required	Description
basin	string	Yes	S2 basin name.
stream	string	Yes	S2 stream name.
auth_token	string	Yes	S2 authentication token. Use a secret reference in production.
endpoint	string	No	Custom S2 endpoint, primarily for local development. When omitted, the S2 cloud endpoint is used.

The connector splits each encoded output buffer into one S2 record per line and appends them as native S2 batches, chunked to S2's per-batch record limit. S2 enforces a 1 MiB metered limit per AppendRecordBatch (including per-record framing overhead); this is a hard, server-side limit, not a Feldera setting. The connector asks the encoder to keep each buffer below that limit, but a single oversized record — for example, a row with a very long string column — can still exceed it and will be rejected by the append call. The connector waits for S2 to acknowledge each batch before returning. It is not a fault-tolerant output connector; a failure after S2 accepted a batch but before Feldera observed its acknowledgement can produce duplicates after a retry.

Example

CREATE TABLE orders (
    id BIGINT,
    amount DECIMAL(12, 2)
);

CREATE VIEW large_orders WITH (
    'connectors' = '[{
        "transport": {
            "name": "s2_output",
            "config": {
                "basin": "commerce",
                "stream": "large-orders",
                "auth_token": "${secret:kubernetes:s2/auth-token}"
            }
        },
        "format": {
            "name": "json",
            "config": {
                "update_format": "insert_delete"
            }
        }
    }]'
) AS
SELECT * FROM orders WHERE amount >= 1000;

S2 is not a key-value store, so output formats or indexes that require keyed writes are not supported by this connector.