Parquet Format

Feldera can ingest and output data in the Parquet format.

• via ingress and egress REST endpoints by specifying ?format=parquet in the URL
• as a payload received from or sent to a connector

We document the Parquet format and how it interacts with different SQL types in this page.

Types

The parquet file is expected to be a valid parquet file with a schema. The schema (row name and type) must match the table definition in the Feldera pipeline program. We use Arrow to specify the data-types in parquet. The following table shows the mapping between Feldera SQL types and Arrow types.

Feldera SQL Type	Apache Arrow Type
BOOLEAN	Boolean
TINYINT, SMALLINT, INTEGER, BIGINT	Int8, Int16, Int32, Int64
TINYINT UNSIGNED, SMALLINT UNSIGNED, INTEGER UNSIGNED, BIGINT UNSIGNED	UInt8, UInt16, UInt32, UInt64
REAL, DOUBLE	Float32, Float64
DECIMAL(P,S)	Decimal128(P,S)
VARCHAR, CHAR, STRING	Utf8
BINARY, VARBINARY	LargeBinary
UUID	Utf8
TIME	Time64(Nanosecond)
TIMESTAMP	Timestamp(Microsecond, None)
TIMESTAMP WITH TIME ZONE	Timestamp(Microsecond, "UTC")
DATE	Date32 (days since unix epoch)
INTERVAL (year, month)	Interval(YearMonth)
INTERVAL (all other units)	Interval(DayTime)
ARRAY	LargeList
ROW and user-defined types	Struct
MAP	Map
VARIANT	Utf8 (JSON-encoded string, see VARIANT documentation)

Output configuration

• buffer_size_records: Maximum number of records to batch into a single output Parquet file (output connectors only). When this threshold is reached, the encoder flushes the buffer to the transport. The default is 100000.

Example

In this example, we configure a table to load data from a Parquet file.

CREATE TABLE parts (
  part BIGINT NOT NULL,
  vendor BIGINT NOT NULL,
  price BIGINT NOT NULL
) WITH ('connectors' = '[{
  "transport": {
    "name": "url_input",
    "config": { "path": "https://feldera-basics-tutorial.s3.amazonaws.com/parts.parquet" }
  },
  "format": {
    "name": "parquet",
    "config": {}
  }
}]');

For reference, the following python script was used to generate the parts.parquet file:

import pyarrow as pa
import pyarrow.parquet as pq

data = {
    'PART': [1, 2, 3],
    'VENDOR': [2, 1, 3],
    'PRICE': [10000, 15000, 9000]
}
table = pa.Table.from_pydict(data)
pq.write_table(table, 'parts.parquet')