Skip to main content

Push data to a SQL table.

The client sends data encoded using the format specified in the ?format= parameter as a body of the request. The contents of the data must match the SQL table schema specified in table_name

The pipeline ingests data as it arrives without waiting for the end of the request. Successful HTTP response indicates that all data has been ingested successfully.

Path Parameters
pipeline_name string REQUIRED

Unique pipeline name

table_name string REQUIRED

SQL table name. Unquoted SQL names have to be capitalized. Quoted SQL names have to exactly match the case from the SQL program.

Query Parameters
force boolean REQUIRED

When true, push data to the pipeline even if the pipeline is paused. The default value is false

format string REQUIRED

Input data format, e.g., 'csv' or 'json'.

array boolean

Set to true if updates in this stream are packaged into JSON arrays (used in conjunction with format=json). The default values is false.

update_format string

Possible values: [insert_delete, weighted, debezium, snowflake, raw]

Supported JSON data change event formats.

Each element in a JSON-formatted input stream specifies an update to one or more records in an input table. We support several different ways to represent such updates.

InsertDelete

Each element in the input stream consists of an "insert" or "delete" command and a record to be inserted to or deleted from the input table.

{"insert": {"column1": "hello, world!", "column2": 100}}

Weighted

Each element in the input stream consists of a record and a weight which indicates how many times the row appears.

{"weight": 2, "data": {"column1": "hello, world!", "column2": 100}}

Note that the line above would be equivalent to the following input in the InsertDelete format:

{"insert": {"column1": "hello, world!", "column2": 100}}
{"insert": {"column1": "hello, world!", "column2": 100}}

Similarly, negative weights are equivalent to deletions:

{"weight": -1, "data": {"column1": "hello, world!", "column2": 100}}

is equivalent to in the InsertDelete format:

{"delete": {"column1": "hello, world!", "column2": 100}}

Debezium

Debezium CDC format. Refer to Debezium input connector documentation for details.

Snowflake

Uses flat structure so that fields can get parsed directly into SQL columns. Defines three metadata fields:

  • __action - "insert" or "delete"
  • __stream_id - unique 64-bit ID of the output stream (records within a stream are totally ordered)
  • __seq_number - monotonically increasing sequence number relative to the start of the stream.
{"PART":1,"VENDOR":2,"EFFECTIVE_SINCE":"2019-05-21","PRICE":"10000","__action":"insert","__stream_id":4523666124030717756,"__seq_number":1}

Raw

This format is suitable for insert-only streams (no deletions). Each element in the input stream contains a record without any additional envelope that gets inserted in the input table.

JSON data change event format (used in conjunction with format=json). The default value is 'insert_delete'.

Request Body REQUIRED

Contains the new input data in CSV.

string
Responses
200

Data successfully delivered to the pipeline.

400

Error parsing input data.

Schema OPTIONAL
details object

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.

404

Pipeline is not currently running because it has been shutdown or not yet started.

Schema OPTIONAL
details object

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.

500

Request failed.

Schema OPTIONAL
details object

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.