Skip to main content

HTTP GET (URL) input connector

Feldera can ingest data from a user-provided URL into a SQL table.

  • The file is fetched using HTTP with the GET method.

The HTTP GET input connector supports fault tolerance. Fault tolerance only makes sense for URLs with static, unchanging data.

HTTP GET input connector configuration​

Use transport name url_input.

PropertyTypeDefaultDescription
path*stringThe URL to fetch, e.g. https://example.com/tools-data.json.
pause_timeoutseconds60How long a paused connector holds its connection to the server open, in seconds. The controller pauses the connector while the pipeline is paused, and whenever the connector reads faster than the pipeline processes. A connector paused for longer than this drops the connection, and reconnects when it starts running again.

[*]: Required fields

Example usage​

We will create a pipeline with an HTTP GET connector.

The file is hosted at https://example.com/tools-data.json, and is in newline-delimited JSON (NDJSON) format with one row per line. For example:

{"insert": {"pid": 0, "name": "hammer", "price": 5}}
{"insert": {"pid": 1, "name": "nail", "price": 0.02}}
{"delete": {"pid": 0}}

SQL example file​

Create a file named program.sql with the following content:

CREATE TABLE price (
    pid BIGINT NOT NULL PRIMARY KEY,
    name VARCHAR NOT NULL,
    price DECIMAL
)
WITH ('connectors' = '[{
    "transport": {
        "name": "url_input",
        "config": {"path": "https://example.com/tools-data.json"}
    },
    "format": {
        "name": "json",
        "config": {
            "update_format": "insert_delete",
            "array": false
        }
    }
}]');

curl​

curl -i -X PUT 'http://127.0.0.1:8080/v0/pipelines/workshop' \
-H 'Content-Type: application/json' \
-d "$(jq -Rsn \
  --rawfile code program.sql \
  '{
    name: "workshop",
    description: "Workshop inventory",
    runtime_config: {
      workers: 4
    },
    program_config: {},
    program_code: $code
  }')"

Python (direct API calls)​

import requests

api_url = "http://127.0.0.1:8080"
headers = { "authorization": f"Bearer <API-KEY>" }
requests.put(
    f"{api_url}/v0/pipelines/workshop",
    headers=headers,
    json={
      "name": "workshop",
      "description": "Workshop inventory",
      "runtime_config": {
        "workers": 4
      },
      "program_config": {},
      "program_code": open("program.sql").read()
    }
).raise_for_status()

Additional resources​

For more information, see: