HTTP¶

omniload reads a file addressed by an HTTP or HTTPS URL, using the same reader stack as every other filesystem source. See that page for the supported formats, format and reader hints, and type normalization.

URI format¶

The URI is the URL of the file, exactly as you would open it.

https://example.org/path/to/data.csv
http://example.org/path/to/data.parquet

The query string is part of the address¶

Unlike the other schemes in this family, an HTTP URL’s query string is not connection configuration: it is sent to the server. That is what makes a presigned URL work, since the signature lives in the query and is computed over its encoded form, and an escape that carries meaning (%2F) is preserved rather than decoded. An escape of an unreserved character is normalized (%7E becomes ~), which every canonical signing scheme treats as the same value:

omniload ingest \
    --source-uri   'https://bucket.s3.amazonaws.com/exports/orders.csv?X-Amz-Signature=...&X-Amz-Expires=900' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.orders'

Quote the URI in your shell: & would otherwise background the command.

Note

Because the query is addressing information, connection options cannot ride in it, the way they do for s3:// and the other schemes. They are available when you build the source yourself, where a keyword argument reaches the underlying fsspec HTTP filesystem:

from dlt_filesystem.source.fsspec.http import HttpFilesystemSource

source = HttpFilesystemSource().dlt_source(
    "https://example.org/data.jsonl", "", block_size=65536
)

omniload ingest and run_ingest have no channel for them.

Authentication¶

Credentials in the URL’s userinfo become an HTTP basic-authentication header. Percent-encode any character that a URI cannot carry literally, @ and / included:

https://<USERNAME>:<PASSWORD>@example.org/private/data.csv

Environment proxy settings and .netrc are honoured.

Extended type support¶

Type	Support	Remarks
Formats	✅	See filesystem.
Ranges	✅	Used when the server supports them.
Globs	✅	Requires an HTML directory index.
Last modified	✅	Requires valid `Last-Modified` on each file.

Examples¶

Load a public CSV file into DuckDB¶

omniload ingest \
    --source-uri   'https://example.org/path/to/data.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Name the format for a URL that has no useful extension¶

An API endpoint rarely ends in .csv. Append a #format fragment:

omniload ingest \
    --source-uri   'https://api.example.org/exports/latest#csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Directory indexes and wildcards¶

A server that exposes an HTML directory index can be read with a glob. * selects links in one directory, while ** follows linked subdirectories recursively:

omniload ingest \
    --source-uri   'https://example.org/exports/**/*.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.exports'

The server’s index defines what can be discovered. A wildcard cannot find files that the index does not link.

Incremental file selection¶

Add --filesystem-incremental to append rows only from new or modified files, as described on the filesystem page. Each selected file must return a valid Last-Modified header. A directory index does not carry that metadata, so an incremental glob makes one metadata request per matched file. A missing or malformed header stops extraction and names the query-free file URL. A plain, non-incremental load still works when the header is absent.

How much is transferred¶

Where the server honours byte ranges, the file is read in ranges rather than downloaded whole, so a line-delimited document starts producing rows before all of it has arrived. Two cases are read whole instead, because they cannot be read any other way:

a server that ignores Range and answers with the entire body;
a response with no Content-Length (ordinary Transfer-Encoding: chunked), which leaves the file with no known size and therefore no seekable form.

Parquet is read whole in every case: pyarrow asks for a file’s entire data section in one read, independently of the transport.

Limitations¶

Read only. There is no portable way to write a file over plain HTTP.

See the WebDAV source for an HTTP-based transport with a defined listing protocol.