(http)=

# HTTP

`omniload` reads a file addressed by an [HTTP] or HTTPS URL, using the same reader
stack as every other {ref}`filesystem <filesystem>` source. See that page for the
supported formats, format and reader hints, and type normalization.

## URI format

The URI is the URL of the file, exactly as you would open it.

```text
https://example.org/path/to/data.csv
http://example.org/path/to/data.parquet
```

## The query string is part of the address

Unlike the other schemes in this family, an HTTP URL's query string is **not**
connection configuration: it is sent to the server. That is what makes a presigned
URL work, since the signature lives in the query and is computed over its encoded
form, and an escape that carries meaning (`%2F`) is preserved rather than decoded.
An escape of an unreserved character is normalized (`%7E` becomes `~`), which every
canonical signing scheme treats as the same value:

```sh
omniload ingest \
    --source-uri   'https://bucket.s3.amazonaws.com/exports/orders.csv?X-Amz-Signature=...&X-Amz-Expires=900' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.orders'
```

Quote the URI in your shell: `&` would otherwise background the command.

:::{note}
Because the query is addressing information, connection options cannot ride in it,
the way they do for `s3://` and the other schemes. They are available when you
build the source yourself, where a keyword argument reaches the underlying [fsspec
HTTP filesystem]:

```python
from dlt_filesystem.source.fsspec.http import HttpFilesystemSource

source = HttpFilesystemSource().dlt_source(
    "https://example.org/data.jsonl", "", block_size=65536
)
```

`omniload ingest` and `run_ingest` have no channel for them.
:::

## Authentication

Credentials in the URL's userinfo become an HTTP basic-authentication header.
Percent-encode any character that a URI cannot carry literally, `@` and `/`
included:

```text
https://<USERNAME>:<PASSWORD>@example.org/private/data.csv
```

Environment proxy settings and `.netrc` are honoured.

## Extended type support

| Type          | Support | Remarks                                            |
|:--------------|:--------|:---------------------------------------------------|
| Formats       | ✅      | See {ref}`filesystem <filesystem>`.                |
| Ranges        | ✅      | Used when the server supports them.                |
| Globs         | ✅      | Requires an HTML directory index.                  |
| Last modified | ✅      | Requires valid `Last-Modified` on each file.       |

## Examples

### Load a public CSV file into DuckDB

```sh
omniload ingest \
    --source-uri   'https://example.org/path/to/data.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'
```

### Name the format for a URL that has no useful extension

An API endpoint rarely ends in `.csv`. Append a `#format` fragment:

```sh
omniload ingest \
    --source-uri   'https://api.example.org/exports/latest#csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'
```

## Directory indexes and wildcards

A server that exposes an HTML directory index can be read with a glob.
`*` selects links in one directory, while `**` follows linked subdirectories
recursively:

```sh
omniload ingest \
    --source-uri   'https://example.org/exports/**/*.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.exports'
```

The server's index defines what can be discovered. A wildcard cannot find files
that the index does not link.

## Incremental file selection

Add `--filesystem-incremental` to append rows only from new or modified files, as
described on the {ref}`filesystem <filesystem>` page. Each selected file must
return a valid `Last-Modified` header. A directory index does not carry that
metadata, so an incremental glob makes one metadata request per matched file. A
missing or malformed header stops extraction and names the query-free file URL.
A plain, non-incremental load still works when the header is absent.

## How much is transferred

Where the server honours byte ranges, the file is read in ranges rather than
downloaded whole, so a line-delimited document starts producing rows before all of
it has arrived. Two cases are read whole instead, because they cannot be read any
other way:

- a server that ignores `Range` and answers with the entire body;
- a response with no `Content-Length` (ordinary `Transfer-Encoding: chunked`),
  which leaves the file with no known size and therefore no seekable form.

Parquet is read whole in every case: pyarrow asks for a file's entire data section
in one read, independently of the transport.

## Limitations

- **Read only.** There is no portable way to write a file over plain HTTP.

See the {ref}`WebDAV <webdav>` source for an HTTP-based transport with a defined
listing protocol.

[fsspec HTTP filesystem]: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.http.HTTPFileSystem
[HTTP]: https://developer.mozilla.org/en-US/docs/Web/HTTP