Reading
How a reader resolves a ref, checks a version and attaches its metadata, reads an earlier version, and what may be cached.
Steps
A reader that does not know which ref to ask for may read refs.json to find the names (see Refs).
- Read
HEAD, or the ref asked for. If the content starts withref:, read the ref it names - Read
versions/<ID>/version.jsonfor the ID found. Check that itsidmatches and that the reader knows itsfdlmajor version - If
versions/<ID>/expired.jsonexists, stop: the version is past retention or withdrawn - Open the file
metadata.pathnames inversions/<ID>/(metadata.db), or ametadata/file the reader can open, read-only
ATTACH 'ducklake:https://example.org/lake/versions/019a3c4e-…/metadata.db' AS lake (
DATA_PATH 'https://example.org/lake/tables/', OVERRIDE_DATA_PATH true, READ_ONLY
);
An earlier version
A reader reads an earlier version by the same steps, with that version’s ID: from a tag, from parents, or from a list of versions. Each version’s metadata holds the dataset at its own snapshot, so going back to a version is opening its metadata, not travelling back in time inside another one.
ATTACH 'ducklake:https://example.org/lake/versions/019a2f10-…/metadata.db' AS lake_v11 (
DATA_PATH 'https://example.org/lake/tables/', OVERRIDE_DATA_PATH true, READ_ONLY
);
- A reader must not read an earlier version by
AT (VERSION => n)orducklake_snapshots()in the metadata of a later version. A metadata file holds only its own snapshot (The frozen metadata). - To compare two versions, a reader attaches both metadata files under different names
- A version past retention or withdrawn has no metadata to open (step 3)
Views that read other datasets
A view’s SQL may read tables of other datasets, under the names dependencies of tables.json gives. Before it reads such a view, a reader attaches each entry under its name, by the steps above: the version version names, or the version main holds when it has none.
ATTACH 'ducklake:https://example.org/census/versions/019a2f10-…/metadata.db' AS census (
DATA_PATH 'https://example.org/census/tables/', OVERRIDE_DATA_PATH true, READ_ONLY
);
A reader that does not read the view need not attach them.
Signatures
A version may carry versions/<ID>/version.json.sig: an SSH signature of the bytes of version.json, made with ssh-keygen -Y sign -n fdl. The signature covers the record, and the record covers the metadata through metadata.sha256.
ssh-keygen -Y verify -f allowed_signers -I alice@example.org -n fdl \
-s version.json.sig < version.json
- Signing is optional. A reader that does not check signatures reads a signed version as any other
- A reader that checks verifies the signature against keys it trusts, gets the metadata and checks its sha256 against
metadata.sha256 - Keys come from the origin, not from the dataset being read.
signersinfdl.jsononly says which keys to expect
Data path
The metadata’s data_path is the absolute URL of the data folder where the writer published. A reader must pass DATA_PATH '<root><data_dir>' and OVERRIDE_DATA_PATH true, where <root> is wherever it reads the dataset from. The statement is then the same at the origin and at a mirror, and a dataset copied to another host is read without changing its metadata files.
Checking the metadata
A metadata file is written by whoever can write the dataset, and DuckDB reads every file it names with the reader’s credentials. Before a reader attaches a metadata file with credentials that reach more than the dataset, it should check that:
- the metadata’s sha256 is
metadata.sha256inversion.json - every data file it names is recorded relative to the data path, with no empty,
.or..segment
A metadata file that names a file by an absolute path, or outside data_dir, makes the reader fetch whatever it points at.
Over HTTP, a reader sends its token only to the origin of the dataset, and only over https unless the host is the reader’s own machine. A redirect to another origin goes without the token.
Caching
- A version’s files never change except as described in Versions, so its metadata may be cached forever
HEAD,refs/andrefs.jsonmust not be cachedexpired.jsonmay appear later. A cache of a version folder should be dropped when it does
Over HTTP, HEAD and refs/ are served as Content-Type: text/plain; charset=utf-8 with Cache-Control: no-store.
A metadata file at a fixed URL
A host may serve the metadata of the version HEAD holds at <root>/metadata.db, for readers that do not know FDL. It must then send the version ID in an FDL-Version response header. This metadata is served, not stored: the store holds no metadata outside versions/. A host that served it at <root>/ducklake.duckdb before may keep serving it there too.