Skip to content
FDL
Esc
↑↓navigate↵open⌘Jpreview
On this page

Inside a repository

The files a repository holds after each step, from init to a tag and a retention run, and the order they are written in.

This page follows one repository through its life and shows which files appear at each step and in what order they are written. The rules behind each step are in the specification; this page shows them at work.

After init

fdl.json
HEAD
{
  "fdl": "0.1",
  "id": "5f0c8a52-6d3e-4f8e-9b1a-2c7d4e9f0a13",
  "data_dir": "data/"
}
ref: refs/heads/main

fdl.json is written with If-None-Match: *, so running init twice cannot replace the id of a repository that already exists. HEAD points at main, which does not exist yet. A reader that follows HEAD now finds no version, and says so.

While writing

After fdl clone and a CREATE TABLE, DuckLake has written Parquet into the repository:

fdl.json
HEAD
tables/main/towns/ducklake-0199f1….parquet

No ref and no version names the file, so no reader can reach it. If the writer gives up, the file is left behind and maintenance deletes it after the grace period.

The first commit

fdl commit does steps 1 to 4 on the writer’s disk, and fdl push does the rest, in this order:

  1. Copy the working metadata from SQLite into a DuckDB file, keeping the snapshots up to the last one
  2. Set the metadata’s data_path to the repository’s data folder, and refuse if any file is recorded with an absolute path elsewhere
  3. Check the declarations against the tables and write tables.json: each table’s data files and columns, each view’s SQL and columns, and the declarations (primary key, asset columns, profile, other datasets the views read)
  4. Write version.json next to them, with a new UUIDv7 as the ID
  5. Check that tables.json lists the data files the metadata names, and that every one is in the repository
  6. Upload versions/<ID>/metadata.db
  7. Upload versions/<ID>/tables.json
  8. Upload versions/<ID>/version.json
  9. Create refs/heads/main with If-None-Match: *
fdl.json
HEAD
refs/heads/main                                   019a2f10-…
versions/019a2f10-…/metadata.db
versions/019a2f10-…/tables.json
versions/019a2f10-…/version.json
tables/main/towns/ducklake-0199f1….parquet

The order is what keeps readers safe. A reader starts from a ref, and the ref is written last. If the push stops after step 6, 7 or 8, the repository holds a version folder no ref points at. Readers never see it, and it changes nothing.

version.json is written after the metadata and tables.json for the same reason: a tool that lists versions/ can treat a folder without version.json as unfinished.

The second commit

Bob checks out main, adds a column and commits:

refs/heads/main                                   019a3c4e-…
versions/019a2f10-…/…
versions/019a3c4e-…/metadata.db
versions/019a3c4e-…/tables.json
versions/019a3c4e-…/version.json
tables/main/towns/ducklake-0199f1….parquet          named by both versions
tables/main/towns/ducklake-019a3b….parquet          named by the second only
{
  "fdl": "0.1",
  "id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
  "seq": 2,
  "parents": ["019a2f10-0c4d-7e88-a1b2-55c0d9e3f7aa"],
  "created_at": "2026-10-15T09:00:00Z",
  "author": { "name": "bob" },
  "message": "Add a note column",
  "metadata": { "path": "metadata.db", "storage": "duckdb-v1.5.0", "bytes": 4194304, "sha256": "9f2c…" },
  "ducklake": { "format": "1.0", "snapshot_id": 12, "snapshots": [8, 12] },
  "tables": { "path": "tables.json", "bytes": 2048, "sha256": "4c1d…" }
}

main is moved with If-Match: <the ETag read at the start of the push>. Had Alice moved it in the meantime, the ETag would not match and the store would refuse the write. Bob’s version folder would stay, unreferenced, and Bob would clone again.

The first Parquet file is not copied. Both metadata files name it at the same path, which is why data files must never be overwritten.

A tag

019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c

A tag is created with If-None-Match: * and never moved. Readers who attach refs/tags/2025-final get the same metadata for as long as the tag exists, whatever happens to main. A version a ref points at is never taken out of retention.

Taking versions out of retention

Metadata files are small, but a dataset that is rebuilt every day keeps every Parquet file any version names. fdl expire --keep 30 takes out of retention the versions behind the newest 30 on each branch, keeping those refs point at and their merge bases with main. Each goes through three steps:

  1. Put versions/<ID>/expired.json
  2. Delete versions/<ID>/metadata.db, tables.json and profiles/
  3. After the grace period, delete data files that no live version names
versions/019a2f10-…/version.json
versions/019a2f10-…/expired.json      {"expired_at": "2026-12-01T00:00:00Z", "reason": "retention"}

version.json stays, so fdl log still lists the version and the graph of parents stays whole. A reader asked for an expired version sees expired.json and stops instead of fetching a metadata file that is gone.

The mark is written first so that a reader never finds a version with no metadata and no explanation. Data files go last, and only after the grace period, because a writer may have placed a file and be about to commit a version that names it.

fdl expire does the first two steps, and fdl gc the third. Deleting data files needs a process that can list the repository and read the tables.json of every live version, since one file can be named by many versions.

Was this page helpful?