---
title: Inside a repository
description: The files a repository holds after each step, from init to a tag and a retention run, and the order they are written in.
---

This page follows one repository through its life and shows which files appear at each step and in what order they are written. The rules behind each step are in the [specification](/spec); this page shows them at work.

## After init

```
fdl.json
HEAD
```

```json fdl.json
{
  "fdl": "0.1",
  "id": "5f0c8a52-6d3e-4f8e-9b1a-2c7d4e9f0a13",
  "data_dir": "data/"
}
```

```text HEAD
ref: refs/heads/main
```

`fdl.json` is written with `If-None-Match: *`, so running `init` twice cannot replace the `id` of a repository that already exists. `HEAD` points at `main`, which does not exist yet. A reader that follows `HEAD` now finds no version, and says so.

## While writing

After `fdl clone` and a `CREATE TABLE`, DuckLake has written Parquet into the repository:

```
fdl.json
HEAD
tables/main/towns/ducklake-0199f1….parquet
```

No ref and no version names the file, so no reader can reach it. If the writer gives up, the file is left behind and maintenance deletes it after the grace period.

## The first commit

`fdl commit` does steps 1 to 4 on the writer's disk, and `fdl push` does the rest, in this order:

1. Copy the working metadata from SQLite into a DuckDB file, keeping the snapshots up to the last one
2. Set the metadata's `data_path` to the repository's data folder, and refuse if any file is recorded with an absolute path elsewhere
3. Check the declarations against the tables and write `tables.json`: each table's data files and columns, each view's SQL and columns, and the declarations (primary key, asset columns, profile, other datasets the views read)
4. Write `version.json` next to them, with a new UUIDv7 as the ID
5. Check that `tables.json` lists the data files the metadata names, and that every one is in the repository
6. Upload `versions/<ID>/metadata.db`
7. Upload `versions/<ID>/tables.json`
8. Upload `versions/<ID>/version.json`
9. Create `refs/heads/main` with `If-None-Match: *`

```
fdl.json
HEAD
refs/heads/main                                   019a2f10-…
versions/019a2f10-…/metadata.db
versions/019a2f10-…/tables.json
versions/019a2f10-…/version.json
tables/main/towns/ducklake-0199f1….parquet
```

The order is what keeps readers safe. A reader starts from a ref, and the ref is written last. If the push stops after step 6, 7 or 8, the repository holds a version folder no ref points at. Readers never see it, and it changes nothing.

`version.json` is written after the metadata and `tables.json` for the same reason: a tool that lists `versions/` can treat a folder without `version.json` as unfinished.

## The second commit

Bob checks out `main`, adds a column and commits:

```
refs/heads/main                                   019a3c4e-…
versions/019a2f10-…/…
versions/019a3c4e-…/metadata.db
versions/019a3c4e-…/tables.json
versions/019a3c4e-…/version.json
tables/main/towns/ducklake-0199f1….parquet          named by both versions
tables/main/towns/ducklake-019a3b….parquet          named by the second only
```

```json versions/019a3c4e-…/version.json
{
  "fdl": "0.1",
  "id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
  "seq": 2,
  "parents": ["019a2f10-0c4d-7e88-a1b2-55c0d9e3f7aa"],
  "created_at": "2026-10-15T09:00:00Z",
  "author": { "name": "bob" },
  "message": "Add a note column",
  "metadata": { "path": "metadata.db", "storage": "duckdb-v1.5.0", "bytes": 4194304, "sha256": "9f2c…" },
  "ducklake": { "format": "1.0", "snapshot_id": 12, "snapshots": [8, 12] },
  "tables": { "path": "tables.json", "bytes": 2048, "sha256": "4c1d…" }
}
```

`main` is moved with `If-Match: <the ETag read at the start of the push>`. Had Alice moved it in the meantime, the ETag would not match and the store would refuse the write. Bob's version folder would stay, unreferenced, and Bob would clone again.

The first Parquet file is not copied. Both metadata files name it at the same path, which is why data files must never be overwritten.

## A tag

```text refs/tags/2025-final
019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c
```

A tag is created with `If-None-Match: *` and never moved. Readers who attach `refs/tags/2025-final` get the same metadata for as long as the tag exists, whatever happens to `main`. A version a ref points at is never taken out of retention.

## Taking versions out of retention

Metadata files are small, but a dataset that is rebuilt every day keeps every Parquet file any version names. `fdl expire --keep 30` takes out of retention the versions behind the newest 30 on each branch, keeping those refs point at and their merge bases with main. Each goes through three steps:

1. Put `versions/<ID>/expired.json`
2. Delete `versions/<ID>/metadata.db`, `tables.json` and `profiles/`
3. After the grace period, delete data files that no live version names

```
versions/019a2f10-…/version.json
versions/019a2f10-…/expired.json      {"expired_at": "2026-12-01T00:00:00Z", "reason": "retention"}
```

`version.json` stays, so `fdl log` still lists the version and the graph of parents stays whole. A reader asked for an expired version sees `expired.json` and stops instead of fetching a metadata file that is gone.

The mark is written first so that a reader never finds a version with no metadata and no explanation. Data files go last, and only after the grace period, because a writer may have placed a file and be about to commit a version that names it.

`fdl expire` does the first two steps, and `fdl gc` the third. Deleting data files needs a process that can list the repository and read the `tables.json` of every live version, since one file can be named by many versions.
