Skip to content
FDL
Esc
↑↓navigate↵open⌘Jpreview
On this page

Data files

Rules for shared objects (data files and assets), when a shared object may be deleted, and how a version is taken out of retention.

Shared objects

Data files and assets are shared objects: written once, never overwritten, and reached by any number of versions, as blobs are in Git. A version reaches them this way:

Object Path A version reaches it when
data file <data_dir><schema>/<table>/ducklake-<uuid>.parquet its metadata names it
asset assets/<sha256> a column its table declares in assets holds the sha256 in a row
  • A writer may place shared objects before making a version. Until a version reaches them, readers do not see them
  • A writer that finds an object already there need not upload it again. It may still be deleted before the version is written, so a push whose version reaches a missing object is refused (Writing)

Data files

  • Named as DuckLake names them. Delete files go in the same place
  • Compacting small files (ducklake_merge_adjacent_files) is allowed at any time: it writes new files and deletes none

Assets

  • Named by the lower-case hex sha256 of the content, so the same content is stored once (see Assets)

When a shared object may be deleted

Only when both hold:

  1. No live version reaches it. A live version is one without expired.json
  2. It was placed longer ago than the grace period. The grace period protects objects a writer has placed but not yet made a version of

Only a file whose path has the form in the table above is a shared object: a data file named ducklake-<name>.parquet under the data directory, or assets/ followed by 64 lower-case hex digits. A process that deletes shared objects must leave every other file alone.

A version’s metadata holds one snapshot, so the data files and delete files in tables of its tables.json are all it reaches.

DuckLake’s own cleanup (ducklake_cleanup_old_files, ducklake_expire_snapshots) sees only one metadata file, so it must not be used to delete from a dataset. Deleting is for a process that sees the whole dataset.

Files outside data_dir

A version reaches data files only under data_dir as fdl.json names it now. A file outside it, such as one left in the old place after data_dir changed, is reached by no version, whatever its name. A process that deletes shared objects looks only under data_dir and assets/, so it never deletes such a file. Deleting it is the host’s own work: the host deletes the old place once no reader can still be reading there, by listing it and deleting key by key, after checking again that fdl.json names the new data_dir.

Deleting safely

A deleted shared object cannot be brought back, and every version that reaches it can no longer be read. A process that deletes shared objects follows these rules:

  1. Delete nothing it cannot account for. When it cannot tell what a live version reaches, because its version.json or tables.json is missing or cannot be read or parsed, or the metadata that holds its assets cannot be opened, it deletes none of that kind: no data file when it cannot tell which data files the version reaches, and no asset when it cannot tell which assets. Stopping without deleting anything also meets this. An object kept too long only costs storage; an object deleted too early breaks a version
  2. Read the versions again right before deleting. A writer may make a version while the process decides. Objects a version it did not see when deciding reaches are kept
  3. Delete the objects it decided on, one key at a time. It never deletes by prefix or a whole directory
  4. Implementations of this page agree on the cases in GC cases: for each dataset there, the objects to delete are the same

A process may move an object to a place readers never read from, such as a separate bucket, and delete it there after a set time, so that a mistake can be undone. A writer does not count an object there as present.

Past retention

A version is taken out of retention in this order:

  1. Put versions/<ID>/expired.json
  2. Delete the version’s metadata files, metadata.db and metadata/, and its tables.json
  3. After the grace period, delete the shared objects that meet the conditions above
{ "expired_at": "2026-12-01T00:00:00Z", "reason": "retention", "by": "nightly retention", "message": "Keep the newest 7 versions" }
  • by may say who or what took the version out: a person, or a process such as a scheduled job. With it, the history shows who removed each version’s data, as it shows who made it. It is free text: FDL gives it no meaning, and the tool or host that writes it chooses the value
  • message may say why, in free text, as message of version.json says what changed. reason is the value readers act on; message is for people
  • version.json stays, so the history still says the version was there and what it was, and the graph of parents stays whole
  • A reader that sees expired.json treats the version as past retention and does not open its metadata, whatever the reason
  • A version a ref points at must not be taken out

Choosing the versions to take out

How long to keep versions is the host’s choice: the newest keep on each branch, those made at or after since, or both. A process that takes versions out by such a choice decides for every live version at once, and keeps a version when any of these holds:

  1. A ref points at it
  2. It is a merge base of a ref and refs/heads/main (Writing): merging the ref into main, or main into the ref, reads it
  3. It is among the keep newest on the first-parent line of a branch (refs/heads/<name>), counting from the version the branch points at
  4. It was made at or after since
  5. It was made within the grace period. A writer puts a version before it moves a ref to it

The others are taken out. A dataset without refs/heads/main has nothing taken out. This also takes out versions on no branch’s first-parent line once nothing above keeps them: the parents a merge brought in other than its first, the versions of a deleted branch, and the head of a request the host closed and deleted. A version counts as made at its created_at.

Implementations of this choice agree on the cases in Expire cases.

Withdrawal

Withdrawal removes data that must not be published, such as personal data published by mistake. Unlike retention, it removes data from every version that holds it, live ones included, and does not wait for the grace period.

  1. Make a new version without the data and move every branch that holds an affected version to it. This is a forward move: the new version’s parent is the current one. Its metadata keeps only its own snapshot (The frozen metadata), so it does not reach the data
  2. Delete the tags that point at affected versions
  3. Delete the shared objects that hold the data, at once, such as a data file with personal data or a screenshot showing it: those an affected version reaches and no version kept reaches. Every version that reaches such an object is an affected version. An affected version already past retention has no metadata to say which objects it reached, so when there is one, every shared object no live version reaches is deleted at once too
  4. For every affected version, put expired.json with "reason": "withdrawn" and deleted, and delete its metadata files

The shared objects go before the metadata files, so a withdrawal that stops half way can be run again: the metadata files still say which files hold the data.

{
  "expired_at": "2026-12-01T00:00:00Z",
  "reason": "withdrawn",
  "deleted": {
    "data": ["main/towns/ducklake-019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c.parquet"],
    "assets": ["9f2c…"]
  }
}
  • deleted names the shared objects step 3 deleted: data files by their path under data_dir, assets by their sha256. Names only, so it holds none of the data. Every affected version of one withdrawal carries the same list. A withdrawal run again after stopping half way keeps the names already there, since objects it deleted the first time may no longer be found

  • A host refuses a push that places an object named in deleted of any version, and a process that copies versions between datasets copies none of them. Otherwise an old copy, such as a local clone, brings the data back

  • A copy of the dataset, such as a mirror, applies the withdrawal by the names in deleted: every live version in it that reaches one of them is an affected version, including versions made only in the copy, and the named objects go even where no version reaches them

  • version.json stays, as with retention, so every dataset holds its history back to the first version. If the record itself holds the data (in author, message or annotations), it is rewritten without it before expired.json is put: the value in author or message is replaced with withdrawn, and the entries of annotations that hold it are dropped. id, seq, parents and the other fields stay as they were

  • by and message of expired.json stay readable after the withdrawal, so they must not hold the data either: say why in general terms, such as “personal data published by mistake”

  • Mirrors keep their own copies. A mirror that copies expired.json with "reason": "withdrawn" deletes the metadata files and the shared objects named in deleted. The origin cannot force this; the host tells the mirrors it knows of

Was this page helpful?