Skip to content
FDL
Esc
↑↓navigate↵open⌘Jpreview
On this page

Specification

The normative description of FDL 0.1 (draft) - what it decides, what it leaves to hosts, and the terms it uses.

This is the specification of FDL 0.1, a draft. Tools are written against this specification, and where a tool and this text disagree, the text is changed first and the tool follows.

FDL is a storage layout for keeping versions of DuckLake metadata, modelled on how Git stores commits. It says which files go where in a store and how readers and writers handle them. No server is needed: the store has to read, write and write on condition.

“Must” is required. “May” is optional. “Should” is recommended, and a tool that does otherwise has a reason.

What it decides

  • the layout of a dataset, and the name and content of each file (Layout)
  • the fields of a version record, the frozen metadata, and the declarations FDL reads beside it (Versions)
  • the content of refs and the rules for moving them (Refs)
  • when shared objects (data files and assets) may be deleted, and how a version is taken out of retention or withdrawn (Data files)
  • how a reader finds and opens a version, and checks who made it (Reading)
  • the order of writes, and when a push is refused (Writing)
  • the requests a host answers to take pushes over HTTP (Pushing over HTTP)

What it leaves to hosts

  • who can read a dataset, and who may move main. Everything in one dataset is readable by the same people
  • authentication, URL shapes and retention defaults
  • the content of the DuckLake metadata itself, which is DuckLake’s specification
  • how to record what a dataset, a table or a column means to people: titles, descriptions, licenses, sources and guides. FDL defines no field or declaration for them

The format names no host and no service. Fields a host adds go where the specification leaves room for them.

Terms

Term Meaning
store Where a dataset is kept: a bucket, or a folder
dataset Everything under one prefix of a store (the root). It corresponds to a Git repository
root The prefix, ending in /. Paths in this specification are relative to it
metadata DuckLake’s record of the tables, their snapshots and their data files, kept in a database (DuckLake calls it the catalog database)
version One frozen metadata file and its record. Never changed once written
ref A file that holds a version ID. Branches, tags and requests are refs
working metadata A writer’s own DuckLake metadata, in SQLite. Never in the dataset
data file A Parquet file DuckLake writes, shared between versions
host A service that keeps datasets and controls access to them
asset A file the rows of a table point at by its sha256, such as a PDF or an image collected as data
declaration What FDL needs to know about a table, a view or the dataset beyond what DuckLake records: a primary key, the columns that hold assets, whether to profile it, and the other datasets views read. Kept in a version’s tables.json
grace period How long a shared object no version reaches is kept before it may be deleted. 7 days unless a host says otherwise

A DuckLake snapshot is to a version what a small commit on a work branch is to the squash commit that lands on main. Snapshots are made on every write to the working metadata. A frozen metadata keeps only its last snapshot, so the unit a reader can go back to is the version, read from its own metadata.

Was this page helpful?