Skip to content
FDL
Esc
↑↓navigate↵open⌘Jpreview
On this page

Python API

The fdl package - stores, refs, and the repository operations without a terminal.

from fdl import repo, refs
from fdl.stores import open_store

store = open_store("s3://my-bucket/towns/")
for version in repo.log(store):
    print(version.seq, version.id, version.message)
Module Holds
fdl.repo the operations: init, clone, commit, push, log, expire, prune
fdl.declared the declarations of a version: primary keys, asset columns, profiles and dependencies, their checks and their merge
fdl.diff what changed between two versions (diff.versions) or in a working metadata file (diff.working)
fdl.refs reading and moving refs under their conditions
fdl.version Version, the record in version.json, and new IDs
fdl.catalog DuckLake metadata files in DuckDB and SQLite, and the copy between them
fdl.layout where each file of a repository sits
fdl.store, fdl.stores what a store has to do, and the folder and S3 stores

Stores

open_store(url) returns a folder store, an S3 store, or an HTTP store for a host that takes pushes over HTTP. Any object with these methods works in their place:

Method Does
put(key, data, *, content_type, if_match, if_none_match) Write and return the ETag. A failed condition raises an error whose status is 412
put_file(key, path, …) The same, from a file
get(key) Read. A missing object raises an error whose status is 404
head(key) The object’s headers (with ETag), or None
list(prefix) Keys under a prefix
delete(keys) Remove. A key already gone is not an error
put_files(files, *, content_type) Optional. Write (key, path) pairs that are written once. The HTTP store asks the host for upload URLs (batch) and sends the files straight to storage

fdl.repo

repo.init(store, *, data_dir="data/") -> Config

Start a repository. An existing one is left as it is and its settings returned.

repo.clone(store, working, *, ref="HEAD", local_data=None, version=None) -> Version | None

Make a working metadata file at working from the version ref holds, or from version when given. Returns that version, or None before the first one. A store over HTTP needs local_data, the folder new data files go to. Refuses a metadata file whose sha256 is not the one version.json records, or that names a data file outside the data folder.

repo.commit(working, out_dir, *, parent, data_path, author, message, annotations=None, allow_empty=True, setup=(), facts=None, inherited=None, merged=None) -> Version

Freeze the working metadata into out_dir/<ID>/ and write its tables.json with facts, the declarations (declared.Facts), after checking them against its tables and views; raises declared.DeclarationError with the problems found. data_path is repo.data_url(store). inherited are the parent’s declarations (repo.declarations(store, parent)), for telling whether anything changed and for the diff. Nothing is uploaded.

repo.record(frozen, out_dir, *, parent, author, message, annotations=None, described=None, inherited=None, merged=None) -> Version

Make a version around a metadata file that is already frozen, for a writer that freezes it itself.

repo.declarations(store, version) -> declared.Facts
repo.tables_of(store, version) -> set[str]

What a pushed version declares, and its tables and views, from its tables.json.

repo.push(store, local, version, *, ref="refs/heads/main", move_ref=True) -> None

Upload a committed version, then move ref to it on condition. Raises NotFastForward when the ref moved, and RepoError when data files are missing or tables.json does not match version.json and the metadata. With move_ref=False it uploads and stops, for a host that moves refs itself.

repo.log(store, name="HEAD") -> Iterator[Version]
repo.head(store, name="HEAD") -> Version | None
repo.is_ancestor(store, ancestor_id, descendant) -> bool

Walk the first-parent line, read the version a ref holds, and check whether a version is behind another, through every parent.

repo.merge_bases(store, ours, theirs) -> list[Version]
repo.forward(store, ref, version) -> None

The common ancestors of two versions that no other common ancestor descends from: a merge needs exactly one. forward moves a branch to a version already in the repository that descends from where it is, for a merge whose base is the branch’s own version.

merge.primary_keys(base, ours, theirs) -> dict[tuple[str, str], list[str] | None]
merge.tables(working, base, theirs, *, keys, repository_data, data_path=None, setup=(), write=True) -> Plan

The pieces of fdl merge. primary_keys gives each table’s primary key from what each version declares and holds ((repo.declarations(store, version), repo.tables_of(store, version))): None for one whose primary key differs between the three, which is a conflict. A table it does not list has no primary key, and its rows are merged as a multiset. tables writes into the working metadata, in one snapshot, what the frozen metadata theirs changed since base, or nothing when there is a conflict; Plan lists the conflicts and what came over. repo.commit and repo.record take merged=<theirs> to make the merge version.

repo.readable(store, name="HEAD") -> Version

The version a ref holds, checked the way a reader must before opening its metadata. Raises RepoError before the first version, for a version.json of a format this does not read, and for a version past retention or withdrawn. fdl url uses it.

repo.expire(store, version_id, *, reason="retention") -> None
repo.plan_retention(store, *, keep=1, since=None, grace=timedelta(days=7), now=None) -> list[str]
repo.prune(store, *, keep=1, since=None, grace=timedelta(days=7), reason="retention", by=None) -> list[str]

Take one version out of retention, or every version plan_retention chooses (Choosing the versions to take out). plan_retention only lists them. Versions a ref points at are kept.

repo.tag(store, name, version_id) -> None
repo.delete_tag(store, name) -> None
repo.tags(store) -> dict[str, str]

Create refs/tags/<name> at a version, delete it, or list the tags with the versions they hold. A tag is never moved: tag raises RepoError when the name is already there at another version.

repo.branch(store, name, version_id) -> None
repo.delete_branch(store, name) -> None
repo.branches(store) -> dict[str, str]

The same for refs/heads/<name>. delete_branch keeps the branch HEAD names.

repo.update_refs_index(store) -> None

Rewrite refs.json from the refs. push, tag and delete_tag call it after moving a ref. A 403 from a host that keeps refs.json itself is taken as done.

repo.plan_withdrawal(store, version_ids, *, objects=None, refuse_refs=True) -> Withdrawal
repo.withdraw(store, version_ids, *, plan=None, by=None) -> Withdrawal

Withdraw versions that hold data that must not be published. plan_withdrawal changes nothing and returns the tags, the data files to delete, and the files kept because a kept version reads them. withdraw deletes the tags, the data files and the metadata files, in that order. Both raise RepoError while a branch or a request points at one of the versions; with refuse_refs=False, plan_withdrawal lists them in blocked instead and withdraw raises. objects ({"data": [...], "assets": [...]}, as deleted of expired.json) applies a withdrawal made in the repository this one was copied from: every live version that reaches one of them is affected too.

fdl.declared

declared.Facts(primary_keys={}, assets={}, dependencies={}, unprofiled=frozenset())
declared.from_index(index) -> Facts
declared.check(facts, found) -> list[str]
declared.merged(base, ours, theirs) -> tuple[Facts, list[str]]
declared.changed(old, new, names) -> list[str]
declared.dependencies_changed(old, new) -> list[str]

Facts holds the declarations of a version: primary_keys and assets by schema.table, dependencies by name ({"url": ..., "version": ...}), and unprofiled, the tables and views whose profile is false. of(name) gives what it declares of one table or view as tables.json writes it, and to_json and from_json keep it in a file. from_index reads them from a tables.json. check returns what the rules refuse, given found_in(working, data_path=...), the tables and views of a working metadata. merged merges three ways and returns the conflicts. changed and dependencies_changed give the names a diff lists.

fdl.diff

diff.versions(store, old, new) -> dict
diff.working(working, base) -> dict
diff.lines(result) -> list[str]

versions compares two versions (old is None before the first one) from the newer one’s diff when the older is its parent, and from their tables.json otherwise. working compares a working metadata file with base, the version it was cloned at. Both return tables (added, dropped), rows, columns and data, as fdl diff --json prints them; versions also returns declarations and dependencies. lines gives the lines fdl diff prints.

fdl.refs

refs.read(store, name) -> Ref | None
refs.write(store, name, version_id, *, expected) -> str

read returns the ref with the ETag a move has to name. write creates the ref when expected is None and moves it otherwise, and raises RefConflict when the condition fails.

refs.listed(store) -> dict[str, str]
refs.write_index(store) -> dict

listed reads every branch and tag afresh. write_index writes refs.json from them and HEAD.

Was this page helpful?