contract store · 10-store.md · 11-snapshot.md · 12-time.md · 13-replica.md · 14-drain.md · 15-endpoint.md · 16-collect.md · 17-merge.md · 18-pull.md · guide · card

Bucket pulls

A pull downloads the bucket into the store, verifying each object, and writes each table pointer last.

pull

Downloading bucket files, verifying digests, advancing table pointers and restoring named generations.

digest-mismatchtestA downloaded object whose digest differs from its manifest entry is discarded and raises SyncObjectDigestMismatch; run states follow run-state-torn instead.
Why
P4
run-state-torntestA downloaded nodes/<node-id>/run-state.json differing from its listed digest is discarded with a warning naming the key, and the pull continues without it.
Why
because an interrupted push leaves a run state its manifest never committed
identity-conflicttestA pull into a store holding a different store-identity raises StoreIdentityConflict before downloading files.
Why
because importing another UUID lets that store’s owner credential read local rows
convergencetestA named key disappearing mid-download makes the pull re-fetch the manifest and retry the shortfall, up to 3 attempts.
unconvergedtestExhausting those retries raises SyncPullDidNotConverge, naming the key that kept moving, and writes no pointer.
Why
P4
unconverged-resumetestA pull refused under unconverged keeps verified objects, so the next sync pull downloads only keys whose digests still differ.
Why
A-store#exhausted-pull-refuses
transfer-widthtestA pull downloads with up to [sync] transfers concurrent requests, default 16 requests, minimum 1, and writes every verified object before returning any refusal.
Why
because one request per round trip makes a cold boot grow with object count
digest-recordtestPull and push reuse a file’s machine-local sha256 record while size, modification time, change time and inode match it, and hash the file otherwise.
Why
because rehashing every held file on each sync grows with the store
pointer-lasttestA pull writes a table’s pointer only once every Parquet part of the named snapshot exists locally, so no reader meets a pointer before its data.
Why
A-store#replica-read-whole
run-manifest-lasttestA pull writes a run’s _manifest.json only after every other listed object it downloads exists locally, so a concurrent read resolves no run missing parts.
Why
because a read resolves every run whose _manifest.json exists
tombstone-appliedtestPull deletes local keys tombstoned and absent from entries; a tombstoned key with no local copy goes unverified and unreported.
Why
because a deletion that never reaches a consumer resurrects once its tombstone ages out
cold-own-keystest testA pull downloads each key its node owns that the store lacks and held-record does not list, so a replacement host or a widened table scope reusing the node id reads every such key.
Why
because only a key the store held and lost is its own deletion; one it never held carries rows nothing local replaces
pointer-readtestPull and push read the pointer of each table the bucket manifest lists by its key, tables/<t>/_pointer.json, and list no table objects, so the read grows with the tables and not with their objects.
Why
because a bucket gathers parts with every landing, and a pointer read listing them grows with the store’s history
pointer-advancetestPull advances a local pointer whose snapshot the bucket pointer’s chain names, keeping one on the same snapshot at a higher fence, and replaces any other only under pointer-adopt, verifying every advancing table before writing any pointer.
Why
A-store#replica-read-whole
pointer-adopttest testPull replaces a local pointer the bucket pointer’s chain does not name, whatever its fence, save one ahead whose ancestry names the bucket pointer’s snapshot, and warns once local-fold-unrecoverable admits its branch.
Why
because a fold outside a lease writes no fence, and a local snapshot off the bucket’s chain misses every run the bucket’s head folded
local-fold-unrecoverabletestA branch back to the bucket’s chain or a root that folds a run neither on disk, listed in the bucket nor folded on that chain, or meets an absent manifest, raises SyncLocalFoldUnrecoverable naming the runs, and the pointer stays.
Why
because those rows otherwise live only in the replaced snapshot’s parts, which collection deletes once no pointer names them
schema-mergetestA pulled schema.json differing from the local copy merges into it column by column through incompatible‘s lattice rather than replacing it.
Why
because rows landed locally carry columns the bucket’s copy may lack
table-scopetestcontextful sync pull --table <pattern>, repeatable, pulls only the tables its patterns cover through one-matcher. Without the flag, a pull takes [sync] pull_tables, and without either, every table.
Why
because a disposable disk serving one table otherwise downloads every table the bucket holds
tables-emptytestA [sync] pull_tables holding no pattern raises SyncPullTablesEmpty when the config is read; ["*"] names every table.
Why
because an empty list reads as no table to one operator and every table to another
scope-sharedtestA table-scoped pull also takes store-id, control/, every nodes/<node-id>/run-state.json and every cursors/ entry, and no other key outside tables/.
Why
because run states and commit-log entries name no table, and a scoped store lacking them restarts each source
before-runtestWith [sync] pull_before_run = true, run start, pipeline run, mcp and serve pull the tables table-scope selects before their first read, a failed pull stopping the command.
Why
because a container on an empty disk otherwise serves without other nodes’ runs
serve-refreshtestUnder before-run, serve pulls again every 30 s and maps the new generation for reads starting after that pull; a failed refresh keeps serving the held generation and logs the failure.
Why
because a separate query face otherwise serves its starting generation until restart
generationtestcontextful sync pull --generation <N> reads gen-<N>.json in place of the bucket manifest, downloads its in-scope keys at their listed digests and writes exactly its pointers regardless of fence.
generation-listtestcontextful sync generations lists the bucket’s gen-<N>.json files in ascending order with committed_at, committed_by and the project’s entry and tombstone counts, a missing stamp printing as -.
Why
because generation numbers carry no time
generation-absenttestA generation the bucket holds no gen-<N>.json for raises SyncGenerationAbsent, naming N and the newest generation, and writes nothing.
Why
because a restore falling back to another generation reads a state nobody named
generation-divergedtestA local syncable file in scope that the generation omits makes generation pull raise SyncGenerationDiverged, naming the file, and write nothing.
Why
because a kept unlisted run adds rows the generation never held
generation-schematestA generation pull merges the bucket’s current schema.json under schema-merge, refusing nothing over a differing generation digest.
Why
because the lattice keeps every column an earlier generation carried
generation-run-statetestGeneration pull exempts nodes/<node-id>/run-state.json from generation-diverged, keeping run states the generation omits; other unlisted files under nodes/ meet that refusal.
Why
because a run state summarizes history outside the store root
run-state-cursortestAt run open, a run resumes from a pulled run state’s cursor when that pipeline and table’s commit marker is newer than every local one.
Why
because a collected run takes its cursor out of the bucket, so a cold node re-reads the source
control-headtestPull downloads the bucket manifest’s project-scoped control head, snapshot and receipt ancestry at their listed digests. The local applied pointer stays unchanged for pulled-control.
Why
A-surface#synced-control-head

lazy parts

A lazy store pulls metadata and fetches each data part and folded run manifest when a reader first needs it.

lazy-partstestWith [sync] lazy_parts = true, pull skips table Parquet data parts, which the store fetches whole on first read or fold use and verifies under digest-mismatch.
Why
A-store#cold-store-boots
lazy-pointertestUnder lazy-parts, pointer-last waits for the named snapshot’s _manifest.json alone; a part’s digest comes from the bucket manifest when a read first fetches it.
Why
A-store#cold-store-boots
lazy-readytest testUnder lazy-parts, before-run waits for every in-scope object except data parts and the manifests lazy-folded-runs skips.
Why
A-store#folded-runs-on-demand
lazy-folded-runstest test testUnder lazy-parts, pull skips each other node’s run _manifest.json that the pointer’s snapshot chain folds, keeping its _group.json, and lists the skipped keys in the store’s .skipped-runs.json.
Why
A-store#folded-runs-on-demand
husks-skippedtest test test testUnder lazy-parts, pull skips each _manifest.json beside a listed _collected.json where the pointer’s snapshot, once home, holds a valid folded-runs index and an ancestry reaching the local pointer, listing it in .skipped-runs.json.
Why
A-store#index-walk
husks-fetchedtest test test testA full chain walk, or one meeting no valid index, fetches the table’s skipped collected snapshot manifests through the part source at the first it reaches; local retention and a walk past a valid index end there.
Why
A-store#index-walk
husks-stalenesstest testA model’s staleness check reads its head snapshot, each input’s snapshots from the pointer down to the watermark snapshot, and the runs folded beneath through folded-index-walk, fetching a skipped manifest only on that path.
Why
A-store#index-walk
skipped-recordtest test testEach pull clearing its refusals rewrites .skipped-runs.json to the manifests the bucket lists and the store lacks, inside its scope or outside it, a generation restore also keeping each entry outside its scope; an empty record leaves no file.
Why
because a manifest the bucket stopped listing was collected, and an eager pull applies its tombstone
folded-runs-fetchedtest test testListing a table’s committed runs, or an as_of read resolving before the pointer’s snapshot, fetches each skipped manifest through the part source within miss-wait‘s deadline; an unbounded read fetches none.
Why
A-store#folded-runs-on-demand
folded-run-collectedtest test test test testA skipped manifest the part source cannot bring home refuses the read as part-collected does, save where the source answers it collected: an unbounded listing leaves it out, and a bounded read does when a kept snapshot folds its run.
Why
A-store#folded-runs-on-demand
folded-runs-unreachabletestA store listing skipped manifests and opened without a part source raises SyncFoldedRunsUnfetched wherever folded-runs-fetched or husks-fetched fetches, naming the table and a manifest.
Why
A-store#folded-runs-on-demand
folded-runs-sourcetestEvery command opening a store whose [sync] declares lazy_parts registers the bucket’s part source once per process, whether or not the command pulls.
Why
because a command reading runs without pulling otherwise meets folded-runs-unreachable
part-cachetest[sync] part_cache_bytes, default 8 GiB, bounds locally fetched parts; above it, the cache deletes the least recently read part no statement or fold holds.
Why
A-store#cold-store-boots
part-accountingtestEach verified lazy download counts once toward part-cache upon reaching disk, despite other fetch failures. If the cache index is missing, reopening adopts receipt-listed parts without bucket reads or receipt-free tree scans.
Why
because a partial acquisition and a lost index leave completed downloads subject to eviction
part-collectedtestA part lazy-parts needs and the bucket lacks raises SyncPartCollected, naming the part and snapshot; the next pull advances the pointer past that snapshot.
Why
because sync collect deletes a tombstoned run’s parts while a container whose pointer predates the tombstone reads them

reland

A reland lands again the runs a refused pull names and moves the local pointer onto the bucket pointer’s snapshot.

relandtest testcontextful sync reland <table> lands each run local-fold-unrecoverable names again as a new run of this node, then writes the bucket pointer’s snapshot and fence as the local pointer, so the next pull proceeds.
Why
because those rows otherwise wait on a landing by hand and a deleted pointer per table
reland-namedtest testA refusal under local-fold-unrecoverable naming runs names contextful sync reland <table> as its remedy; one meeting an absent manifest names landing the rows by hand.
Why
because an operator reads the remedy where the pull stops
reland-rowstest testA relanded run holds the local snapshot’s rows whose _run_id, and _commit_seq and _ingested_at where the folded-runs index records them, match the run’s, committed at its instant with its site, author and taint.
Why
because one run id repeats across nodes, and the original instant keeps a keyed read’s ranking
reland-keyedtestA relanded run of a keyed table holds only the rows its snapshot kept, and the reland reports beside them the rows the folded-runs index records the run held.
Why
because a later run in the branch superseded the rest, and that run folds again or is relanded too
reland-lineagetestA relanded run’s id is reland- and 16 hex digits of SHA-256 over the local snapshot id and the run key, and its manifest’s relanded_from names that key.
Why
because a repeat then replays the committed run instead of landing its rows twice
reland-ordertestA reland commits every relanded run and syncs its files to disk before writing the pointer; a failure before that write leaves the local pointer and the snapshot’s parts as they stood.
Why
because the pointer write is the step after which no pointer names the snapshot’s parts
reland-unneededtest testA reland of a table holding no local pointer, against a bucket naming no snapshot, of a pointer pointer-advance keeps or advances, or of a branch folding no unrecoverable run raises SyncRelandUnneeded, writing nothing.
Why
because with nothing stranded the pull already brings every row home
reland-head-absenttestA reland while the bucket pointer’s snapshot lacks its manifest on disk, or a Parquet part outside lazy-parts, raises SyncRelandHeadAbsent, naming the snapshot; the refused pull brings it home.
Why
because without the head’s chain on disk every run the head folds reads as unrecoverable
reland-unknowntest testA branch meeting an absent manifest, or a run whose rows no folded-runs row tells apart from another node’s run of its id, refuses the reland as local-fold-unrecoverable does, writing nothing.
Why
because neither names the rows a relanded run holds

A pull converges on the bucket manifest and writes each table pointer last.

Scenarios

store.pull.control-head: WHEN a cold node pulls a signed control head, THEN its local applied pointer stays absent until reconciliation verifies the head.