Docs / Specification

A-store — The store and its sync decisions

Status: accepted

synced-uuid-binds — A synced UUID binds a store across clones

Cited bystore.init.store-identitystore.init.store-identityAn initialized store holds one synced, canonical version-4 UUID in store-id, which repeated init and clones keep and a recreated root replaces.store.init.identity-invalidstore.init.identity-invalidA malformed store-id, or an owner mint naming no initialized store root, raises StoreIdentityInvalid.

A signed owner claim from a removed root does not follow its filesystem path. A pull into an independently identified store refuses before copying data.

empty-complete-snapshot — An empty complete snapshot carries a replacement frontier

Cited byrun.advance.snapshot-completionrun.advance.snapshot-completionA snapshot source reports complete inventory or unchanged skip independently of row count; a replacing run carries only complete inventory into its manifest.run.advance.zero-row-commitrun.advance.zero-row-commitA complete empty snapshot of a replacing table commits empty-replacement; a skipped unchanged input holds the frontier and position, and a failed pull publishes neither.

A zero-row result means either a complete empty inventory or a skipped unchanged input. Only the complete result clears current reads.

schema-lattice-has — The schema lattice has one promotion, and a key never widens

Cited bystore.reconcile.latticestore.reconcile.latticeThe type lattice holds one promotion, Int64 with Float64 to Float64; a JSON type absorbs Utf8.store.reconcile.key-wideningstore.reconcile.key-wideningPrimary-key columns take no Float64 promotion: a widening batch raises StoreKeyWidened before any Parquet, and so does a fold meeting a key already reconciled to Float64.

Int64 with Float64 is the only pairing that produces physically mixed Parquet, and the scan widens it with no per-column cast. A widened key past the exact-integer range merges distinct keys under the fold.

Revisit: a second pairing that some source emits routinely and the scan resolves losslessly.

coordination-probed-compare — Coordination is probed compare-and-swap with fences the storage enforces

Cited bytopology.coordinate.primitivetopology.coordinate.primitiveCoordination rests on one property: a linearizable conditional write. No component depends on a stronger one, and the tree ships no consensus implementation or external coordination service.topology.coordinate.inventorytopology.coordinate.inventorySingle-writer operations comprise lease-row updates, cursor compare-and-swap, pointer-commit, lease acquisition, leased pipeline commit-log entries, bucket manifest commits and configuration version claims. No other operation needs a single writer.topology.coordinate.lease-rowtopology.coordinate.lease-rowA lease key names a pipeline, source partition, table compaction or deployment cadence. One conditional update assigns the row's holder, expiry instant and fence.topology.coordinate.catalog-clocktopology.coordinate.catalog-clockThe catalog evaluates a lease row's expiry against its own clock, never a caller-supplied instant.topology.coordinate.fence-advancestopology.coordinate.fence-advancesEach acquisition of a lease row increments its fence, and release keeps the fence, so the fence never repeats for a key.topology.coordinate.fenced-committopology.coordinate.fenced-commitA commit under a lease is a conditional write predicated on the holder's fence; a predicate matching nothing is stale-fence.topology.coordinate.cadence-lease-ttltopology.coordinate.cadence-lease-ttlThe catalog grants a cadence lease for 90 s.topology.coordinate.cadence-lease-renewaltopology.coordinate.cadence-lease-renewalThe reconciler renews a cadence lease every 30 s while its applied document schedules a dispatchable unit.topology.coordinate.cadence-fallbacktopology.coordinate.cadence-fallbackA deployment's fallback cron fires only after taking the cadence lease itself, so one fire holds the current fence.topology.coordinate.weak-backendtopology.coordinate.weak-backendA catalog backend whose conditional update is not linearizable refuses at open with weak-conditional-backend.store.fold.compaction-leasestore.fold.compaction-leaseA pass holds the table's compaction lease and stamps its fence into the snapshot manifest and the pointer.store.fold.pointer-commitstore.fold.pointer-commitA pass publishes by replacing _pointer.json conditioned on the ETag read at pass start. Object stores use If-Match; filesystems use a version-checked rename.store.push.pointer-fencedstore.push.pointer-fencedA local pointer fenced below the bucket pointer stays local, and the push warns beside its commit.store.probe.inconclusivestore.probe.inconclusiveAn unsupported-method response, a forbidden response or a transport error raises SyncProbeInconclusive and counts as capability not demonstrated.store.probe.unprovenstore.probe.unprovenDeclaring cas against a backend the probe did not demonstrate raises SyncCoordinationUnproven and stops the push.store.lease.heldstore.lease.heldIf a run finds an unexpired lease, the engine raises LeaseHeld, names the holder and expiry, and skips the run. The engine retries the run at the next reconciliation tick.store.lease.stale-fencestore.lease.stale-fenceIf a commit-log create, pointer replacement or catalog UPDATE loses its condition to a higher commit fence, the operation raises LeaseFenced. The associated run or snapshot stays unreadable.store.lease.not-heldstore.lease.not-heldReleasing a lease another node holds, or one a later acquisition of the same node fenced past, raises LeaseNotHeld and leaves the object untouched.store.lease.acquirestore.lease.acquireAcquisition creates the lease object under If-None-Match or replaces an expired or released object under If-Match on its ETag. The acquired commit fence is one above the object's commit fence.store.lease.pointer-fencestore.lease.pointer-fenceTaking a table's compaction lease raises the fence stored in its bucket pointer, so a publish carrying a lower fence loses its condition.store.lease.commit-logstore.lease.commit-logUnder a machine lease, a run commits by creating the next cursors/<pipeline-id>/<node-id>/<seq>.json. Acquisition creates an entry carrying its fence. A manifest marked logged becomes readable when that log records the manifest.

A fence the writer checks before commit lets a holder paused between check and write commit after its successor. The storage evaluates the fence inside the conditional write instead.

Consequences: a backend without demonstrated conditional writes runs single-writer by refusal, never by silent degradation. Revisit: a backend whose access policy enforces a generation bound for every writer.

replica-read-whole — A replica is a read-only whole-snapshot cache

Cited bystore.replicate.write-refusedstore.replicate.write-refusedA write verb against a replica raises ReplicaWriteRefused, naming the canonical store.store.replicate.partial-parquetstore.replicate.partial-parquetA replica lacking any of a snapshot's Parquet files makes refresh raise ReplicaPartialParquet and leave that snapshot unpublished.store.replicate.descriptorstore.replicate.descriptorA replica's pull writes replica.json beside derived.sqlite, naming per table every held snapshot whose parts and sidecar paths exist on disk; push excludes the file.store.replicate.missing-indexstore.replicate.missing-indexA query needing a sidecar or partition the replica lacks raises ReplicaMissingIndex, naming the refresh supplying it.store.replicate.sensitive-refusedstore.replicate.sensitive-refusedA refresh requesting a replicate-off table raises ReplicaSensitiveTable. The consumer reads that table through the query proxy.store.pull.pointer-laststore.pull.pointer-lastA pull writes a table's pointer only once every Parquet part of the named snapshot exists locally, so no reader meets a pointer before its data.store.pull.pointer-advancestore.pull.pointer-advancePull advances a local pointer whose snapshot the bucket pointer's chain names, keeping one on the same snapshot at a higher fence, and replaces any other only under pointer-adopt, verifying every advancing table before writing any pointer.

A replica never answers from what is present and never fetches mid-query, so its answer is complete for the snapshot it names, or a refusal. Cursor and lease rows live in machine.sqlite, which no pull or rebuild touches.

bucket-index-commits — The bucket index commits by compare-and-set over a scoped union of the remote

Cited bystore.merge.tombstone-ownerstore.merge.tombstone-ownerA tombstone whose owner differs from the owner of the entry it names raises SyncTombstoneForeign, and the merge keeps the entry.store.merge.tombstone-signedstore.merge.tombstone-signedPush signs each Tombstone with the node's issuer key over the key it names, its owner and its deletion instant. A node without an issuer key writes unsigned tombstones.store.merge.tombstone-unverifiedstore.merge.tombstone-unverifiedUnsigned tombstones, invalid signatures, or keys absent from the key-set ledger's verifying set at deleted_at raise SyncTombstoneUnverified. Merge preserves the named entry; pull deletes nothing.store.merge.retriesstore.merge.retries[sync] push_retries, default 5 attempts, bounds consecutive re-commit rounds whose re-read manifest carries no newer generation than the round before.store.merge.round-ceilingstore.merge.round-ceilingA push stops its manifest commit loop after 64 rounds, whatever generations those rounds meet.store.merge.exhaustedstore.merge.exhaustedExhausting retries or round-ceiling raises SyncManifestRebaseExhausted, reports every uploaded object as in the bucket and asks for a re-run.store.merge.scoped-unionstore.merge.scoped-unionA merge takes local entries owned by this writer or unowned, and all other remote entries. It tombstones a writer-owned remote entry only when held-record lists the key and the store lacks the file.store.merge.held-recordstore.merge.held-recordPush records locally held owned keys under its node id in .sync-held.json, adding before commit and replacing that node's keys after; a pull holding the push guard adds each such key the bucket manifest lists.store.merge.held-absentstore.merge.held-absentWith no keys for the node in .sync-held.json, a surviving .push.lock means the store holds every remote key the node owns, and none without it.store.merge.ownershipstore.merge.ownershipA key names its owner by the node in its run directory, request-ledger filename, commit-log directory or run-state directory; any other key is unowned and propagates no deletion.

The remote contributes an entry only where this writer has never written, so an owned entry absent locally drops out and retention reaches a consumer.

Consequences: a contended bucket can leave a writer’s objects durable and unlisted, visible in the refusal. Revisit: a key shape whose ownership is not derivable from the key; the index outgrowing a single-object commit.

canonical-metadata-sits — Canonical metadata sits inside an authenticated envelope

Cited bystore.encrypt.metadata-envelopestore.encrypt.metadata-envelopeWith a bound key, schema, manifest, pointer, counter and commit-log files enclose canonical JSON or text in an authenticated versioned envelope; without one they keep their canonical bytes.store.encrypt.run-state-envelopestore.encrypt.run-state-envelopeWith a bound key, a node's synced run-state JSON encloses canonical bytes in an authenticated envelope, and readers refuse a file sealed under another key.

Metadata files carry names and values from rows, so a bound store holds zero plaintext canary bytes in them. Schema merge operates on decrypted canonical structures.

machine-catalog-seals — The machine catalog seals a SQLite snapshot

Cited bystore.lay-out.machine-catalogstore.lay-out.machine-catalogmachine.sqlite holds the machine's journal, cursor cache, leases and run-row cache; sync excludes it and pull preserves it. Rebuild refills run rows from reserved-table only when none exists, and rebuilds nothing else.store.encrypt.machine-catalog-sealingstore.encrypt.machine-catalog-sealingA bound machine.sqlite holds an authenticated SQLite snapshot: each catalog transaction locks the file, reads the snapshot into memory and seals the committed bytes before unlocking.

A bound store protects catalog row values on disk and keeps one linearizable compare-and-swap across processes.

sidecar-memory-mapped — A sidecar is memory-mapped only over plaintext

Cited bystore.encrypt.metadata-envelopestore.encrypt.metadata-envelopeWith a bound key, schema, manifest, pointer, counter and commit-log files enclose canonical JSON or text in an authenticated versioned envelope; without one they keep their canonical bytes.store.encrypt.run-state-envelopestore.encrypt.run-state-envelopeWith a bound key, a node's synced run-state JSON encloses canonical bytes in an authenticated envelope, and readers refuse a file sealed under another key.store.encrypt.machine-catalog-sealingstore.encrypt.machine-catalog-sealingA bound machine.sqlite holds an authenticated SQLite snapshot: each catalog transaction locks the file, reads the snapshot into memory and seals the committed bytes before unlocking.store.encrypt.sidecar-readerstore.encrypt.sidecar-readerUnencrypted projects' readers memory-map sidecars read-only; encrypted projects' readers decrypt each sealed sidecar into anonymous process memory, writing no cleartext to disk.read.retrieve.sidecar-size-capread.retrieve.sidecar-size-capA vector sidecar exceeding 64 MiB resident memory stays unloaded; retrieval uses an exact scan. Resident memory includes each graph node's 4 B probe mark and, for sealed sidecars, the graph file.read.retrieve.fulltext-sealed-capread.retrieve.fulltext-sealed-capA sealed full-text sidecar file larger than 256 MiB stays unopened, and its arm adds no candidates.

A sealed file cannot be memory-mapped, and the HNSW and full-text readers scale past memory only by mapping plaintext.

Revisit: a builder that reads through a caller-supplied page source, making per-page decryption a reader rather than a format.

sidecar-names-id — Every sidecar names one id column

Cited bystore.declare.empty-replacementstore.declare.empty-replacementA complete zero-row replace run marks its manifest a replacement frontier, which push, pull and fold preserve; current reads exclude earlier files, and bounded reads before it keep earlier rows within retention.store.declare.fold-jobstore.declare.fold-jobA [[job]] of kind fold with a schedule, enabled absent or true and target absent or naming the table's destination covers the table; pipeline validate warns on stderr for each keyed table no job covers.store.index.id-columnstore.index.id-columnWithout valid_time, id_column defaults to the table's single-column primary key; a table's sidecars share one id_column, whether it is keyed, composite-keyed or unkeyed.store.index.id-column-unresolvedstore.index.id-column-unresolvedManifest validation raises StoreIndexIdColumnUnresolved when id_column is missing without a single-column primary key or with valid_time, names a valid-time table's single key, or differs across sidecars.store.index.id-uniquestore.index.id-uniqueTwo staged snapshot rows sharing an id_column value make fold raise StoreIndexIdNotUnique, naming the table, column and value, and publish nothing.store.index.bloom-partsstore.index.bloom-partsWithout a project encryption key, each landed or folded part carries bloom filters, sized to its row count, on the columns bloom_filter names; erasure rewrites carry none.store.index.vector-by-foldstore.index.vector-by-foldFold builds each declared vector sidecar from staged rows with non-null identifiers and non-zero vectors, of the declared model where embedding_model exists, and records it in the snapshot manifest.store.index.fulltext-by-foldstore.index.fulltext-by-foldFold builds each declared full-text sidecar from staged rows with non-null identifiers and text yielding a term, recording it in the snapshot manifest.

Keyed, composite-key and unkeyed tables alike need a sidecar, with ids stable across folds. A valid-time table repeats its key across lines, so it declares the column.

binary-fixed-size — Binary and fixed-size vector columns take no promotion

Cited bystore.reconcile.incompatiblestore.reconcile.incompatibleAny other pair of types observed for one column raises StoreSchemaIncompatible at the write, naming the column, stored type and arriving type.store.reconcile.binary-and-vectorstore.reconcile.binary-and-vectorThe lattice includes Binary, FixedSizeBinary(n) and dimension-n FixedSizeList of Float32 or Float16, none promoting; a width, item or dimension change meets incompatible.store.reconcile.typed-landingstore.reconcile.typed-landingA producer declares these types; JSON batches carry bytes as padded base64 and vectors as number arrays of the declared dimension, other values meeting incompatible.store.reconcile.stored-typestore.reconcile.stored-typeOnce schema.json records a binary, vector or nested type, later undeclared JSON values land in that type without a declaration.store.reconcile.half-widthstore.reconcile.half-widthA Float16 vector stores elements at half width in Parquet; one relation cast reads its elements as FLOAT, either binary type as BLOB, and a vector as an ARRAY of its dimension.read.respond.bytes-and-vectorsread.respond.bytes-and-vectorsThe serializer encodes binary as padded base64. Fixed-size float arrays, including vector columns, become JSON arrays whose elements use float-cell encoding.authority.mask.typed-strategyauthority.mask.typed-strategyA binary column masks by drop or by hash over its bytes, a vector or nested column by drop alone over the whole column; another strategy on either raises EnforceStrategyOutsideType at manifest load.

An embedding stored as a JSON array costs several times its float16 size and a parse per vector read.

store-s-catalogs — The store’s catalogs sit behind a port with SQLite in its own package

Cited bytopology.coordinate.cursor-castopology.coordinate.cursor-casA cursor compare-and-swap is one conditional update predicated on the stored version, read back by its affected-row count.topology.coordinate.catalog-porttopology.coordinate.catalog-portEngine code accesses catalog backends through the Catalog trait without naming a backend. contextful-cli selects the backend through dependency injection.topology.coordinate.backendstopology.coordinate.backendsSingle-node self-hosting uses a local catalog file. Clusters use Postgres through pg-catalog. Cloudflare execution uses D1 through d1-catalog.store.lay-out.derived-catalogstore.lay-out.derived-catalogderived.sqlite is a cache: contextful context rebuild-catalog reconstructs it from pointers, reachable manifests, committed run manifests and every schema.json, and sync excludes it.store.lay-out.catalog-portsstore.lay-out.catalog-portsThe store accesses machine.sqlite through catalog-port and derived.sqlite through DerivedCatalog. Both traits reside in contextful-core; contextful-sqlite implements both. A host implements either trait over its own connection.

Cargo admits one links = "sqlite3" package per graph, so a store forcing a libsqlite3-sys major or bundled breaks a host linking its own.

variant-column-keeps — A variant column keeps each row’s scalar kind in one Struct

Cited bystore.reconcile.variantstore.reconcile.variantA variant column stores string, Int64, Float64, boolean or padded-base64 bytes values in one Struct of kind and a matching field, landing scalar or tagged inputs losslessly; conflicting non-null fields meet incompatible.

One attribute key arrives as a string from one sender and an integer from another. The Struct reads back each value with the type it was sent as, under one Arrow type.

Revisit: Parquet VARIANT support measured.

sqlite-link-rule — The SQLite link rule reads normal dependencies alone

Cited bytopology.package.sqlite-adaptertopology.package.sqlite-adaptercontextful-sqlite alone declares the SQLite binding as a normal dependency and enables no link feature itself; only contextful-cli turns on its bundled feature. Any other normal-dependency declaration or enablement raises SqliteLinkForced, naming the manifest line.topology.package.store-sqlite-freetopology.package.store-sqlite-freecontextful-context reaching libsqlite3-sys through a normal dependency, with its default features, on any target, raises StoreLinksSqlite, naming the path that pulled it.

Under Cargo’s resolver 2 a dev-dependency enters only test builds, and a build-dependency resolves apart. Neither forces SQLite into a profile.

keyed-table-covered — A keyed table is covered by a scheduled fold job

Cited byrun.seed.compaction-cadencerun.seed.compaction-cadenceA seeded table that no job meeting fold-job covers raises PipelineSeedCompactionMissing at validate and before a fire, naming the table.

A pass fires only when something invokes context compact, so an uncovered keyed table accumulates runs without bound. Coverage is computable at validation from the declaration alone.

struct-list-map — Struct, list and map columns reconcile field by field

Cited bystore.reconcile.nested-latticestore.reconcile.nested-latticeStructs gain fields, which files lacking them read as null; list items and map values reconcile under lattice. A kind change, scalar against nested or list against map, meets incompatible, naming the column path.store.reconcile.nested-landingstore.reconcile.nested-landingProducers or declarations type columns struct<…>, list<…> or map<utf8, …>, which JSON batches carry as objects and arrays; a struct key outside its fields meets incompatible.read.respond.nested-valuesread.respond.nested-valuesLists encode as JSON arrays. Structs and maps encode as JSON objects keyed by field name or key text. Each item, field or value uses cell encoding.run.normalize.native-storerun.normalize.native-storeOn the store sink, native lands each undeclared column of objects and arrays as a struct or list column inferred over the batch; a column mixing kinds, or one schema.json holds as a scalar, lands as Json.

A record reads with its parent in one scan, and the lattice stays the one place a type widens.

manifest-carries-format — A manifest carries a format version whose major refuses

Cited bystore.lay-out.table-pointerstore.lay-out.table-pointertables/<t>/_pointer.json names the table's current snapshot and commit fence; readers reach only snapshots reachable from it directly or through parent links.store.lay-out.format-versionstore.lay-out.format-versionRun and snapshot manifests carry format_version as <major>.<minor>: major 2 for snapshots referencing parts or segments, else 1; minor 1 for snapshots carrying folded_runs, else 0. An absent version reads as 1.0; readers parse a newer minor as their own.store.lay-out.format-newerstore.lay-out.format-newerA run or snapshot manifest of a newer format_version major raises StoreManifestFormatNewer, naming the file and both versions, and the table refuses reads until a newer build reads it.store.lay-out.format-migratestore.lay-out.format-migrateA major-2 reader reads a 1.x snapshot manifest as one part per directory, counting rows from footers; the next pass references those parts under major 2, rewriting no manifest in place.

A minor adds defaulted fields an older build ignores; a major changes meaning. The version sits in the file it governs.

pass-carries-untouched — A pass carries untouched parts and sidecar segments by reference

Cited bystore.fold.carry-by-referencestore.fold.carry-by-referenceFor append tables, a pass writes unfolded rows as one delta part per affected partition and rewrites only parts replacement or retention touches, referencing every other part in its origin snapshot.store.fold.tiered-mergestore.fold.tiered-mergeIn each written partition a size tier spans row counts between consecutive powers of 4, and a pass merges 4 parts of one tier into one part until every tier holds fewer than 4 parts.store.fold.reference-retentionstore.fold.reference-retentionCollection keeps the parts, sidecar segments and folded-runs segments a retained snapshot references and removes the rest of the collected snapshot's directory, its manifest included save under manifest-kept.

A store.foldstore.foldCompaction passes, their triggers and retention, and snapshot publication through a pointer update. pass that rewrites every part writes bytes in proportion to the table, so each landing over a growing history costs more than the last.

Consequences: a mixed fleet upgrades every reader before a writer folds by reference. Revisit: restatements that rewrite large parts on every pass.

sidecar-s-builder — A sidecar’s builder is recorded, not encoded in its path

Cited bystore.index.identitystore.index.identityA sidecar's identity is its one path under paths; derived.sqlite records the builder and builder version of each sidecar a reachable snapshot holds.store.index.rebuildstore.index.rebuildA pass rebuilds in full, into a new snapshot, a current sidecar whose builder or builder version differs from this build's; readers keep the prior snapshot until pointer publication.store.index.graph-formatstore.index.graph-formatThe vector graph's byte layout carries no compatibility guarantee: a reader opens a sidecar only at its own builder and builder version, else sidecar-falls-back.store.index.graph-extensionsstore.index.graph-extensionsA vector or full-text entry counts in extensions the segment-adding passes since its last full build; a pass finding 16 extensions rebuilds in full and records 0.store.index.sidecar-segmentsstore.index.sidecar-segmentsA delta replacing and removing no stored row adds a segment to each current sidecar, referencing earlier ones; probes read every segment, scoring full text over their summed statistics.

The graph layout changes with its builder, and one path serves each declaration.

Consequences: writers of two builder versions rebuild each other’s sidecars on every pass, so an upgrade replaces every writer first. Revisit: a second builder declared over one column.

tenant-value-bytes — A tenant value is its bytes

Cited bystore.index.tenant-verbatimstore.index.tenant-verbatimLanding applies no canonical form to tenant values: writes and comparisons keep bytes without trimming, case folding or Unicode normalization, and percent-escaping changes only the directory name.

A tenant scope binds the partition column by byte equality, so the grant, the directory and the row agree without a rule each applies.

Revisit: a source whose tenant spelling varies between batches.

sidecar-copies-tenant — A sidecar copies only the tenant, zone and availability instant

Cited bystore.index.filter-copiesstore.index.filter-copiesA sidecar copies only id_column, the indexed column, and the tenant column, zone label and availability instant filtered traversal needs; every other filter applies at candidate-ids.

Any column copied into a sidecar sits outside the enforced relation. A filter beyond tenant and zone costs a pre-scan of the relation, never exposure.

Revisit: a declared filter whose pre-scan misses latency targets.

validity-interval-row — A validity interval is one row

Cited bystore.bound-time.interval-rowsstore.bound-time.interval-rowsA keyed table declaring valid_time stores one row per key and valid-time interval, deduplicating on its primary key and from, so a fold keeps every interval.store.available-time.vintage-dedup-keystore.available-time.vintage-dedup-keyA vintage table reads through dedup-view, or interval-rows under valid_time, and folds under dedup-view or valid-time-line, each adding the availability column to the key and keeping every vintage; an unclustered valid_time fold sorts by from, then primary key.

A key can be valid over several intervals, and a bounded read stays one predicate over the declared pair.

Revisit: a key whose interval count makes the row set dominate the table.

exhausted-pull-refuses — An exhausted pull refuses and resumes

Cited bystore.pull.unconverged-resumestore.pull.unconverged-resumeA pull refused under unconverged keeps verified objects, so the next sync pull downloads only keys whose digests still differ.

Pushes arriving faster than a re-fetch shrinks its shortfall exhaust a pull. A contended pull then converges across runs, each downloading less.

Revisit: pulls refusing under ordinary write rates.

tombstone-signed-node — A tombstone is signed by its node’s issuer key

A node id carries no key material, so an unsigned tombstone lets any writer claim another node’s deletion. The issuer key adds no new key custody.

Revisit: a deployment whose writers hold no issuer key.

replica-advertises-descriptor — A replica advertises a descriptor beside its catalog

Cited bystore.replicate.descriptorstore.replicate.descriptorA replica's pull writes replica.json beside derived.sqlite, naming per table every held snapshot whose parts and sidecar paths exist on disk; push excludes the file.store.replicate.missing-indexstore.replicate.missing-indexA query needing a sidecar or partition the replica lacks raises ReplicaMissingIndex, naming the refresh supplying it.

A router needs a replica’s holdings before sending a query, with no central service.

availability-declared-column — Availability is a declared column resolved at landing

Cited bystore.available-time.declarationstore.available-time.declaration[pipeline.tables.available_time] declares AvailableTime, omitted from canonical serialization when unset. column holds the source availability instant as timestamptz, which landing keeps and availability bounds compare alone; zone defaults to UTC.store.available-time.available-as-ofstore.available-time.available-as-ofavailable_as_of filters the inner source with <column> <= available_as_of under beneath-enforcement, combining with as-of and valid-as-of when the read supplies them.

A backfill stamps _ingested_at with its commit instant, so a 2026 load of 2014 history answers a 2014 as_of read with nothing. The instant is the source’s, not the engine’s.

Revisit: a source publishing a per-row availability stamp the engine verifies.

restatement-vintage-row — A restatement is a vintage row

Cited bystore.available-time.vintage-rowsstore.available-time.vintage-rowsA keyed table declaring available_time is a vintage table, keeping every vintage per primary key, valid-time from and availability instant; a read returns the latest at or before any available_as_of, or under vintage = "first" the earliest.store.available-time.vintage-dedup-keystore.available-time.vintage-dedup-keyA vintage table reads through dedup-view, or interval-rows under valid_time, and folds under dedup-view or valid-time-line, each adding the availability column to the key and keeping every vintage; an unclustered valid_time fold sorts by from, then primary key.

A keyed table keeps one survivor per key, so a revision overwrites the value an earlier reader saw. A past availability bound answers the same in every later year.

Revisit: an estimate-revision source whose vintage rows outgrow the table’s partition bounds.

vintage-landing-reads — A vintage landing reads only its own keys’ history

Cited bystore.available-time.vintage-scopestore.available-time.vintage-scopeA vintage landing compares only its own keys' stored vintages, skipping row groups that text or signed-integer key bloom filters, or footer bounds on text, integer or temporal keys or temporal valid time, exclude.

The content-hash comparison runs under the commit lock, so decoding every stored vintage makes a daily backfill quadratic in its days. Bloom filters stay off in an encrypted project, where Parquet keeps them in plaintext.

Revisit: a random-key vintage table declaring no bloom filter whose landings decode its history.

column-records-commit — A column records the commit that first landed it

Cited byread.horizon.late-columnsread.horizon.late-columnsA column's first merge sets schema.json's first_landed_at once to the batch's earliest availability instant, otherwise commit instant. Columns with first_landed_at after the horizon are absent from registered schemas and describe payloads.

On a table declaring available_time, the instant is the batch’s earliest availability, so a late backfill keeps its fields visible at the horizons its rows answer.

snapshot-corpus-declared — A snapshot corpus is a declared table pinned by snapshot id

Cited bystore.snapshot.capturestore.snapshot.capture[pipeline.tables.snapshot] declares SnapshotDecl, the columns describing each captured document. Each capture commits as one store snapshot, and an evaluation case pins it by snapshot id.store.snapshot.post-horizon-capturestore.snapshot.post-horizon-captureA document captured after the reading horizon carries _available_reconstructed true under live-since, and a horizon read admits it only while the block's leak_score column holds 0.

A capture uses the ordinary write path. An unscored late capture never answers a horizon read.

adjustment-factors-apply — Adjustment factors apply at read time, back from each factor’s effective date

Cited bystore.available-time.vintage-rowsstore.available-time.vintage-rowsA keyed table declaring available_time is a vintage table, keeping every vintage per primary key, valid-time from and availability instant; a read returns the latest at or before any available_as_of, or under vintage = "first" the earliest.store.available-time.vintage-dedup-keystore.available-time.vintage-dedup-keyA vintage table reads through dedup-view, or interval-rows under valid_time, and folds under dedup-view or valid-time-line, each adding the availability column to the key and keeping every vintage; an unclustered valid_time fold sorts by from, then primary key.

A measurement table keeps raw values. Under available_as_of, each listed column multiplies by the factors available by the bound that take effect after the row’s valid-time from.

push-tombstones-owned — A push tombstones only the owned keys its store held

Cited bystore.push.manifest-commitstore.push.manifest-commitA push commits by replacing <prefix>/manifest.json under If-Match on the ETag it read; this bucket manifest lists each key's sha256, size and owner.

Ownership reads the node id off a key, so two disks pushing under one id each lack the other’s runs. A replacement host keeps its node’s history with no liveness service.

Revisit: two disks under one node id overwriting one run state.

lost-commit-round — A lost commit round counts only when no newer generation landed

In lockstep each commit round has one winner, so a fixed bound of 5 rounds exhausts every writer past the fifth.

Revisit: deployments whose concurrent writers exceed 64.

cold-store-boots — A cold store boots on metadata and fetches parts on first read

Cited bystore.fold.cached-inputsstore.fold.cached-inputsUnder lazy-parts, a fold fetches input parts through part-cache; inputs exceeding the cache raise FoldInputsExceedCache, naming both byte counts, and publish nothing.store.pull.lazy-partsstore.pull.lazy-partsWith [sync] lazy_parts = true, pull skips table Parquet data parts, which the store fetches whole on first read or fold use and verifies under digest-mismatch.store.pull.lazy-pointerstore.pull.lazy-pointerUnder lazy-parts, pointer-last waits for the named snapshot's _manifest.json alone; a part's digest comes from the bucket manifest when a read first fetches it.store.pull.part-cachestore.pull.part-cache[sync] part_cache_bytes, default 8 GiB, bounds locally fetched parts; above it, the cache deletes the least recently read part no statement or fold holds.read.cache.miss-waitread.cache.miss-waitUnder lazy-parts, each absent part has a 30 s acquisition deadline covering manifest lookup, GET attempts and backoff sleeps. Expiry raises ReadPartUnavailable, naming the part.read.cache.lazy-hydrationread.cache.lazy-hydrationUnder lazy-parts, hydration completes once the pull returns and the parts on disk are mapped, and a query's first read of an absent part takes the one fetch miss-wait bounds.

A container booting on an empty disk downloads every bucket object, and every deploy, eviction or scale-out repays it. The read face stays network-free: caller SQL reaches no network, the engine holds no bucket credential, and a read uses only parts verified against their digests. Faster object-store reads in the engine change none of these, so DuckDB 2.0’s asynchronous reads reopen nothing here.

Revisit: cold-table first reads past the wait bound under ordinary bucket latency.

bucket-collect — Collection reclaims a superseded snapshot’s parts and keeps its manifest

Cited bystore.push.manifest-formatstore.push.manifest-formatThe bucket manifest and every generation manifest carry format: 1 while they hold no collected record, 2 while they hold one; a manifest without format reads as 1.store.push.collectedstore.push.collectedA push uploads no key a standing collected record names in the manifest it reads, and collected keeps that key out of its commit; each commit round first uploads each key of its plan its manifest neither lists nor records.store.collect.supersededstore.collect.supersededCollect records as collected every object of a bucket chain snapshot's directory once a child created retain_runs ago supersedes it and no unexpired hold names it, save _manifest.json and each part or segment a retained snapshot names.store.collect.manifest-keptstore.collect.manifest-keptCollect deletes no snapshot _manifest.json and lists _collected.json beside each snapshot it reclaims, the only writer of that marker, so every chain reaches its root and each includes_runs still names its folded runs.store.collect.folded-runsstore.collect.folded-runsCollect records as collected every object of a run directory, _manifest.json included, once the bucket chain snapshot naming the run in includes_runs supersedes a snapshot superseded reclaims, save a _group.json group-window keeps.store.collect.retain-generationsstore.collect.retain-generations[sync] retain_generations, default 64 generations, sets how many newest generation manifests a collect keeps, the current one included; a value of 0 fails the configuration's parse.store.collect.drop-firststore.collect.drop-firstCollect commits a bucket manifest recording each key it unlists as collected, by the conditional replace of manifest-commit, before deleting any object, so no push lists a deleted object and a lazy read meets part-collected.store.merge.collectedstore.merge.collectedA merge drops from its entries each key a collected record names, a local copy included, and carries the record, unsigned, until it reaches the age tombstone-ttl sets.

A reader counts a run as folded only by its key in some chain snapshot’s includes_runs, so a chain missing a folding manifest reads a returning copy of that run twice. Collect therefore keeps every snapshot’s _manifest.json and reclaims its parts, sidecars and the run directories it folds. The bucket pointer’s chain changes only at its head: publish drops a link only younger than the pass that drops it, so a link older than the window never moves. Collects serialize on a bucket lease, and a synced hold reaches the bucket before its receipt, so no collect records a held build; one recorded anyway keeps its objects. A record precedes deletion, which waits for retained generations, and lapses with its object; marker and chain then keep the key out.

Revisit: a chain walk dominates a cold read, or kept manifests a bucket’s bytes.

folded-index — A snapshot carries every run its chain folds

Cited bystore.lay-out.folded-indexstore.lay-out.folded-indexA snapshot manifest's folded_runs names Parquet segments whose rows list each run its chain folds, by run key and folding snapshot with commit instant, _commit_seq, batch_digest and row count, and through, the snapshot they describe.store.lay-out.folded-index-validstore.lay-out.folded-index-validA reader trusts folded_runs only where it decodes, through names its snapshot and every segment reads at its recorded digest and row count, and otherwise walks, so the index never names fewer runs than the walk.store.lay-out.folded-index-walkstore.lay-out.folded-index-walkAn unbounded read reads the pointer's snapshot and its parents down to the first valid index; a bounded read reads on to the first snapshot created at or before its bound.store.fold.folded-indexstore.fold.folded-indexA pass writes folded_runs over its parent's valid index, carried by reference with the runs walked above it, or over a walk reaching a root, merging 4 segments of one size tier into one; otherwise it writes none.

Kept manifests make a chain whole, and a read walking it grows with history. Each pass writes the set of folded runs as Parquet segments beside its snapshot, carrying its parent’s by reference and merging 4 segments of one tier into one, so a pass writes few rows and a reader reads one snapshot. A reader trusts the set only where it names its own snapshot and every segment matches its digest, and otherwise walks, so a stale or damaged set costs time, never rows. A pass whose parent’s chain holds no valid set and ends at a missing manifest writes none. The set raises only the minor, 1.1 on a self-contained snapshot and 2.1 on one carrying parts by reference, so an older reader of the same major ignores it and walks the kept manifests. A collect by such a reader deletes the segments, and reads walk until the next pass writes the set again.

Revisit: reading a snapshot’s segments dominates a cold read.

run-digest — A folded run replays by the digest of its landing

Cited bystore.lay-out.run-digeststore.lay-out.run-digestA landing's run manifest records batch_digest, the SHA-256 of its batches' producer rows and fixed column types after writer rules, and of its position; injected columns and the commit instant stay outside it.store.lay-out.run-folded-replaystore.lay-out.run-folded-replayRe-landing a run whose manifest is gone and whose key the table's folded runs name, with an equal batch_digest, writes nothing and returns a replay carrying the folded-runs row in place of a manifest.store.lay-out.folded-committedstore.lay-out.folded-committedStore::committed_rows answers a run whose manifest a collection removed from its folded-runs rows, summing their row counts; a run only a walked snapshot names, or one without a count, reads as uncommitted.

A collection removes a folded run’s manifest and parts, so a retry after a lost acknowledgement finds no manifest to compare its rows against, and a new landing reusing the key reads as folded and its rows vanish. Each landing records the SHA-256 of its producer rows, column types and position, and the folded-runs row carries it beside the run’s row count. A re-landing whose key the folded runs name replays on an equal digest and conflicts otherwise. A row without a digest, or a key only a walked snapshot names, conflicts, since nothing tells a retry from a reuse. A staged commit records none: its rows never sit in memory together. A receive attempt whose run row never closed settles with the folded row’s count.

Revisit: the per-landing index read dominates a high-rate one-shot landing path.

index-walk — Collect, push and a lazy pull read the head’s index instead of the husks

Cited bystore.collect.index-walkstore.collect.index-walkUnder the pointer's valid bucket index, collect's walk stops at a child within the cutoff, by clock and id, whose parent is collected; older folded runs come from the index, older collected snapshots from the listing.store.merge.collected-indexstore.merge.collected-indexA push finds the snapshot folding each candidate run of collected-directory in the pointer's valid bucket index, reading that snapshot's manifest for its parent; without a valid index it walks the chain.store.pull.husks-skippedstore.pull.husks-skippedUnder lazy-parts, pull skips each _manifest.json beside a listed _collected.json where the pointer's snapshot, once home, holds a valid folded-runs index and an ancestry reaching the local pointer, listing it in .skipped-runs.json.store.pull.husks-fetchedstore.pull.husks-fetchedA full chain walk, or one meeting no valid index, fetches the table's skipped collected snapshot manifests through the part source at the first it reaches; local retention and a walk past a valid index end there.store.pull.husks-stalenessstore.pull.husks-stalenessA model's staleness check reads its head snapshot, each input's snapshots from the pointer down to the watermark snapshot, and the runs folded beneath through folded-index-walk, fetching a skipped manifest only on that path.

Every collected snapshot keeps its manifest, so a walk from the pointer reads one manifest per snapshot the table ever held, and a cold pull downloads them all. The head’s folded-runs index already names every run the chain folds, and the snapshot folding it. A lazy pull leaves each husk’s manifest in the bucket once the head’s index reads valid at home and its ancestry reaches the local pointer, which adopting the head otherwise walks. A walk fetches the husks only where no valid index answers, and local retention never does. Collect walks the retained snapshots down to the first collected parent, takes older runs from the index, the oldest folding snapshot reading as the root, and revisits every older husk from the listing. A push reads the manifest of the snapshot folding each candidate. A table holding a hold record walks whole.

Revisit: a full walk on a lazy store runs often enough that fetching husks dominates its cost.

unpushed-run-marked — An unpushed run is marked in the execution database

Cited bystore.drain.markstore.drain.markUnder postgres or cloudflare-d1, opening a run marks its row unpushed with the node before landing commits; closing clears the mark when no table the push includes holds a manifest of the run.store.drain.before-readstore.drain.before-readUnder these profiles, before-run pushes after pulling for unlost marks on its node with a local manifest or expired owner lease. A failed push names those runs and stops the command before its first read.

On a disposable disk a run commits locally before its push, so a failed push leaves the run on a disk the next release discards. The execution database is already durable.

folded-runs-on-demand — A lazy store fetches folded run manifests when a reader lists runs

Cited bystore.pull.lazy-readystore.pull.lazy-readyUnder lazy-parts, before-run waits for every in-scope object except data parts and the manifests lazy-folded-runs skips.store.pull.lazy-folded-runsstore.pull.lazy-folded-runsUnder lazy-parts, pull skips each other node's run _manifest.json that the pointer's snapshot chain folds, keeping its _group.json, and lists the skipped keys in the store's .skipped-runs.json.store.pull.folded-runs-fetchedstore.pull.folded-runs-fetchedListing a table's committed runs, or an as_of read resolving before the pointer's snapshot, fetches each skipped manifest through the part source within miss-wait's deadline; an unbounded read fetches none.store.pull.folded-run-collectedstore.pull.folded-run-collectedA skipped manifest the part source cannot bring home refuses the read as part-collected does, save where the source answers it collected: an unbounded listing leaves it out, and a bounded read does when a kept snapshot folds its run.store.pull.folded-runs-unreachablestore.pull.folded-runs-unreachableA store listing skipped manifests and opened without a part source raises SyncFoldedRunsUnfetched wherever folded-runs-fetched or husks-fetched fetches, naming the table and a manifest.

Within retain_runs, every run the pointer’s snapshot folds keeps its _manifest.json in the bucket, so a cold pull grows with retained runs rather than with tables. An unbounded read never resolves a folded run; an as_of read before that snapshot, and every reader listing a table’s runs, does.

Revisit: a reader whose first listing of a cold table passes the miss-wait bound under ordinary bucket latency.