Contextfulv0.23.0

Docs / Specification

contract store · 10-store.md · 11-snapshot.md · 12-time.md · 13-replica.md · 14-drain.md · 15-endpoint.md · 16-collect.md · 17-merge.md · 18-pull.md · guide · card

The store and its sync — card

Generated by contextful-spec state from the contract’s files and spec/terms/store.toml; not authored.

Operations

OperationCoversClausesProved
lay-outStore directories, run and snapshot manifests, table pointers, local catalogs and node identifiers.410
initProject initialization, and project discovery when a command omits --project.100
declareTable declarations, primary keys, row ordering, write modes and replacement behavior.130
reserveReserved column and table names, injected columns, _commit_seq assignment and request ledgers.230
reconcileSchema reconciliation across table files, type promotion, added columns and scan behavior.190
foldCompaction passes, their triggers and retention, and snapshot publication through a pointer update.336
indexSidecar indexes, bloom filters, row clustering and partitions.330
encryptAt-rest encryption of Parquet, sidecars, ledgers, metadata, machine catalogs and node run-state files.80
emitPush planning from local files without reading or writing the bucket.20
pushUploading local files, constraining bucket prefixes and committing the bucket manifest.340
snapshotDocument capture records, snapshot identifiers for evaluation cases and access to documents captured after the reading horizon.20
bound-timeTransaction-time and valid-time columns, their read parameters and the snapshots or rows each parameter selects.150
available-timeThe source availability timestamp, its value at landing and row selection under available_as_of.120
replicateReplica refreshes, required snapshot files, local snapshot records and refused reads or writes.50
drainTracking unpushed runs, retrying pushes before reads, pushing during shutdown and reporting status to release checks.110
endpointBucket endpoint schemes, transport, request addressing and signing, credentials and backend response handling.160
probeTesting conditional writes with a sentinel object and selecting the backend’s coordination mode.40
leaseExclusive pipeline and compaction writer leases, their commit fences, and cursor commits in the commit log.132
collectUnlisting and deleting the bucket objects no reader needs, and the generation manifests past the retained count.210
mergeMerging bucket manifests across writers, verifying tombstones and retrying conditional manifest commits.165
pullDownloading bucket files, verifying digests, advancing table pointers and restoring named generations.540

Error cases

ErrorWhenClause
StoreUnknownTableA table name resolves to no schema.json in the tree.store.lay-out.unknown-table
StoreRunConflictA run id already committed on its node, or folded under its run key, lands again as other than its replay.store.lay-out.run-conflict
StoreManifestFormatNewerA run or snapshot manifest carries a format_version major newer than the build reads.store.lay-out.format-newer
StoreManifestUnreadableA run manifest or reachable snapshot manifest fails to parse.store.lay-out.manifest-unreadable
StoreNodeIdInvalidA resolved node id is too long or outside the path-safe pattern.store.lay-out.node-id-shape
StoreNodeIdSharedA node id is declared in a control-plane configuration applied to many machines.store.lay-out.node-id-shared
StoreIdentityInvalidA store UUID is malformed or owner mint names no initialized root.store.init.identity-invalid
StoreProjectConflictAn init meets a contextful.toml declaring another project, or a project with no name.store.init.name-conflict
StoreProjectNameInvalidA project name is not path-safe /-separated segments.store.init.name-conflict
StoreProjectUndiscoveredA command given no --project finds no contextful.toml naming a project from its working directory up.store.init.undiscovered
StoreRetentionColumnInvalidA row-retention column or age is invalid, or a landed value is null.store.declare.retain-rows
StoreKeyUnknownColumnA primary_key names a column neither declared nor injected.store.declare.key-unknown
StoreNestedKeyColumnA struct, list or map column takes a key, ordering, clustering, partition or valid-time role.store.declare.key-unknown
StoreOrderByUnknownColumnAn order_by names a column neither declared nor injected.store.declare.key-unknown
StoreReservedColumnNameA producer column is spelled inside the reserved namespace and outside the optional set.store.reserve.column-name
StoreReservedTableNameA declared table collides with a reserved table namespace.store.reserve.table-name
StoreCommitLockTimeoutA commit outwaited its table’s commit lock or a lower value’s publication.store.reserve.commit-wait
StoreSchemaIncompatibleTwo observed types for one column have no supertype in the lattice.store.reconcile.incompatible
StoreKeyWidenedA primary-key column takes the float promotion.store.reconcile.key-widening
FoldInputsExceedCacheA fold’s input parts exceed the part cache bound.store.fold.cached-inputs
StorePartialSnapshotA commit would expose snapshot data without its declared sidecars, or the reverse.store.fold.partial-snapshot
StoreIndexIdColumnUnresolvedA table’s sidecars resolve no single id_column.store.index.id-column-unresolved
StoreIndexIdNotUniqueOne id_column value names two rows of a staged snapshot.store.index.id-unique
StoreIndexPathCollisionTwo sidecar declarations of one table resolve to one path.store.index.path-collision
StoreIndexColumnAbsentAn index is declared over a column the reconciled schema lacks.store.index.column-absent
StoreIndexColumnTypeAn indexed column or id_column carries a type the sidecar does not read.store.index.column-type
StorePartitionColumnTypeA partition_by column is typed binary or vector.store.index.partition-type
StoreEncryptionKeyUnboundA declared key source names a binding the process lacks.store.encrypt.key-unbound
StoreIndexOverRedactedColumnAn index is declared over a column redacted at write time.store.encrypt.redacted-index
SyncPrefixEscapeA key resolved outside the configured prefix.store.push.prefix-escape
SyncPrefixUnboundThe variable named by prefix_from is unset at startup.store.push.prefix-unbound
SyncPrefixOverspecifiedprefix and prefix_from are declared together.store.push.prefix-overspecified
SyncPushInFlightA push of one store waited past its bound for the push guard another push on the machine holds.store.push.in-flight
SyncManifestFormatUnsupportedA bucket or generation manifest carries a format newer than the reader parses.store.push.format-unsupported
SyncControlDivergedA signed control head does not descend from the bucket’s head.store.push.control-diverged
SyncGenerationConflictA generation manifest already holds another commit’s bytes under the number a push committed.store.push.generation-conflict
StoreValidTimeNotTimestampA declared valid-time column is not timestamp-typed.store.bound-time.valid-time-type
StoreAsOfUnretainedAn as_of precedes the oldest retained snapshot of a table whose history was collected.store.bound-time.as-of-unretained
StoreValidTimeUndeclaredA valid-time-bounded read names a table declaring no pair.store.bound-time.valid-time-undeclared
StoreAvailableTimeNullA landing row resolves to a null availability instant.store.available-time.undeclared
StoreAvailableTimeUndeclaredAn availability-bounded read names a table declaring no availability time.store.available-time.undeclared
StoreVintageHashUndeclaredA vintage table declares no content_hash_column.store.available-time.vintage-validation
StoreVintageRetainRowsA vintage table declares a row-age retention cutoff.store.available-time.vintage-validation
ReplicaWriteRefusedA write verb was invoked against a replica.store.replicate.write-refused
ReplicaPartialParquetA replica holds a strict subset of a snapshot’s data files.store.replicate.partial-parquet
ReplicaMissingIndexA query needs a sidecar or partition the replica lacks.store.replicate.missing-index
ReplicaSensitiveTableA refresh requested a replicate-off table.store.replicate.sensitive-refused
SyncRunsUnpushedA run opened on a disposable disk carries an unpushed mark no committed push has cleared.store.drain.status
SyncStatusUnreadA status check reached no answer from the execution database holding the unpushed marks.store.drain.status-unread
SyncEndpointUnsupportedAn endpoint’s scheme names no bucket adapter this build links.store.endpoint.unsupported-scheme
SyncEndpointInsecureA plain http:// endpoint names a host off loopback.store.endpoint.plaintext
SyncCredentialUnboundAn S3 or R2 endpoint’s [sync] binds a credential key to no reference, or to an unset variable.store.endpoint.credential-unbound
SyncRoleUnavailableNo workload-role source answers a bucket whose [sync] selects role credentials.store.endpoint.role-unavailable
SyncCredentialConflictRole credentials beside a key, or on an R2 endpoint.store.endpoint.role-conflict
SyncProbeInconclusiveThe conditional-write probe met an unsupported method, a forbidden response or a transport error.store.probe.inconclusive
SyncCoordinationUnprovencas is declared against a backend the probe did not demonstrate.store.probe.unproven
LeaseHeldA run or pass found an unexpired lease another node holds.store.lease.held
LeaseFencedA commit, pointer replace or catalog update losing its condition to a higher fence.store.lease.stale-fence
LeaseNotHeldA release was attempted against a lease another node holds.store.lease.not-held
LeaseCursorKindMismatchA cursor whose kind takes no lease reached cursors/.store.lease.cursor-kind
LeaseNodeIdLocalA bucket lease was attempted under the reserved node id local.store.lease.local-node
SyncTombstoneForeignA tombstone names an entry another node owns.store.merge.tombstone-owner
SyncTombstoneUnverifiedA tombstone is unsigned, fails its signature, or names a signer the key-set ledger does not verify.store.merge.tombstone-unverified
SyncManifestRebaseExhaustedThe bucket manifest commit lost its race past the retry bound or the round ceiling.store.merge.exhausted
SyncCursorConflictA cursor was resolved by recency instead of through its commit.store.merge.cursor-recency
SyncHeldRecordUnreadableThe local ownership record cannot be read or decoded.store.merge.held-unreadable
SyncObjectDigestMismatchA downloaded object’s digest differs from its entry.store.pull.digest-mismatch
StoreIdentityConflictA pull would replace one store UUID with another.store.pull.identity-conflict
SyncPullDidNotConvergeA pull exhausted its retries against keys moving beneath it.store.pull.unconverged
SyncLocalFoldUnrecoverableA local pointer off the bucket’s chain folds runs neither the store nor the bucket holds.store.pull.local-fold-unrecoverable
SyncPullTablesEmptyA store’s [sync] pull_tables holds no pattern.store.pull.tables-empty
SyncGenerationAbsentA pull names a generation the bucket holds no generation manifest for.store.pull.generation-absent
SyncGenerationDivergedA generation pull meets a local file the generation does not list.store.pull.generation-diverged
SyncFoldedRunsUnfetchedA store whose lazy pull skipped folded run manifests reads its runs without a part source.store.pull.folded-runs-unreachable
SyncPartCollectedA part a lazy store needs is gone from the bucket.store.pull.part-collected
SyncRelandUnneededA reland names a table whose local pointer the next pull keeps or replaces without a refusal.store.pull.reland-unneeded
SyncRelandHeadAbsentA reland finds the bucket pointer’s snapshot not whole on disk.store.pull.reland-head-absent

Bounds

BoundValueBasisClause
store-snapshot-id-width20 charschosenstore.lay-out.snapshot-id
store-snapshot-ancestors256 entrieschosenstore.lay-out.ancestors
store-run-id-length128 charschosenstore.lay-out.run-id-shape
store-node-id-length64 charschosenstore.lay-out.node-id-shape
store-commit-lock-wait30 schosenstore.reserve.commit-wait
store-ledger-retention365 dchosenstore.reserve.ledger-retention
store-fold-merge-fanout4 partschosenstore.fold.tiered-merge
store-run-retention7 dchosenstore.fold.retention
store-compaction-interval6 hchosenstore.fold.triggers
store-compaction-run-count50 runschosenstore.fold.triggers
store-graph-extensions16 extensionschosenstore.index.graph-extensions
store-partition-count1000 partitionschosenstore.index.partition-warnings
store-partition-median16 MiBchosenstore.index.partition-warnings
store-push-wait300 schosenstore.push.in-flight
store-drain-deadline15 schosenstore.drain.deadline
store-drain-status-runs20 runschosenstore.drain.status-summary
store-drain-lost-retention7 dchosenstore.drain.prune
store-endpoint-role-refresh300 schosenstore.endpoint.role-refresh
store-endpoint-answer10 schosenstore.endpoint.transport
store-endpoint-answer-first2 sstandard:aws-s3-performancestore.endpoint.transport
store-endpoint-answer-second4 sstandard:aws-s3-performancestore.endpoint.transport
store-endpoint-attempts3 attemptschosenstore.endpoint.transport
store-endpoint-connect5 schosenstore.endpoint.transport
store-endpoint-backoff-base100 mschosenstore.endpoint.throttled
store-endpoint-backoff-cap5 schosenstore.endpoint.throttled
store-endpoint-throttled5 attemptschosenstore.endpoint.throttled
store-lease-ttl10 minchosenstore.lease.ttl
store-lease-renewal200 schosenstore.lease.renewal
store-clock-skew30 schosenstore.lease.clock-skew
store-retain-generations64 generationschosenstore.collect.retain-generations
store-tombstone-ttl30 dchosenstore.merge.tombstone-ttl
store-push-retries5 attemptschosenstore.merge.retries
store-push-round-ceiling64 roundschosenstore.merge.round-ceiling
store-pull-convergence3 attemptschosenstore.pull.convergence
store-transfer-width16 requestschosenstore.pull.transfer-width
store-pull-serve-refresh30 schosenstore.pull.serve-refresh
store-part-cache8 GiBchosenstore.pull.part-cache

Types

Each struct or enum a {{type:<path>}} pointer names, with the keys serde reads and their doc comments.

contextful_core::store::declare::AvailableTime

The struct at crates/contextful-core/src/store/declare.rs, named by store.available-time.declaration.

KeyDoc
column
zoneThe zone a date-typed value resolves in; UTC when unset (store.available-time.live-since).
lagThe duration filling a null instant from valid time (store.available-time.lag-fill).
live_sinceThe instant the source began publishing live (store.available-time.live-since).
adjustmentThe factor table a bounded read multiplies listed columns by (store.available-time.adjustment-factors).

contextful_core::store::declare::SnapshotDecl

The struct at crates/contextful-core/src/store/declare.rs, named by store.snapshot.capture.

contextful_core::store::declare::TableDecl

The struct at crates/contextful-core/src/store/declare.rs, named by store.declare.table-block, store.declare.erasure-metadata, run.model.restricted-input, run.publish.disclosure-digest.

KeyDoc
name
primary_key
order_by
write_mode
replicate
subject_id
erasure_key
referenced_by
on_erase
class
policy
redaction
visibility
valid_time
available_timeThe availability instant an available_as_of read compares (store.available-time.declaration).
snapshotA snapshot corpus’s capture columns (store.snapshot.capture).
cluster_by
partition_by
bloom_filterColumns whose Parquet parts carry a bloom filter (store.index.kinds).
retain_runs
fold_max_unfolded_runsThe unfolded run count that fires the table’s pass in place of 50 (store.fold.declared-trigger); zero refuses the block.
fold_max_unfolded_bytesThe part bytes, summed over the unfolded runs’ manifests, that fire the table’s pass (store.fold.declared-bytes); zero refuses the block.
allowed_latenessThe window a monotonic poll of this table re-reads behind its stored position, spelled as a span (run.advance.allowed-lateness).
retain_rows
retain_versionsA derive output table keeping each task version’s rows current under that version (run.emit.version-retained).
columnsColumn types every landing reads (store.declare.table-block), spelled as [ColumnType::parse] reads them; an unreadable spelling refuses the block.
indexesSidecar declarations (store.index.declaration).
agent_description
agent_hint
column_hintsHints for named columns in a table description (read.register.describe-payload).
example_queries
content_hash_columnThe column carrying each row’s content hash, the row key a ranked read keeps one row per (read.retrieve.row-key-dedup).
result_cacheThe time to live of a cached read result touching the table, which opts the table into the result cache (read.cache.cache-is-opt-in), spelled as retain_runs.
privateA table whose rows never enter the result cache (read.cache.cache-is-opt-in).
retrieveWhat a ranked read scores the table’s rows on (read.retrieve.scored-text).

contextful_core::store::index::IndexBlock

The struct at crates/contextful-core/src/store/index.rs, named by store.index.declaration.

KeyDoc
kind
column
id_column
model
dim
metric
m
ef_construction
storage
tokenizer

contextful_core::store::index::VectorStorage

The enum at crates/contextful-core/src/store/index.rs, named by store.index.graph.

contextful_core::store::sync::Tombstone

The struct at crates/contextful-core/src/store/sync.rs, named by store.merge.tombstone-signed.

KeyDoc
owner
deleted_at
signerThe signing key’s public text, ed25519:<hex> or es256:<hex>; absent when the writing node holds no issuer key.
signatureHex of the signature over [Tombstone::message].

Decisions

SectionCited by
A-store#adjustment-factors-applystore.available-time.vintage-rows, store.available-time.vintage-dedup-key
A-store#availability-declared-columnstore.available-time.declaration, store.available-time.available-as-of
A-store#binary-fixed-sizestore.reconcile.incompatible, store.reconcile.binary-and-vector, store.reconcile.typed-landing, store.reconcile.stored-type, store.reconcile.half-width, read.respond.bytes-and-vectors, authority.mask.typed-strategy
A-store#bucket-collectstore.push.manifest-format, store.push.collected, store.collect.superseded, store.collect.manifest-kept, store.collect.folded-runs, store.collect.retain-generations, store.collect.drop-first, store.merge.collected
A-store#bucket-index-commitsstore.merge.tombstone-owner, store.merge.tombstone-signed, store.merge.tombstone-unverified, store.merge.retries, store.merge.round-ceiling, store.merge.exhausted, store.merge.scoped-union, store.merge.held-record, store.merge.held-absent, store.merge.ownership
A-store#canonical-metadata-sitsstore.encrypt.metadata-envelope, store.encrypt.run-state-envelope
A-store#cold-store-bootsstore.fold.cached-inputs, store.pull.lazy-parts, store.pull.lazy-pointer, store.pull.part-cache, read.cache.miss-wait, read.cache.lazy-hydration
A-store#column-records-commitread.horizon.late-columns
A-store#coordination-probed-comparetopology.coordinate.primitive, topology.coordinate.inventory, topology.coordinate.lease-row, topology.coordinate.catalog-clock, topology.coordinate.fence-advances, topology.coordinate.fenced-commit, topology.coordinate.cadence-lease-ttl, topology.coordinate.cadence-lease-renewal, topology.coordinate.cadence-fallback, topology.coordinate.weak-backend, store.fold.compaction-lease, store.fold.pointer-commit, store.push.pointer-fenced, store.probe.inconclusive, store.probe.unproven, store.lease.held, store.lease.stale-fence, store.lease.not-held, store.lease.acquire, store.lease.pointer-fence, store.lease.commit-log
A-store#empty-complete-snapshotrun.advance.snapshot-completion, run.advance.zero-row-commit
A-store#exhausted-pull-refusesstore.pull.unconverged-resume
A-store#folded-indexstore.lay-out.folded-index, store.lay-out.folded-index-valid, store.lay-out.folded-index-walk, store.fold.folded-index
A-store#folded-runs-on-demandstore.pull.lazy-ready, store.pull.lazy-folded-runs, store.pull.folded-runs-fetched, store.pull.folded-run-collected, store.pull.folded-runs-unreachable
A-store#index-walkstore.collect.index-walk, store.merge.collected-index, store.pull.husks-skipped, store.pull.husks-fetched, store.pull.husks-staleness
A-store#keyed-table-coveredrun.seed.compaction-cadence
A-store#lost-commit-round
A-store#machine-catalog-sealsstore.lay-out.machine-catalog, store.encrypt.machine-catalog-sealing
A-store#manifest-carries-formatstore.lay-out.table-pointer, store.lay-out.format-version, store.lay-out.format-newer, store.lay-out.format-migrate
A-store#pass-carries-untouchedstore.fold.carry-by-reference, store.fold.tiered-merge, store.fold.reference-retention
A-store#push-tombstones-ownedstore.push.manifest-commit
A-store#replica-advertises-descriptorstore.replicate.descriptor, store.replicate.missing-index
A-store#replica-read-wholestore.replicate.write-refused, store.replicate.partial-parquet, store.replicate.descriptor, store.replicate.missing-index, store.replicate.sensitive-refused, store.pull.pointer-last, store.pull.pointer-advance
A-store#restatement-vintage-rowstore.available-time.vintage-rows, store.available-time.vintage-dedup-key
A-store#run-digeststore.lay-out.run-digest, store.lay-out.run-folded-replay, store.lay-out.folded-committed
A-store#schema-lattice-hasstore.reconcile.lattice, store.reconcile.key-widening
A-store#sidecar-copies-tenantstore.index.filter-copies
A-store#sidecar-memory-mappedstore.encrypt.metadata-envelope, store.encrypt.run-state-envelope, store.encrypt.machine-catalog-sealing, store.encrypt.sidecar-reader, read.retrieve.sidecar-size-cap, read.retrieve.fulltext-sealed-cap
A-store#sidecar-names-idstore.declare.empty-replacement, store.declare.fold-job, store.index.id-column, store.index.id-column-unresolved, store.index.id-unique, store.index.bloom-parts, store.index.vector-by-fold, store.index.fulltext-by-fold
A-store#sidecar-s-builderstore.index.identity, store.index.rebuild, store.index.graph-format, store.index.graph-extensions, store.index.sidecar-segments
A-store#snapshot-corpus-declaredstore.snapshot.capture, store.snapshot.post-horizon-capture
A-store#sqlite-link-ruletopology.package.sqlite-adapter, topology.package.store-sqlite-free
A-store#store-s-catalogstopology.coordinate.cursor-cas, topology.coordinate.catalog-port, topology.coordinate.backends, store.lay-out.derived-catalog, store.lay-out.catalog-ports
A-store#struct-list-mapstore.reconcile.nested-lattice, store.reconcile.nested-landing, read.respond.nested-values, run.normalize.native-store
A-store#synced-uuid-bindsstore.init.store-identity, store.init.identity-invalid
A-store#tenant-value-bytesstore.index.tenant-verbatim
A-store#tombstone-signed-node
A-store#unpushed-run-markedstore.drain.mark, store.drain.before-read
A-store#validity-interval-rowstore.bound-time.interval-rows, store.available-time.vintage-dedup-key
A-store#variant-column-keepsstore.reconcile.variant
A-store#vintage-landing-readsstore.available-time.vintage-scope