Docs / Guides

contract store · 10-store.md · 11-snapshot.md · 12-time.md · 13-replica.md · 14-drain.md · 15-endpoint.md · 16-collect.md · 17-merge.md · 18-pull.md · guide · card

The store and its sync

The problem the store solves

Writers append rows while readers query and a background pass compacts. A reader needs a consistent answer, and a crashed writer must leave nothing half visible. The store holds Parquet files, JSON manifests and one pointer object per table (componentsstore.lay-out.componentsA store holds Parquet table data, JSON run and snapshot manifests, one pointer object per table, landed blobs, and SQLite catalogs derived.sqlite and machine.sqlite. Parquet, manifests and pointers are canonical.). Every commit is a conditional write, so a batch is either whole or absent (primitivetopology.coordinate.primitiveCoordination rests on one property: a linearizable conditional write. No component depends on a stronger one, and the tree ships no consensus implementation or external coordination service.).

Setting up a project

contextful init writes contextful.toml and creates the store root beside it (declaration-filestore.init.declaration-filecontextful init <name> writes [project] name = "<name>" into the working directory's contextful.toml, appending to a file declaring no project key and keeping its bytes, and creates the store root store-root beside it.). The store receives an identity that clones keep, so two different stores do not merge by accident (store-identitystore.init.store-identityAn initialized store holds one synced, canonical version-4 UUID in store-id, which repeated init and clones keep and a recreated root replaces.).

One table, from write to read

The example table documents declares primary_key = ["document_id", "page"] and order_by = "revised_at". A keyed table reads through a view that keeps one survivor per key (dedup-viewstore.declare.dedup-viewA keyed table reads through ROW_NUMBER() OVER (PARTITION BY <pk> ORDER BY <order_by> DESC, _ingested_at DESC, _run_id DESC, _row_seq DESC) = 1 over the current snapshot, if any, unioned with the committed runs it omits.); an unkeyed table reads as the union of its runs (unkeyed-unionstore.declare.unkeyed-unionA table declaring no primary_key reads as the byte-identical union of its visible committed runs, under replace-frontier when replacing.).

A run writes parts into its own directory and commits by creating that run’s manifest conditionally (run-manifeststore.lay-out.run-manifestA run commits by conditionally creating its node directory's _manifest.json, a lay_out::RunManifest whose node_id equals the directory's node segment.). Until the manifest exists, no read lists the run’s parts (uncommitted-runstore.lay-out.uncommitted-runA node directory without _manifest.json holds an uncommitted run, whose parts file lists exclude; a leased pipeline's run also waits for its commit-log entry.).

The engine injects provenance columns such as _ingested_at and _run_id, replacing any value the producer sent (injectedstore.reserve.injectedThe engine injects non-null _ingested_at as Parquet TIMESTAMP(UTC, NANOS), _run_id, _site_id, int32 _batch_seq within a batch scope, and _authored_by for writes an authenticated subject authorized, replacing producer values.). It also assigns _commit_seq above every value the table holds, so later commits sort later (commit-seqstore.reserve.commit-seqThe engine injects non-null int64 _commit_seq above every value the table holds, so a run committing after a read carries a value above every returned row, whatever its _ingested_at.).

Schemas grow additively: an unseen column joins, and older files read it as null (additivestore.reconcile.additiveAn unseen column joins the merged schema, and files written before it read it as null.). One batch sends page as a float. That batch fails before any Parquet, because a key column takes no float promotion (key-wideningstore.reconcile.key-wideningPrimary-key columns take no Float64 promotion: a widening batch raises StoreKeyWidened before any Parquet, and so does a fold meeting a key already reconciled to Float64.).

A snapshot is a compacted, immutable set of a table’s parts, named by a sortable id (snapshot-idstore.lay-out.snapshot-idA snapshot id is snapshot- and a nanosecond value zero-padded to 20 chars: the greater of the commit instant and the previous id plus one.). A query with no time bound reads the current snapshot plus the committed runs it omits (unbounded-lateststore.bound-time.unbounded-latestA read carrying neither bound resolves each table to its current snapshot plus the committed runs it omits.). Every read passes an explicit sorted file list, not a glob (explicit-file-liststore.reconcile.explicit-file-listEvery read passes read_parquet an explicit sorted file list of the parts the pointer's manifests name, with no glob, so stray files and indexes/ stay out.).

Leases and a paused writer

A lease gives one node exclusive ownership of a pipeline or of compaction, and a fence is the number that rises with each new holder. A paused machine that lost its lease later tries to commit. The higher fence wins the condition, and the stale run stays unreadable (stale-fencestore.lease.stale-fenceIf a commit-log create, pointer replacement or catalog UPDATE loses its condition to a higher commit fence, the operation raises LeaseFenced. The associated run or snapshot stays unreadable.).

Compaction

Runs accumulate until a fold compacts them. A pass stages Parquet and every declared sidecar, an index file beside the snapshot, then commits (passstore.fold.passA pass selects committed runs omitted by the current snapshot, deduplicates by key or unions, reconciles schemas, sorts by cluster_by, then partitions. The pass stages Parquet and every declared sidecar, then commits through pointer-commit.). Each table declares its sidecars under indexes (declarationstore.index.declarationEach table declares sidecars under indexes as IndexBlock. A vector sidecar requires model and dim, and a key outside its kind refuses the declaration.).

The commit replaces the table pointer, which names the current snapshot, on the ETag the pass read (table-pointerstore.lay-out.table-pointertables/<t>/_pointer.json names the table's current snapshot and commit fence; readers reach only snapshots reachable from it directly or through parent links.). A reader sees a snapshot and its sidecars together or neither (partial-snapshotstore.fold.partial-snapshotA reader observes a snapshot and every declared sidecar together or neither; a commit exposing one without the other raises StorePartialSnapshot.).

A statement already running keeps the snapshot it started on, so a fold blocks no reader (non-blockingstore.fold.non-blockingA statement running during a pass reads the snapshot the pointer named when it started; statements starting after the commit read the new one.).

A fold job puts compaction on a schedule, which pipeline serve keeps (scheduled-maintenancesurface.fire.scheduled-maintenanceA scheduled fold or sync-push job dispatches contextful job fire <name> through the cadence lease and pool under job:<name>; its next fire follows next-fire-from-history from its dispatch record.); each fire folds the job’s target table (job-passstore.fold.job-passcontextful job fire on a fold job runs a pass over its target, or over every table the store holds when it names none, reporting each table under result and exiting non-zero once any table fails.).

A sidecar is a vector graph, a full-text index or a key lookup (kindsstore.index.kindsFive index kinds exist: footer zone maps, sorted-Parquet sparse-map primary-key lookups, HNSW vector graphs, positional full-text indexes and opt-in per-column bloom filters.). A sidecar returns candidate ids only, and retrieval joins them through the access-controlled relation (candidate-idsstore.index.candidate-idsA sidecar returns only candidate id_column values, which retrieval joins on that column through the relation access controls protect before finalizing top-K.).

Three clocks

Transaction time records when a commit happened, valid time when a fact held, and availability time when the source released it. The diagrams follow one quarterly measurement through all three.

An auditor reads documents as of last week and gets the snapshot current then, so a later fold changes no answer (as-ofstore.bound-time.as-ofInside each table's FROM-source, as_of selects the newest reachable snapshot created at or before the bound plus the committed runs at or before it that the snapshot omits.). An instant older than retained history fails and names the oldest answerable instant (as-of-unretainedstore.bound-time.as-of-unretainedAn as_of before the oldest snapshot retention kept, or selecting a snapshot collected-snapshot counts or whose successors fold a run absent here, raises StoreAsOfUnretained, naming the oldest answerable instant.).

A table declaring available_time keeps every revision as a vintage beside the earlier value (vintage-rowsstore.available-time.vintage-rowsA keyed table declaring available_time is a vintage table, keeping every vintage per primary key, valid-time from and availability instant; a read returns the latest at or before any available_as_of, or under vintage = "first" the earliest.).

A document capture commits as one store snapshot that an evaluation case names by id (capturestore.snapshot.capture[pipeline.tables.snapshot] declares SnapshotDecl, the columns describing each captured document. Each capture commits as one store snapshot, and an evaluation case pins it by snapshot id.).

When a project binds an encryption key, Parquet files use modular encryption and sidecars use authenticated per-file keys (cipherstore.encrypt.cipherParquet modular encryption encrypts Parquet files, footers included; every sidecar and ledger file uses AES-256-GCM under a per-file data key the project key wraps.).

Sync to a bucket

A bucket mirrors the store tree. The [sync] endpoint scheme selects a filesystem, S3 or R2 bucket (schemesstore.endpoint.schemes[sync] endpoint maps file://<directory> to a filesystem bucket and s3://<region> to regional AWS S3. r2://<account-id> selects that account's R2 endpoint at region auto. https://<host> uses [sync] region, default us-east-1.). Before trusting conditional writes, a probe writes a sentinel object and tests its conditions (sentinelstore.probe.sentinelThe probe creates a sentinel under _contextful/cas-probe/, demonstrates cas when a second create and a stale If-Match fail while a current If-Match replaces it, then deletes it.).

contextful sync manifest --emit prints the plan a push commits, without touching the bucket (planstore.emit.plancontextful sync manifest --emit prints the JSON bucket manifest a push commits, each key with sha256, size and owner, reading and writing no bucket object.). A push uploads files the bucket lacks and commits the bucket manifest under If-Match (manifest-commitstore.push.manifest-commitA push commits by replacing <prefix>/manifest.json under If-Match on the ETag it read; this bucket manifest lists each key's sha256, size and owner.).

When two writers race, the push merges manifests and retries within a bound (retriesstore.merge.retries[sync] push_retries, default 5 attempts, bounds consecutive re-commit rounds whose re-read manifest carries no newer generation than the round before.). Deletions travel as signed tombstones (tombstone-signedstore.merge.tombstone-signedPush signs each Tombstone with the node's issuer key over the key it names, its owner and its deletion instant. A node without an issuer key writes unsigned tombstones.).

contextful sync collect removes tombstoned objects, superseded parts and folded runs but keeps every _manifest.json, so a returning copy of a run reads once (manifest-keptstore.collect.manifest-keptCollect deletes no snapshot _manifest.json and lists _collected.json beside each snapshot it reclaims, the only writer of that marker, so every chain reaches its root and each includes_runs still names its folded runs.). Each key is recorded first, so no push lists it again (drop-firststore.collect.drop-firstCollect commits a bucket manifest recording each key it unlists as collected, by the conditional replace of manifest-commit, before deleting any object, so no push lists a deleted object and a lazy read meets part-collected.).

A pull checks each downloaded object against its manifest digest (digest-mismatchstore.pull.digest-mismatchA downloaded object whose digest differs from its manifest entry is discarded and raises SyncObjectDigestMismatch; run states follow run-state-torn instead.). Before downloading anything, a pull refuses a store holding a different identity (identity-conflictstore.pull.identity-conflictA pull into a store holding a different store-identity raises StoreIdentityConflict before downloading files.).

A laptop replica is a read-only copy that records each snapshot it holds (descriptorstore.replicate.descriptorA replica's pull writes replica.json beside derived.sqlite, naming per table every held snapshot whose parts and sidecar paths exist on disk; push excludes the file.). It refreshes from the bucket and refuses writes (write-refusedstore.replicate.write-refusedA write verb against a replica raises ReplicaWriteRefused, naming the canonical store.). A query needing a sidecar the replica lacks fails and names the refresh that supplies it (missing-indexstore.replicate.missing-indexA query needing a sidecar or partition the replica lacks raises ReplicaMissingIndex, naming the refresh supplying it.).

Under a database profile, each run carries an unpushed mark until a push clears it (markstore.drain.markUnder postgres or cloudflare-d1, opening a run marks its row unpushed with the node before landing commits; closing clears the mark when no table the push includes holds a manifest of the run.). If the push at shutdown fails, the marks stay and name the runs still unpushed (unfinishedstore.drain.unfinishedA shutdown push failing or exceeding deadline keeps the run marks and names the runs on stderr, and pipeline serve exits non-zero.).