Docs / Specification

contract store · 10-store.md · 11-snapshot.md · 12-time.md · 13-replica.md · 14-drain.md · 15-endpoint.md · 16-collect.md · 17-merge.md · 18-pull.md · guide · card

Bucket endpoints and leases

The [sync] endpoint decides how each request reaches the bucket: its scheme, its region and the credentials that sign it. The adapter speaks the S3 object API to AWS S3, R2 and compatible services, and reads a filesystem bucket directly.

endpoint

Bucket endpoint schemes, transport, request addressing and signing, credentials and backend response handling.

• schemestest[sync] endpoint maps file://<directory> to a filesystem bucket and s3://<region> to regional AWS S3. r2://<account-id> selects that account’s R2 endpoint at region auto. https://<host> uses [sync] region, default us-east-1.
• unsupported-schemetestAny other scheme, or an S3 or R2 endpoint in a build without the s3-sync feature, raises SyncEndpointUnsupported, naming the endpoint.
  Why

  because an unreachable bucket refuses when it opens, before a push uploads anything
• plaintexttestAn http:// endpoint opens on a loopback host alone; any other host raises SyncEndpointInsecure, naming it.
  Why

  because plaintext objects and signed requests are readable and replayable by anyone on the path
• addressingtestS3 and R2 address <endpoint>/<bucket>/<key> path-style, percent-encoding each key segment, and sign every request with AWS Signature Version 4 for service s3 at the endpoint’s region.
• credentialstest[sync] access_key_id, secret_access_key and the optional session_token each bind secret://<name>, hydrated as whole-value-referenceconnector.reference.whole-value-referenceA whole-value secret://<name> reference outside templates, such as a [sync] credential key, resolves through the assembled adapter chain, including process environment, under first-hit-wins and shadowed-name., or env://NAME, read whole from the process environment, as the bucket opens.
• credential-unboundtestAn S3 or R2 [sync] omitting access_key_id or secret_access_key, supplying a non-reference credential, or naming an unset variable raises SyncCredentialUnbound, naming the key.
  Why

  because a literal key in config.toml sits in plaintext on every disk and backup
• role-credentialstest[sync] credentials = "role" on an s3:// or https:// endpoint signs with workload-role credentials in place of both keys, from the ECS container credential endpoint, then STS web identity, then IMDSv2.
  Why

  because a workload role supplies short-lived keys in place of long-lived stored ones
• role-unavailabletestUnder role-credentialsstore.endpoint.role-credentials[sync] credentials = "role" on an s3:// or https:// endpoint signs with workload-role credentials in place of both keys, from the ECS container credential endpoint, then STS web identity, then IMDSv2., a bucket open that no role source answers raises SyncRoleUnavailable, naming each source tried.
  Why

  because an unsigned request otherwise fails as forbidden, hiding the missing role
• role-conflicttestcredentials = "role" beside a key credential credentialsstore.endpoint.credentials[sync] access_key_id, secret_access_key and the optional session_token each bind secret://<name>, hydrated as whole-value-reference, or env://NAME, read whole from the process environment, as the bucket opens. binds, or on an R2 endpoint, raises SyncCredentialConflict, naming the endpoint.
  Why

  because two credential sources for one bucket leave unstated which identity signed a write
• role-refreshtestRole credentials refresh 300 s before their stated expiry and never sign once expired; a failed refresh keeps still-valid credentials and retries at the next request.
• conditional-answerstestA 412 or 409 means a failed condition; a 404 naming NoSuchKey does so for an If-Match put. 501 means an unsupported method; 403 means forbidden credentials. Both follow inconclusivestore.probe.inconclusiveAn unsupported-method response, a forbidden response or a transport error raises SyncProbeInconclusive and counts as capability not demonstrated..
• missing-buckettestA 404 naming a code other than NoSuchKey, NoSuchBucket included, is neither an absent object nor a failed condition: the request fails as transport, naming the code.
  Why

  because a missing bucket otherwise reads as empty, a free lease or a lost compare-and-set
• transporttestAn incomplete S3 get, head, list page, delete or unconditional put retries within 3 attempts. Connections allow 5 s; responses allow 2 s, 4 s, then 10 s. Conditional puts attempt once.
  Why

  because a resent conditional put whose answer was lost reads its own write as a failed condition
• throttledtestS3 requests retry within 5 attempts after 429 or 503, or 500 except on conditional puts. Each retry waits a uniform draw below a ceiling doubling from 100 ms to 5 s.
  Why

  because a throttled request never applied, and jitter spreads throttled clients
• unavailabletestA request unanswered within transportstore.endpoint.transportAn incomplete S3 get, head, list page, delete or unconditional put retries within 3 attempts. Connections allow 5 s; responses allow 2 s, 4 s, then 10 s. Conditional puts attempt once., or answered 429 or 5xx past throttledstore.endpoint.throttledS3 requests retry within 5 attempts after 429 or 503, or 500 except on conditional puts. Each retry waits a uniform draw below a ceiling doubling from 100 ms to 5 s., fails as unavailable, naming the request and claiming nothing about the bucket’s content.
• list-pagestestA list follows continuation tokens until the backend reports completion, returning every key under the prefix in sorted order.

probe

Testing conditional writes with a sentinel object and selecting the backend’s coordination mode.

• inconclusivetestAn unsupported-method response, a forbidden response or a transport error raises SyncProbeInconclusive and counts as capability not demonstrated.
  Why

  A-store#coordination-probed-compare
• unproventestDeclaring cas against a backend the probe did not demonstrate raises SyncCoordinationUnproven and stops the push.
  Why

  A-store#coordination-probed-compare
• sentineltestThe probe creates a sentinel under _contextful/cas-probe/, demonstrates cas when a second create and a stale If-Match fail while a current If-Match replaces it, then deletes it.
• network-volumetestA filesystem bucket on a mount type outside apfs, hfs, ext4, xfs, btrfs, zfs, tmpfs and overlay resolves single-writer without a sentinel, naming the mount type; a declared cas there meets unprovenstore.probe.unprovenDeclaring cas against a backend the probe did not demonstrate raises SyncCoordinationUnproven and stops the push..
  Why

  because a sentinel run from one client of a network share cannot observe another client

lease

Exclusive pipeline and compaction writer leases, their commit fences, and cursor commits in the commit log.

• ttltestA writer lease grants ownership for 10 min.
• renewaltesttestThe process holding a compaction lease renews it every 200 s by If-Match replace on its ETag; sync compact renews while it pulls, folds and pushes.
• operator-renewtestsync lease renew <table> renews the compaction lease this node holds, for sync compact --held.
• clock-skewtestBucket leasing assumes the clocks of two machines differ by 30 s or less, and every expiry judgment reads the judging machine’s monotonic clock.
  Why

  because correctness rests on the commit fence, and the skew bound only sets how early a holder stops committing
• heldtestIf a run finds an unexpired lease, the engine raises LeaseHeld, names the holder and expiry, and skips the run. The engine retries the run at the next reconciliation tick.
  Why

  A-store#coordination-probed-compare
• stale-fencetesttesttheoremIf a commit-log create, pointer replacement or catalog UPDATE loses its condition to a higher commit fence, the operation raises LeaseFenced. The associated run or snapshot stays unreadable.
  Why

  A-store#coordination-probed-compare
• not-heldtestReleasing a lease another node holds, or one a later acquisition of the same node fenced past, raises LeaseNotHeld and leaves the object untouched.
  Why

  A-store#coordination-probed-compare
• cursor-kindtestA cursor whose kind takes no lease reaching cursors/ raises LeaseCursorKindMismatch.
  Why

  A-run#incremental-position-bound
• local-nodetestA bucket lease attempted under the node id local raises LeaseNodeIdLocal, logging the variable that sets a node id; that machine keeps the machine lease.
  Why

  P3
• acquiretestAcquisition creates the lease object under If-None-Match or replaces an expired or released object under If-Match on its ETag. The acquired commit fence is one above the object’s commit fence.
  Why

  A-store#coordination-probed-compare
• pointer-fencetesttesttheoremTaking a table’s compaction lease raises the fence stored in its bucket pointer, so a publish carrying a lower fence loses its condition.
  Why

  A-store#coordination-probed-compare
• network-volumetestA compaction lease on a bucket resolving single-writer under network-volumestore.probe.network-volumeA filesystem bucket on a mount type outside apfs, hfs, ext4, xfs, btrfs, zfs, tmpfs and overlay resolves single-writer without a sentinel, naming the mount type; a declared cas there meets unproven. meets unprovenstore.probe.unprovenDeclaring cas against a backend the probe did not demonstrate raises SyncCoordinationUnproven and stops the push., naming the mount type. Acquisition writes no lease object or pointer fence.
  Why

  because a conditional put holding on one machine only lets two machines each take the lease
• commit-logtestUnder a machine lease, a run commits by creating the next cursors/<pipeline-id>/<node-id>/<seq>.json. Acquisition creates an entry carrying its fence. A manifest marked logged becomes readable when that log records the manifest.
  Why

  A-store#coordination-probed-compare

unsettled: Which process carries out renewal for a pipeline lease? owner: store affects: store.lease