Docs / Specification

contract store · 10-store.md · 11-snapshot.md · 12-time.md · 13-replica.md · 14-drain.md · 15-endpoint.md · 16-collect.md · 17-merge.md · 18-pull.md · guide · card

Bucket collection

A bucket keeps every object a push writes until a collect deletes it. contextful sync collect unlists, then deletes, the objects no reader needs: objects the bucket manifest no longer lists, a superseded snapshot’s parts and sidecars, the run directories an aged snapshot folds, and the generation manifests past a retained count. Every snapshot’s _manifest.json stays.

collect

Unlisting and deleting the bucket objects no reader needs, and the generation manifests past the retained count.

• commandtesttestcontextful sync collect prints JSON naming, per class unlisted, superseded, folded_runs and generations, the keys it records as collected, the objects it deletes, their bytes and unknown_size, then undeclared, shared_prefix and referenced; --dry-run prints the same and writes nothing.
  Why

  because an operator sizes a reclamation before running it
• unlistedtesttestCollect records as collected an object of the project the bucket manifest does not list when a tombstone or a generation past retain-generationsstore.collect.retain-generations[sync] retain_generations, default 64 generations, sets how many newest generation manifests a collect keeps, the current one included; a value of 0 fails the configuration's parse. names its key, save a _group.json.
  Why

  because a key some manifest once listed is one a push wrote, while an unnamed object may be an upload awaiting its commit
• supersededtestCollect records as collected every object of a bucket chain snapshot’s directory once a child created retain_runs ago supersedes it and no unexpired hold names it, save _manifest.json and each part or segment a retained snapshot names.
  Why

  A-store#bucket-collect
• manifest-kepttestCollect deletes no snapshot _manifest.json and lists _collected.json beside each snapshot it reclaims, the only writer of that marker, so every chain reaches its root and each includes_runs still names its folded runs.
  Why

  A-store#bucket-collect
• folded-runstestCollect records as collected every object of a run directory, _manifest.json included, once the bucket chain snapshot naming the run in includes_runs supersedes a snapshot supersededstore.collect.supersededCollect records as collected every object of a bucket chain snapshot's directory once a child created retain_runs ago supersedes it and no unexpired hold names it, save _manifest.json and each part or segment a retained snapshot names. reclaims, save a _group.json group-windowstore.collect.group-windowCollect keeps a folded run's _group.json while a member table holding a copy of the run has not folded it out under folded-runs, under that member's retain_runs. keeps.
  Why

  A-store#bucket-collect
• group-windowtestCollect keeps a folded run’s _group.json while a member table holding a copy of the run has not folded it out under folded-runsstore.collect.folded-runsCollect records as collected every object of a run directory, _manifest.json included, once the bucket chain snapshot naming the run in includes_runs supersedes a snapshot superseded reclaims, save a _group.json group-window keeps., under that member’s retain_runs.
  Why

  because a member run reads as committed only beside its group root’s marker, an as_of read inside the window included
• index-walktesttestUnder the pointer’s valid bucket index, collect’s walk stops at a child within the cutoff, by clock and id, whose parent is collected; older folded runs come from the index, older collected snapshots from the listing.
  Why

  A-store#index-walk
• chain-onlytestCollect reclaims only snapshots the bucket pointer reaches through parent links; another snapshot’s includes_runs collects no run.
  Why

  because a snapshot that lost its pointer commit folded nothing a bucket reader sees
• declared-windowtestCollect takes each table’s retain_runs from the collecting store’s declarations, a declared model defaulting as its builds’ retention does, and reclaims no snapshot or run of a table it declares none for, naming it under undeclared.
  Why

  because the window is a table declaration, not bucket state
• two-phasetestCollect records a key as collected whether or not a retained generation lists it, and deletes the object at the first collect after generation-listedstore.collect.generation-listedCollect deletes no object whose key a retained generation lists, and leaves it for a later collect. stops holding it.
  Why

  because a restore of a retained generation reads the object, while the record already keeps it out of every merge
• record-standstestWhile the bucket holds a recorded object, each collect stamps its record again once it passes half the age tombstone-ttlstore.merge.tombstone-ttlA tombstone leaves the manifest 30 d after its deleted_at. sets; past its object, a record lapses and collected-directorystore.merge.collected-directoryA merge keeps out each key under a snapshot directory whose _collected.json the bucket lists, save its _manifest.json, and each run file folded-runs reclaims, whatever any record says. keeps the key out.
  Why

  because a record lapsing before its object goes lets a merge list the key again
• retained-kepttesttestCollect deletes no recorded object a retained snapshot names, reporting its key under referenced, and the key stays recorded.
  Why

  because a hold reaching the bucket outside hold-bucket-firstrun.model.hold-bucket-firstOn a store declaring [sync], build hold commits the hold to the bucket before printing a receipt the bucket's copy names; a failed write, or meeting a collect of the build, undoes the local hold, listing any written object. then leaves a refusal rather than a loss
• serializedtestCollect holds the bucket’s collect lease from before it reads a hold until its last deletion, and refuses as heldstore.lease.heldIf a run finds an unexpired lease, the engine raises LeaseHeld, names the holder and expiry, and skips the run. The engine retries the run at the next reconciliation tick. does while another collect holds it.
  Why

  because a hold or a record a concurrent collect commits otherwise reaches the deletes of a collect that never read it
• lease-renewedtestCollect renews its lease at a fresh clock reading before its deletes and at renewalstore.lease.renewalThe process holding a compaction lease renews it every 200 s by If-Match replace on its ETag; sync compact renews while it pulls, folds and pushes.‘s interval through them, and deletes nothing more once stale-fencestore.lease.stale-fenceIf a commit-log create, pointer replacement or catalog UPDATE loses its condition to a higher commit fence, the operation raises LeaseFenced. The associated run or snapshot stays unreadable. answers a renewal.
  Why

  because a collect outliving its lease otherwise deletes beside a successor that read newer holds
• metadata-kepttestCollect records no snapshot _manifest.json or _collected.json, pointer, schema.json, hold or run state, and no key a retained snapshot names, a tombstoned one included.
  Why

  because those keys hold the chain and the coordination state every reader walks
• marker-first-writertestA collect uploads _collected.json only where no object holds its key, and lists the bytes the bucket then holds.
  Why

  because a losing concurrent collect overwriting a listed marker fails every pull’s digest check
• generation-listedtestCollect deletes no object whose key a retained generation lists, and leaves it for a later collect.
  Why

  because sync pull --generation restores a retained generation’s keys at their listed digests
• retain-generationstest[sync] retain_generations, default 64 generations, sets how many newest generation manifests a collect keeps, the current one included; a value of 0 fails the configuration’s parse.
  Why

  A-store#bucket-collect
• generationstesttestCollect heals an absent copy of the read generation, then deletes every manifests/gen-<N>.json older than the newest retain-generationsstore.collect.retain-generations[sync] retain_generations, default 64 generations, sets how many newest generation manifests a collect keeps, the current one included; a value of 0 fails the configuration's parse., so a restore of one meets generation-absentstore.pull.generation-absentA generation the bucket holds no gen-<N>.json for raises SyncGenerationAbsent, naming N and the newest generation, and writes nothing..
  Why

  because each generation manifest lists the whole store, and a push after every run writes one per run
• shared-prefixtesttesttestWhile the bucket manifest names another project’s key, record or control head, collect records no key as collected, deletes no generation manifest, treats every generation as retained, and reports shared_prefix.
  Why

  because generation manifests belong to the prefix, and one project’s retain_generations binds no other project’s restores
• drop-firsttesttestCollect commits a bucket manifest recording each key it unlists as collected, by the conditional replace of manifest-commitstore.push.manifest-commitA push commits by replacing <prefix>/manifest.json under If-Match on the ETag it read; this bucket manifest lists each key's sha256, size and owner., before deleting any object, so no push lists a deleted object and a lazy read meets part-collectedstore.pull.part-collectedA part lazy-parts needs and the bucket lacks raises SyncPartCollected, naming the part and snapshot; the next pull advances the pointer past that snapshot..
  Why

  A-store#bucket-collect

unsettled: Does a collect reclaim another project’s unlisted objects when the generations it prunes are the last to name them? owner: store affects: store.collect

Scenarios

• store.collect.drop-firststore.collect.drop-firstCollect commits a bucket manifest recording each key it unlists as collected, by the conditional replace of manifest-commit, before deleting any object, so no push lists a deleted object and a lazy read meets part-collected.: WHEN an owner restoring a tombstoned run reads the manifest, a collect then commits and deletes the run’s objects, and the owner’s push commits, THEN the push loses its condition, re-merges, and lists no deleted object.
• store.collect.generation-listedstore.collect.generation-listedCollect deletes no object whose key a retained generation lists, and leaves it for a later collect.: WHEN a tombstoned run is unlisted while 64 retained generations list it, THEN the collect deletes none of its objects.
• store.collect.manifest-keptstore.collect.manifest-keptCollect deletes no snapshot _manifest.json and lists _collected.json beside each snapshot it reclaims, the only writer of that marker, so every chain reaches its root and each includes_runs still names its folded runs.: WHEN a node holding a copy of run r but no snapshot pulls after a collect reclaimed the snapshot folding r, THEN its chain reaches that snapshot’s manifest and r reads once.
• store.collect.folded-runsstore.collect.folded-runsCollect records as collected every object of a run directory, _manifest.json included, once the bucket chain snapshot naming the run in includes_runs supersedes a snapshot superseded reclaims, save a _group.json group-window keeps.: WHEN the owner of a collected run lists its restored copy again after the record’s age, THEN every reader folds the run through the kept manifest and reads it once.
• store.collect.group-windowstore.collect.group-windowCollect keeps a folded run's _group.json while a member table holding a copy of the run has not folded it out under folded-runs, under that member's retain_runs.: WHEN a member table folds its copy of a grouped run one day before a collect, THEN the root run’s directory leaves save its _group.json.

A collect, from the manifest commit to the generation manifests it deletes.