contract run · 30-run.md · 31-pipeline.md · 32-derive.md · guide · card
Durable runs
A run executes a compiled plan: read a source, record steps, commit rows and advance the
source position. Implementers use this contract for the journal, retries and cancellation
shared by pipelines and derive tasks. Row processing follows 31-pipeline.md.
The runner records steps, commits rows and publishes run status:
journal
The journal records step values, returns those values on replay and deletes records after execution retirement.
step-outputtesttesttheoremThe journal records a step’s value exactly once. Its effect runs at least once: a crash between effect and write repeats the effect. Every later call under the key returns the recorded value.Why
because exactly-once effects need the vendor’s cooperation, and the recorded value is the one fact a replay can rely onentry-keytesttesttheoremThe journal keys entries by(execution_id, step_label, input_hash). The first caller claims the key with a pending row before running the effect. A concurrent caller waits for the recorded value without computing.Why
because two racers computing one key issue two billed calls and spend two meter permitsclaim-takeovertesttesttesttheoremIf a pending claim’s run owner lease expires,owner-lease, the next caller under that key takes over the claim.idempotency-keytestEvery outbound request a step makes carries an idempotency key derived from its entry key, identical on every re-entry of that effect.Why
because a vendor honoring the key turns an at-least-once effect into one billed callinline-cutofftestThe journal stores values of 1 MiB or smaller inline as row bytes. For larger values, the journal stores a content-addressed blob named by its sha256 and a reference in the row.blob-writetestConcurrent writers of one blob hash converge on one stored value and none errors on another’s write; no partial write survives beside it.missing-blobtestIf a row references a missing blob, the journal raisesBlobMissingcarrying the reference. The journal returns no empty replacement value.Why
P6collectiontesttestRetiring an execution owner deletes its journal rows in the retiring transaction; a journal store apart from the catalog deletes them after the catalog commits the retirement, and the next open under that scope repeats the delete.Why
because recorded work is needed only while a replay can reach it, and an uncollected journal grows with total ingestone-cutofftesttestEvery step kind followsinline-cutoff: pulls, recorded effects, input steps and awakeable payloads, whose pending row references an offloaded blob. No step kind overrides the cutoff.Why
because one placement rule lets every adapter and the blob sweep read a row without knowing which step kind wrote itfan-out-jointestParallel branches rejoin only at an explicit join node. A failed branch reaching the join fails the run. Withallow_partial, the join instead records each failed branch’s label and failure tag on the run record.Why
because an implicit join hides which branch failed, and a partial result recorded as complete reads as oneblob-sweeptesttesttesttheoremA mark-and-sweep pass every 24 h deletes each blob that no journal row or pending awakeable references and that is older than 1 h.Why
because the grace window covers a blob written ahead of the row that names iteffect-boundarytestA body replays faithfully when every observable side effect passes through a recorded step, a cursor commit or an awakeable; the work between them is pure.unwired-capabilitytestRequesting a capability absent from the running profile raisesCapabilityUnwiredat the first request, before any partial work.machine-statetestThe host’s own stores hold journal rows, execution owners and awakeables. The derived catalog holds none. A catalog rebuild changes none and reconstructs none from the file tree.
pulls
A journaled pull records its guarded batch, and a journal hit hands that batch to the land path.
recorded-batchtestA journaled pull records its guarded source batch, or its canonically prepared replacement underbefore-the-record, ahead of landing.replay-landstesttesttesttesttesttestA journal hit hands the recorded batch to the land path, so a resumed run lands bytes identical to an uninterrupted one, batch ordinal included.ledger-settles-firsttestThe outbound request ledger settles durably before the entry recording its batch commits.redacting-sourcetestA journaled source declaring removal without canonical prepared-recording admission raisesJournalRedactionConflictbefore its first pull; a boolean removal assertion supplies no admission.prepared-ownertestProtected recording binds the plan, shape, canonical declarations, normalization and pepper identities into owner pins before source execution; a changed identity followspinned-plan-changed.prepared-replaytestA protected journal hit admits the recorded rewritten payload against its canonical writer without repeating shaping, normalization or removal and without fetching the source again.opt-outtestPull journaling defaults on. A source whose pre-pull cursor cannot name the content it reads opts out through one constant, pinned against each source’s declaration by a test. An empty pull is never journaled.escape-hatchtestJournaling permits only two exceptions:journal.unsafe(label, effect)for an idempotent read, and a source declaring no pull journaling. Each exception states its idempotency argument at a searchable call site.
storage
Journal rows, blobs and awakeables persist through storage ports that SQLite, PostgreSQL and D1 serve.
substrate-porttestOne execution interface provides: open or resume under a scope and content-hashed plan reference; record step output; commit a cursor; suspend on an awakeable with timeout; attach a retry schedule; describe capabilities.storage-portstestJournal rows, blobs and awakeables persist through a journal store, a blob store and an awakeable store; everyrun.journalandrun.suspendclause holds for each adapter, the file tree included.sqlite-storestestcontextful-sqliteserves the journal, blob and awakeable stores from one SQLite file in write-ahead-log mode over one connection, a blob as a row keyed by its sha256; an awakeable update commits with the journal writes inside it.Why
because a resolution recording its payload through a second connection waits on the write lock its own update holdssqlite-write-locktestEverycontextful-sqlitestore write opens an immediate transaction, so concurrent writers of one key across connections to one file serialize: one record lands, and every other returns the standing value instead of a locked-database failure.Why
because under write-ahead logging a deferred read upgrading to a write fails on the lock at once instead of waiting through the busy timeoutpostgres-storestestcontextful-pgserves journal rows, blobs and awakeables throughstorage-portsin the catalog’s PostgreSQL database; conditional journal writes serialize across connections, and recorded values survive replacement of every local disk.container-profiletestCONTEXTFUL_STORAGE_PROFILE=postgresselectspostgres-storeswith shared owners, cadence andremote-state; corpus objects use a remote S3-compatible bucket with pull-before-run and push-after-run enabled.d1-profiletestCONTEXTFUL_STORAGE_PROFILE=cloudflare-d1selects D1 for catalog, journal, awakeables and export state through a private Worker outbound binding; R2 holds blob content and corpus objects.d1-blobtestD1 commits metadata atomically and stores each blob digest, whose R2 object key isblobs/<sha256>; an R2 write precedes its D1 reference, an R2 deletion follows removal of every reference, and a read verifies the digest.d1-transporttestA D1 read, health probe or content-addressed blob request left without an HTTP answer resends, within 3 attempts of at most 15 s each; a writing statement outsided1-initializegoes once over a fresh connection and raisesStorageBindingUnavailable.Why
because a write whose answer is lost may have committed, and resending it could apply it twiced1-initializetestStorage initialization’s D1 batch, every statementIF NOT EXISTSDDL or an insert or update guarded to apply once, resends after a lost answer as a read does underd1-transport.Why
because a second application of that batch leaves the state its first application leftd1-guardtestA committed D1 guard blocks transfer of the rows it protects throughout an admitted external callback and outlives the callback’s crash, releasing only underrelease,replayormanual.d1-recoverytestThe D1 recovery identity binds one project and schema to an R2 object frontier; a missing object or mismatched generation followsexecution-recovery.d1-backuptestA D1 recovery captures the execution database with its guard and export rows, the R2 blob and corpus frontier, project recovery identity and executable schema before writers resume.storage-incompletetestMissing profile bindings, corpus state, strict execution guards or a supported outbox raisesStorageCapabilityMissingbefore execution admission; absent compiled adapters followcapability-absent.storage-unsupportedtestAn unknown storage profile or an encrypted corpus paired with an unsealed execution adapter raisesStorageProfileUnsupportedbefore execution admission.Why
because an unrecognized durability claim cannot select a local fallback safelyexecution-schematestAn execution database whose schema differs from the executable’s supported schema raisesExecutionSchemaIncompatiblebefore execution admission; preflight creates no missing execution table.Why
because creating an empty replacement for a missing table erases the evidence of a partial restoreexecution-recoverytestMissing execution tables or a mismatched project recovery identity raiseExecutionRecoveryMismatch. Normal startup provisions no recovery identity. A corpus-only restore admits no paid or effectful work.current-ownertestAn execution checks its durable owner, admitted pins and active attempt immediately before effect dispatch and publication; a superseded attempt enters neither boundary, and a late stop still followsland-path-uncut.durable-dispatchtestPostgreSQL admission holds shared owner and unexpired lease locks through each bounded provider or destination callback, including item ancestors and entity fences; owner replacement, retirement and takeover serialize after admitted callbacks.guard-callbacktestA guarded callback may update run metadata; mutating its admitted owner or lease refuses. Nested callbacks retain only scopes and leases their caller already admits. Local adapters expose cooperative checks without strict execution-guard capability.Why
because nested lock expansion and owner mutation can deadlock the admission that invokes them
store-driven input
A store-driven run records its input once and runs its registered body once per input row.
plan-pintestA run resolves the plan reference it started against for its whole life.store-inputtestA store-driven run reads its statement once, through the read face under the job’s grant at itsas_of, and records the resolved snapshot id per table and the ordered input rows as its first step.open-as-oftesttestA store-driven job declaring noas_ofresolves it to the instant its execution opens, recorded in the input step; a resume iterates that recorded input without re-reading the statement, so rows landed afteras_of, or folded since, stay out.input-pintesttestA store-driven run’s plan reference hashes its body name, statement, declaredas_ofand any host-supplied structural identity, which also joins its owner pins, so a resume under a changed one meetspinned-plan-changedbefore any step replays.input-truncatedtestAn input statement whose response the face truncates at its row ceiling raisesRunInputTruncatedbefore any row runs.Why
because a silently shortened input set closessuccessas though every row ranrow-steptestEach input row runs the registered body under a row key, its ordinal in the recorded input; every step the body records carries a label scoped to that key, and a paid call carriesidempotency-key.Why
because two rows sharing one label under one execution resolve to one recorded valuerow-concurrencytestA batch store-driven run holds at mostmax_in_flightrows inside the body at once; a failing row admits no further row, and the run closes after the rows in flight return.row-outputtestA batch’s emitted rows land after the last input row completes and before the owner retires, one run per declared output table throughstage-order.Why
because a landing that fails then holds the owner, and its resume replays every paid call instead of paying againbody-projectiontestA registered body’s closed effect labels, declared roots, types, transforms and normalization bind its owner before input or effect execution; a caller-supplied projection differing from that registration refuses.
callback-guard
A D1 callback guard under d1-guard: the rows it protects, the effect identity it records, its release after the holder stops, and what an operator sees of it.
scopetesttestA guard protects each owner scope its callback admits, every ancestor scope and every lease key, entity fences included; another job’s admission, an unrelated export and transfer of any unprotected row proceed.acquiretestGuard acquisition checks the exact owner, active attempt and fence of every row it protects and binds them in one D1 statement; a row another guard protects admits no second callback.holder-identitytestAn engine draws a random holder identity at start and registers it in D1 before admitting its first guard.identitytestA guard records its execution, attempt, holder identity, journal entry key, the keyidempotency-keysends, admission time, and its destination’s declared deduplication window,nonewhen the destination declares none.windowtestA deduplication window covers a guard’s key from its admission time until that time plus the declared window; anonewindow covers nothing.Why
because the request leaves after admission, so a window counted from admission ends no later than the destination’s ownreleasetesttestA holder deletes its own guard when its callback returns, before journaling the callback’s value.holder-gonetestA guard’s holder is gone once its holder identity records an exit, written by the holder on stop or by the serve process reaping it underchildren-reaped; a lapsed owner lease confirms nothing.Why
because a stalled holder misses renewals while its request may still reach the destinationcontainer-stoptestA container that stops before its serve process reaps a holder leaves that holder’s guards held untilmanualreleases them once the owner lease lapses; nothing else records the holder’s exit.Why
because no platform guarantee confirms that a reported container stop has ended every process the container ran, and a surviving one may still send the effectreplaytestWithout an operator, a guard releases only when its holder is gone, its owner lease has lapsed andwindowstill covers its key; release and takeover commit in one D1 batch, and replay sends that key.manualtestApart fromreleaseandreplay, a guard releases only throughcontextful storage reconcile-guard, which deletes it and appends an audit row naming token, evidence and operator in one D1 batch.live-holdertestReconciling a guard whose owner lease is unexpired and whose holder records no exit raisesExecutionGuardHeldand leaves the guard.Why
because its callback may still be sending the effect the guard protectsevidencetestidempotency:<key>releases a guard when the key equals its recorded key andwindowstill covers it;receipt:<key>:<id>when the receipt’s key equals it. Other evidence raisesExecutionGuardEvidenceMismatchand leaves the guard.Why
because evidence the guard cannot tie to its own effect says nothing about whether that effect completedclasstestA guard’s evidence class isreplay-safewhilewindowcovers its key; otherwisereceipt-requiredfor a publication callback andunknownfor any other.inspecttesttestcontextful storage guard --project <name>prints each guard’s token, protected scopes and lease keys, execution, attempt, holder identity, journal entry key, idempotency key, admission time, window and evidence class.Why
because an operator checks the destination for the guard’s key before supplying evidenceblockedtestA transfer a guard blocks answers retryable, naming the guard’s token and evidence class, andcontextful storage reconcile-guardunless the class isreplay-safe.Why
because a generic transient failure leaves an operator retrying a guard that only reconciliation releaseslegacy-guardtestA schema-3 database holding aglobalguard row stays at schema 3 underexecution-schema, its refusal naming the token and the step: reconcile it with the release that wrote it, then runcontextful storage initialize.Why
because aglobalrow records no idempotency key, so no evidenceevidenceaccepts ties to its effect
Scenarios
run.callback-guard.scope: WHEN a callback on one item holds a guard, THEN takeover of that item’s owner, its ancestors and its entity lease refuses, while another job’s admission and an unrelated export proceed.run.callback-guard.scope: WHEN a guard protects a job scope alone, THEN an item owner beneath it transfers, and that item’s next callback waits on the shared job row.run.callback-guard.acquire: WHEN two callbacks protect disjoint rows, THEN both run concurrently; WHEN they share a protected row, THEN the second admits no callback.run.callback-guard.replay: WHEN a guard’s owner lease has lapsed but its holder records no exit, THEN the guard stays.run.callback-guard.replay: WHEN a gone holder’s window has passed or isnone, THEN the guard stays formanual.run.callback-guard.replay: WHEN a gone holder’s window still covers its key, THEN release and takeover commit together and the replayed request carries the original key.run.callback-guard.live-holder: WHEN an operator reconciles a guard whose owner lease is unexpired, THEN reconciliation raisesExecutionGuardHeldand the guard stays.run.callback-guard.evidence: WHEN an operator supplies an idempotency key other than the guard’s recorded key, THEN reconciliation raisesExecutionGuardEvidenceMismatchand the guard stays.run.callback-guard.evidence: WHEN a guard’s window isnone, THEN its recorded key asidempotency:evidence raisesExecutionGuardEvidenceMismatch, and a receipt carrying that key releases it.run.callback-guard.container-stop: WHEN a container stops before serve reaps its holder, THEN takeover refuses while the owner lease runs and after it lapses, and matching evidence releases the guard once it lapses.
items
The engine executes isolated input items with independent completion receipts, bounded retries and leased entity keys.
opt-intestA store-driven job defaults to batch execution;isolated_itemsexplicitly declares logical and entity key columns, an entity namespace and a positive failed-attempt budget.Why
because independent completion changes aggregate failure and retry semanticsidentitytestAn isolated item’s logical identity derives from its declared key and namespace; its execution identity binds that identity, job, input row, policy and pinned plan, independently of input order.completiontesttestEach successful item lands independently with a durable completion receipt that a later fire under its execution identity skips; once every item holds a terminal receipt, the aggregate releases its pinned input owner, so later fires read fresh input.outcomestestIndividual outcomes distinguish success, failure, parked, deferred and waiting; the aggregate succeeds only when every required item succeeds.retry-budgettestFailed attempts consume the declared item budget across fires; permanent failures terminate immediately, parked entries consume no failed attempt, and terminal failure receipts prevent further body entry.Why
P6replaytestA nonterminal failed or parked item retains its execution owner and recorded paid calls; its next admitted attempt reuses that journal.entity-admissiontestAdmission holds one leased entity namespace and key across triggers, job names and execution versions; unrelated entities occupy at most the job’smax_in_flightbody slots.fencetestRecorded effects and output publication check their execution owner and entity fence immediately before dispatch; a superseded owner sends no provider request and publishes no output.Why
because an owner that resumes after transfer cannot authorize another effectdelegated-itemtestUnder a delegated body, each call ordinal is one request underenvelope, itsAnswerrecorded under the item’s entry key, so a resume replays every recorded answer and sends only the unrecorded call.outcome-recordtestEach terminal item appends one row to the reserved_itemstable once its receipt commits: job, logical key, entity namespace and key, attempt count, terminal outcome, reason, attempt run id, execution id and end instant.terminal-outcometestAn item row’s outcome is anItemTerminal,exhaustedon the attempt spendingretry-budget; its reason carries the failure tag anderror-capmessage.waiting-itemtestUnderisolated_items, every retry wait closes its itemwaiting-stateand frees its slot; once the pass ends, an item whose instant falls withinretry-after‘s ceiling re-enters in the same fire.Why
because one throttled item sleeping in a slot delays every sibling queued behind itserved-keytestAn item row carries its logical and entity keys as the job’s input relation served them, so a key its source masks or classes lands under that rule, never in the clear.
Scenarios
run.items.completion: WHEN one input fails permanently among three isolated items, THEN both healthy items land and reordering a later fire repeats neither completed body.run.items.fence: WHEN a newer owner takes an entity’s lease while an earlier job’s item resumes, THEN that resume sends no delegate request and publishes nothing.run.items.entity-admission: WHEN three store-driven jobs declare one entity namespace and key column and two fire in one window, THEN at most one item per key enters a body and the later job’s item reports deferred.run.items.outcome-record: WHEN one of three items fails transiently under a one-attempt budget, THEN_itemsholds twosucceededrows and oneexhaustedrow naming the job and entity key, and a later fire adds none.run.items.served-key: WHEN the input table hashes the entity key column, THEN every_itemsrow carries the 32-char digest and no clear key.
unsettled: Does a job credential holding a mask exception record its entity key in the clear on _items, or does the item record apply the source column’s mask itself? owner: run-path affects: run.items
advance
The engine commits source positions according to cursor kind and re-reads rows at each poll’s boundary.
cursor-kindtestA cursor declares aCursorKind. An undeclared kind resolves toopaque-token.concurrency-by-kindtestAmonotonicposition is concurrent-safe and commits the highest value observed. A position of anotherCursorKindmoves only under a single-writer lease. Last-write-wins governs no cursor.Why
because two writers resolving one continuation token by recency skip or duplicate records with no count showing itcommit-with-rowstestA table’s run commit marker,run-manifest, publishes its rows and new position together. The catalog cursor caches the newest committed marker.Why
because rows and position committed in two stores leave a crash window that re-lands a batchcursor-bytestesttestA position is opaque bytes the connector owns, one per table, and its kind a manifest fact; the connector moves it only by committing through the engine, a recorded step that replays as a resolved read.inclusive-boundarytesttestA polled load admits a row whose clock value is at or after the stored position, re-landing the boundary instant’s rows on every poll, idempotently on a keyed table and accumulating in a keyless table’s union view.Why
because a clock coarser than the row grain otherwise drops a sibling sharing the boundary value, permanently and invisiblywatermark-shapetestA watermark position serializes as{"field": "<name>", "at": <value>}.fieldnames the column whose values determine the watermark.field-renametestOpening a position whose stored field differs from the declaredincrementalfield raisesCursorFieldMismatchbefore any request leaves the host.frontiertestThe frontier counts every fetched row, landed or not. A committed position advances or stays unchanged. An empty or older window does not rewind the position.walk-positiontestUnder amonotoniccursor, a pull reporting more and carrying a position other than the one it asked continues its walk: the next pull asks that position and admits rows from the walk’s floor, and the run commits the frontier.unorderable-positiontestA row with no orderable value in the clock field, or a stream switching between text and numeric positions mid-pass, raisesCursorPositionUnorderable, terminal for the pull.allowed-latenesstestA table declaresallowed_lateness, default 0 s; amonotonicpoll re-reads from its stored position minus that window, so a row arriving that late still lands.Why
because lateness belongs to the source feeding a table, and a position records only where the frontier standsturning-incremental-ontestEnablingincrementalon a pipeline that holds a position starts from none, re-landing the source’s current window once.skip-unchangedtestA snapshot-shaped source withskip_unchanged = truerecords its input’s raw-byte digest as{ sha256, rows }; a matching digest returns zero batches, holds the position and closes a zero-row success. Undeclared, it is false.snapshot-completiontestA snapshot source reports complete inventory or unchanged skip independently of row count; a replacing run carries only complete inventory into its manifest.zero-row-committestA complete empty snapshot of a replacing table commitsempty-replacement; a skipped unchanged input holds the frontier and position, and a failed pull publishes neither.
Scenarios
run.advance.zero-row-commit: WHEN a complete snapshot holds zero rows after a non-empty replacing run, THEN the current read is empty and a bounded read before the commit retains the older rows.run.advance.walk-position: WHEN amonotonicwalk’s pull returns no rows, the position it asked and more, THEN the run ends and commits its frontier.
suspend
The engine suspends runs durably until an external callback resolves an awakeable or its deadline expires.
awakeabletestAn awakeable suspends a run durably: the engine mints an opaque single-use token and persists apendingrow through the awakeable store; an external party resumes by posting the token back.resume-is-a-step-outputtesttestThe journal records a resume payload as the awaited step’s output under keysha256("awakeable:" + token), so a resolution repeating the identical payload returns it, and a run that resumes then crashes reads it back without suspending again.deadlinetesttestA deadline is the creation instant plus a time-to-live, both caller-supplied RFC3339 Zulu strings; the core reads no wall clock, and a suspension past its deadline at an injected instant staystimed_outunder any later one.conflicting-resolutiontestA second resolution with a different payload raisesAwakeableAlreadyResolvedand leaves the recorded value untouched.expired-tokentestResolving a token past its deadline raisesAwakeableTimedOut.unknown-tokentestA token with no row raisesAwakeableUnknownand allocates no state.Why
P2resume-routetestPOST /awake/:tokenon the served face answers200with the recorded payload,404forunknown-token,409forconflicting-resolutionand410forexpired-token.GET /awake/:tokenreports state without resuming.resume-authenticatestestThe resume route admits the request’s credential before reading the registry, answering401without one, and passes the credential’s verifiedon_behalf_ofas the caller subject.caller-bindingtestAn awakeable may bind a verified caller subject at mint; resolving it then takes the token and a credential for that subject, and any other caller answers asunknown-token. An unbound token is its whole authority.Why
because a token leaked through a log otherwise resumes a run, and answering unknown discloses nothing to the wrong holderpostgres-reconciliationtestA PostgreSQL callback row interrupted after its resume value commits recovers its resolved payload from that journal value; a conflicting resolution followsconflicting-resolution.Why
because a separately committed callback row cannot overwrite the journal value that already determines replaysurvives-restarttestA durable awakeable store persists every registry row, so a process reopening it over the same location drops no pending callback; an in-process store ends with its process.row-parkstestA store-driven row awaiting an unresolved awakeable parks and frees its slot; the other rows run on, and the parked row re-enters its body once the awakeable resolves or its deadline passes.recorded-timeouttestA row’s wait past its deadline records a timeout value as that wait’s step output, so every re-entry of the body reads the identical value.Why
because a body branching on the clock rather than on a recorded value replays differently after a crash
retry
The runner classifies failures and schedules step retries through one retry layer.
failure-taxonomytestOne tagged failure type,FailureTag, crosses every port. A caller branches on the tag, never on a message.Why
P2scheduletestA schedule attaches to a step and carries a fixed or exponential backoff with base and factor, a total attempt count including the first, a per-delay clamp, a jitter ceiling andretry_on.default-policytestThe default schedule is exponential from a 100 ms base with factor 2, to 5 attempts in total, each delay clamped at 30 s, plus seeded jitter below 250 ms.Why
because zero jitter synchronizes every run that hit one shared throttlesingle-attempttestA schedule of 1 attempts is the no-retry schedule.retryable-classestestTransientandRateLimitedretry; every other tag stands as the step’s answer whatever budget remains.retry_onnarrows that pair.Why
P6unretryable-tagtestAretry_onnaming any tag outsideTransientandRateLimitedraisesRunRetryTagUnretryableat plan compile.Why
P6step-failedtestA non-retryable failure, or a step exhausting its schedule, raisesStepFailedcarrying the step label and the underlying tag.Why
P6deterministic-verdicttestA refusal whose check is pure over static input is terminal and consumes no attempt budget.Why
P6retry-aftertestA server-suppliedRetry-Aftersupersedes the computed delay; a remaining wait up to 300 s sleeps in process, and a longer one closes the attemptRateLimitedunderwaiting-stateinstead of sleeping.Why
because an unclamped upstream delay holds a run slot for as long as the vendor asksnot-beforetesttestBefore any wait, a retrying step records its next-attempt instant and attempt count on its journal row, so a restart, a container replacement included, resumes the remaining budget and wait.Why
because a wait held only in memory restarts on every replacement and calls the vendor earlywaiting-statetestAn attempt closed on its next-attempt instant reportswaitingwith that instant on its run row, counts as neither refused nor failed, and keeps its owner; only exhausting the schedule’s attempts fails the step.Why
because a vendor’s requested pause is not evidence the work is wrongresume-attesttestUntil a step’s stored instant passes, an attempt reaching it, a crash mid-sleep included, waits only the remaining time without invoking the effect; the first attempt once it passes re-enters under its work key, pins and attempt count.Why
because resuming under another key or a fresh budget repeats paid work or retries without boundwaketestA waiting attempt records its instant on its owner; the scheduler fires that owner’s unit at the first beat at or after it, ahead of the unit’s regularschedulefire, and the next attempt’s claim clears it.Why
because a resume gated on the regular cadence adds up to a full interval to every rate-limited waitno-double-spendtestA resumed attempt replays every recorded step, so it reserves no meter permit and issues no call an earlier attempt already recorded.Why
because each resume reserving afresh spends a budget once per waitone-layertesttestThe step’s schedule is the only retry layer: an adapter, a credential mint and a meter acquire return their failure to it, an HTTPRetry-After, throttled mint or limiter denial asRateLimitedcarrying its wait undernot-before.Why
because nested retry layers multiply attempts and amplify a vendor outagedecision-is-puretestThe retry decision is a pure function of the completed one-based attempt and classified failure. Jitter derives from an injected seed. The runner owns the sleep and attempt counter.partial-failuretestA run failing after it staged batches commits none of them underone-commit-per-runand closesfailed, holding its owner so the journal keeps the resume point; onlyorphan-reapcloses a runpartial_failure.no-compensationtestThe engine offers no compensation combinator; undoing a partial effect is an author-written recorded step, replayed like any other.Why
because compensation depends on each vendor’s undo semantics, which the engine cannot knowpressure-is-localtestRate-limit pressure against one source paces only that source’s runs.bound-hit-is-transienttestA connector strikinglinear-memoryorcall-deadlinefails the attemptTransient, the failure naming the bound it struck.bound-hit-recordedtestThe run record lists every bound hit with its step label, attempt and bound, a hit a later attempt recovered from included.
own
An execution owner binds pending work to a scope, connector build and plan identity.
execution-ownertestOne durable execution owner exists per scope; a table or chunk owner holds the connector identity with its component world, the pipelinecontent_hashand the input hash.host-scopetestThe catalog keys every owner on its scope — live table, backfill chunk or host-declared id — a table owner keeping its stored key byte for byte, and a host owner pinning the plan reference and identities the host supplies.execution-id-keys-the-journaltestRecorded work keys on the execution id; a catalog run id is the provenance of one attempt, so attempts under one owner resolve the same recorded values.retirementtestRetiring an owner and caching its committed position share one catalog transaction; a later fire receives a fresh execution id even at a byte-identical position. A completed backfill chunk retires the same way.marker-reconcilestestAt run open, a commit marker newer than the catalog’s cached position retires the pending owner that produced it before any replay.Why
because a crash between marker and catalog otherwise replays a recorded pull into rows already committedpinned-plan-changedtestWhile an owner is pending, a changed pin — connector identity, component world, pipelinecontent_hash, or a host’s plan reference or identities — raisesExecutionPinMismatchbefore replay, terminal and non-retryable, naming the scope and both pin sets.pin-mismatch-recoverytesttestRestoring the recorded build and resuming to completion clears a pending owner;rewindretires a host or table owner, a chunk rewind those its window covers. Thepinned-plan-changedrefusal names these recoveries with the pending execution id, and no other.Why
because an operator acts on the refusal’s text, and a recovery no verb provides leaves the scope refusing every firescope-independencetestA finishing table releases nothing another table’s unfinished execution holds, and a seeding scope carries its own source identity.unclosed-executiontestAn execution handle dropped without close records no status; its owner stays pending, and the next open under its scope resumes it onceowner-leaselapses.live-ownertestAn open under a host scope whose pending owner has an attempt on an unexpiredowner-leasefailsTransientnaming that attempt, and its run row closesfailedwithout joining the owner.Why
because two live handles on one owner share its journal, and either one retiring it strands the otherpin-releasetestsuccess, a failure that wrote no batch, and replay-stable failures understaged-bytes,late-type, orsource-stage-schemarelease the owner; every other status holds it.admission-pintestA run pins its connector identity at admission and a replay resolves the artifact from that pin; a connector rebuilt later reaches no in-flight or replayed run.
body pins
A registered body’s owner pins bind its executable, its dependencies and its parent execution before any input read.
body-implementationtestStore-driven owner pins include the host-verified digest of its executing binary and concrete registered body type; configuration selects a body name and supplies no implementation identity.body-dependenciestestRegistration declares prompt/template, public provider/model configuration and policy identities affecting execution; owner pins carry their digests independently of structural input and output declarations.body-admissiontesttestBody implementation and dependency pins bind before input reads, recorded-call replay or body entry; a registration declaring no dependency receives the host implementation pin alone, and unavailable, changed or absent pins followpinned-plan-changed.Why
because an unversioned record cannot establish which executable produced its paid callsbody-freshtestA fresh execution under changed implementation or dependencies receives a distinct plan identity and preserves earlier run records; pending work requires its recorded host build and registered dependencies.body-public-configurationtestSemantic provider configuration contains public endpoint, model and generation parameters; credential-bearing endpoints refuse registration, credentials enter neither exported diagnostics nor recorded pins, and credential rotation alone changes no semantic identity.Why
because credential custody and executable semantics are independent identitiesbody-parenttestPrepared body outputs require the unchanged parent execution and current in-flight, unstopped attempt at admission, staging and commit; a stopped parent reachesabandoned-workindependently of the output table’s stop policy.
operator retirement
An operator verb or an applied declaration retires a pending owner and records the retirement as a run row.
rewindtestcontextful run rewind <execution-id> --reason <text>retires the pending host or table owner holding that execution and leaves the scope’s committed position alone, so the scope’s next fire claims a fresh execution.rewind-recordtestA retirement underrewindorapply-retires-supersededwrites onecanceledrun row under the scope carrying the retired execution id, its pinned plan reference and the reason, started at the owner’s open.Why
because a row started at the retirement reads as the job’s latest start and postpones its next due firerewind-not-pendingtestA rewind naming an execution no pending host or table owner holds raisesExecutionRewindNotPending.Why
because a completed or retired execution holds nothing to release, and a chunk owner retires through its window’s rewindrewind-plan-currenttestIf no attempt or refused open requested another plan reference, rewinding the owner raisesExecutionRewindPlanCurrent. The operator may override this refusal with--force.Why
because such an owner resumes on its next fire, and retiring it discards its recorded workrewind-unsafetestRetiring an owner with an attempt on an unexpiredowner-lease, awaitingattempt, or an awakeable pending before its deadline raisesExecutionRewindUnsafenaming it;--forceoverrides none.apply-retires-supersededtestA claimedpipeline applyretires a build or sweepless store-driven job’s pending owner when the applied snapshot produces no matching declared plan reference: a verified body’sinputidentity, otherwise its plan reference.apply-retire-refusedtestAn applied retirement meetingrewind-unsafeleaves the owner pending and the snapshot claimed, and the apply names the execution andrewindon standard error.
staging and checkpoints
A run stages each batch as a part, bounds its staged bytes and commits its parts once or at checkpoints.
backpressuretestThe runner stages each shaped batch through the destination as one part before its next pull, so a run holds one pulled batch in memory; the commit names the staged parts.Why
because holding every batch until the commit sizes a run’s memory by the sum of its pullsone-commit-per-runtestOne run produces one atomic commit per table, plus one percheckpoint-commit; a crash mid-run leaves parts under that run’s own directory, and a resumption continues from the last recorded step or commit.staged-bytestestA stage carrying the parts a run staged since its last commit past 1 GiB raisesRunStagedBytesExceeded, deterministic; the run commits nothing more and retires its owner, so the next fire pulls afresh from the stored cursor.Why
because a replay of the run’s recorded pulls stages the same bytes past the bound and fails identically, holding the table’s owner indefinitelycheckpoint-committestWhen the parts a run of a position-addressed source staged since its last commit reach 512 MiB, the run commits them with the position after its last staged pull before staging more rows, then keeps pulling under the same execution.checkpoint-frontiertestUnder amonotoniccursor, a checkpoint commits only before a pull whose every clock passes the run’s frontier, so no commit’s position splits the rows of one instant.checkpoint-resumetestA checkpoint caches its position and marker in the catalog cursor while the owner stays pending, and collects the journal’s recorded pulls, so a resumption pulls afresh from that position.checkpoint-run-idtestA run’s first commit lands under its run id and each later one under<run id>.<n>,nits zero-padded six-digit ordinal, so every commit names its own parts.Why
because the store holds one manifest per run id, and a later commit naming an earlier one’s directory finds it committedcheckpoint-id-lengthtestA checkpoint whose next<run id>.<n>passesrun-id-shaperaisesRunCheckpointIdTooLongbefore it commits; the run commits nothing and retires its owner.Why
because a run committing its first checkpoint and then failing on the next one’s run id leaves a partial window publishedcheckpoint-landedtestA run failing after a checkpoint’s commit lands records on its row the rows and bytes every landed checkpoint commit carried, including one whose cursor update then fails.Why
because the checkpoints stay committed, and a failed row counting no rows hides rows every read returnscheckpoint-scopetestA source is position-addressed when each pull’s position names a watermark, page token or stable offset. Ahttp-url-walk, a replacing table, a body’s prepared landing, and a seeding or receiving scope commit once.stage-schematestA stage refuses againstschema.jsonand the run’s earlier stages, then merges its columns into the run’s own staged schema; only the commit merges them intoschema.json.Why
because otherwise rows no commit publishes fix a column’s type in the table’s schemasource-stage-schematestA deterministicSchemaIncompatiblefrom staging an ordinary source batch, raw or prepared, retires its owner unless another live attempt holds it; the failed run changes neither committed cursor nor published rows. This exception excludes prepared body-emission staging.Why
because a recorded source batch that fails the same schema check on replay prevents a repaired source from being fetchedstage-commit-seqtestA staged part carries no_commit_seq; the commit assigns it undercommit-orderand writes it into each part it names, so a run committing between a stage and that commit takes the lower value.Why
because a value fixed at the stage orders a run by its first pull, and the readable runs then hold no prefix of the sequencestage-instanttestA staged part carries no_ingested_at; the commit writes its commit instant into each part it names, so every row of a run carries the instant its marker carries.Why
because a stage-time instant ranks a run by its first pull, so underdedup-viewan earlier committer outranks a later one on a shared keystage-discardtestA run that fails removes every part it staged, and a commit failing before its manifest exists removes the parts it copied; a run’s crash leaves its staged parts.Why
because a run id names one attempt, so no later commit names a failed run’s staged parts
Scenarios
run.own.unclosed-execution: WHEN a host execution records two steps and its process dies mid-third, THEN the next open under its scope after the lease lapses replays both steps without running their effects.run.own.live-owner: WHEN attemptjob-aholds host scopeindex-42open andjob-bopens under it, THENjob-bfailsTransientnamingjob-a, andjob-aalone holds the owner.run.own.pinned-plan-changed: WHEN a host scope’s pending owner pins plan referenceplan-aand an open asks forplan-b, THEN the open refuses withExecutionPinMismatchand its run row closesfailed.run.own.rewind: WHEN jobscore’s pending owner pinsplan-a, a fire askingplan-brefuses, and an operator rewinds that execution, THEN the next fire claims a fresh execution underplan-band completes.run.own.apply-retires-superseded: WHEN applied v2 changes the model a build job targets while its pending owner pins v1’s plan reference, THEN the apply retires that owner and the job’s next fire completes.run.own.backpressure: WHEN a source serves three pages, THEN the destination stages part 0 after the first pull, part 1 after the second and part 2 after the third, and one commit names all three.run.own.staged-bytes: WHEN each of three pulls of a replacing table stages a 400 MiB part, THEN the third stage fails the runPermanentwithRunStagedBytesExceeded, the cursor stays where it stood, and the next fire pulls all three pages under a fresh execution.run.own.checkpoint-commit: WHEN each of sixteen pulls stages a 128 MiB part, THEN one fire commits four times, four parts each, carrying the positionsp4,p8,p12andp16.run.own.checkpoint-frontier: WHEN five pages of clocks1 1,1,2,2and3each stage a 512 MiB part, THEN the run commits at positions 1, 2 and 3, and no instant’s rows span two commits.run.own.checkpoint-id-length: WHEN a run id of 122 chars stages sixteen 128 MiB parts, THEN the run fails withRunCheckpointIdTooLongand commits nothing.run.own.checkpoint-landed: WHEN a run commits checkpoints atp4andp8and its tenth pull fails, THEN itsfailedrow counts eight rows.run.own.checkpoint-resume: WHEN a run commits checkpoints atp4andp8and dies pullingp9, THEN the next fire pulls fromp8and lands every later row once.run.own.checkpoint-scope: WHEN an appending table’s next-URL walk dies on its third page and the listing gains a first row, THEN the next fire lands each listed row once.run.own.stage-schema: WHEN runrun-sstagestsas text and fails, THEN a later run landstsas a timestamp, andrun-s’s commit refuses withStoreSchemaIncompatible.run.own.stage-commit-seq: WHEN runrun-sstages a part, runrun-cthen commits as_commit_seq1, andrun-scommits after it, THEN every row ofrun-sreads_commit_seq2.run.own.stage-instant: WHEN runrun-astages keykat 00:01, runrun-bcommitskat 00:03 andrun-acommits at 00:05, THEN the keyed read returnsrun-a’s row,_ingested_at00:05.run.own.stage-discard: WHEN a run stages its first page and its second pull fails, THEN its run directory holds no staged part.
unsettled: Which pass collects the staged parts of a run that crashed before its commit, and after what age? owner: run-path affects: run.own
cancel
The engine stops work at run or pipeline scope through a shared cancellation token and records the remaining durable state.
catalog-channeltesttestThe catalog stores each stop request on the run row as requested-at instant, scope and optional reason, a newer request overwriting an older mark. The catalog is the only stop-request channel.two-grainstestArun-scoped stop halts one run and the fire continues; apipeline-scoped stop halts every in-flight run of the pipeline and every table or chunk the fire had left. An unrecognized stored scope reads asrun.host-graintestApipeline-scoped stop on a host execution’s run halts every in-flight run under the same host-declared scope and no run of another scope.one-tokentestEach run holds one cancellation token that every await selects on: the pull, a retry sleep, an awakeable wait and a subprocess wait. A meter reservation holds no wait of its own, underacquire-call.Why
because a stop observed only at the pull leaves sleeps and child processes running after the record reads canceledpoll-intervaltestThe engine updates the token from one catalog read before the run’s first await, then every 500 ms. Each await evaluates cancellation before work.engine-keepertestOne keeper per engine renews every registered execution’s owner lease and feeds its token, sleeping until the earliest deadline; opening an execution registers it, and close or drop deregisters it.keeper-panictestA keeper job that panics warns and leaves the keeper running: every other registration keeps its cadence, the panicking one retries at its next deadline, and dropping it returns.Why
because the keeper is shared, so one adapter’s panic that ended its thread lapses every open execution’s lease under a live runland-path-uncuttestA table run’s own stop does not interrupt landing. The engine finishes landing bytes already received and records the unfulfilled stop. Prepared body outputs additionally requirebody-parent.abandoned-worktestAbandoned work surfaces as theCanceledtag through the ordinary failure path, which settles the request ledger, closes the record and leaves the position alone.child-reapedtestA stop signals a running subprocess chain’s process group. The engine recordscanceledonly after reaping the group.Why
because a record reading canceled while a child still runs misstates what the machine is doingstorage-bliptestA failed poll read warns and keeps polling.not-in-flighttestOnly a row holding a non-terminalRunStatusaccepts a mark; a stop matching none raisesCancelTargetNotInFlight, answering409over HTTP and exiting non-zero at the terminal.distinct-terminal-statustestcanceledis a terminal status apart fromfailed, and upstream health observations skip it.resumable-remainstestA stopped run’s recorded pull keeps its owner and replays next attempt; a stopped pull that recorded nothing releases it. A stopped chunk returns to pending, its attempt count unchanged. A stop advances no position.authority-from-the-recordtestA stop authorizes against the pipeline read off the run record, never off the request.stop-unauthorizedtestA credentialed stop needs anexecutegrant covering the pipeline read off the run record; an uncovered or unrecorded run raisesCancelUnauthorizedwithout naming the pipeline, answering403over HTTP.Why
because a stop halts workexecutefires, and a refusal naming the pipeline discloses it to a credential that cannot read itstop-routetestPOST /runs/:run_id/stopwrites a stop under a per-request network credential, its optional JSON body carryingscopeandreason, and answers200with the marked run ids.
Scenarios
run.cancel.engine-keeper: WHEN 20 executions are open on one engine, THEN one keeper thread renews all 20 owner leases, and it exits once the last one closes or drops.
record
The store records run status, owner leases and execution history. History requests select a time window and row ceiling.
reserved-tabletestThe durable run record is a reserved store table holding two append-only rows per run, one at plan and one at commit; the latest row per run and phase wins. The local catalog is a cache in front of it.columnstestA run row is aRunRow.trace-contexttestA run a command opens takes as trace id the trace of the caller’straceparent, read fromTRACEPARENTandTRACESTATE, or a fresh trace; both run rows carry it.run-spantestWith an exporter configured, a run exports onecontextful.runspan, child of the caller’s span, carrying run id, pipeline, table, status, row and byte counts, and the failure tag.job-tracetestA job fire’s execution and each output table run take the tracetrace-contextnames from the fire’s caller, and each table run’s span is a child of the execution’s run span.Why
because a fire’s table runs are its execution’s work, and a table run rooted apart from it hides what the fire landedstatus-settestA run status is aRunStatus, spelled identically on the record and the wire snapshot; every surface classifying a run covers all seven.Why
because two vocabularies for one lifecycle leave a suspended or partly failed run without a readingrow-at-opentestThe engine writes a run row at open, before the first pull. Readers distinguish zero-rowsuccess,failedwith an error kind, and no run row.record-foldstestAn append toreserved-tableruns a pass on it oncetriggersfires; a pass that fails leaves the row appended and the append succeeding, and the next append retries it.rebuild-restores-historytestA catalog rebuild restores run history fromreserved-table, so a rebuilt catalog lists every run the store recorded instead of starting empty.Why
because the catalog is a cache, and a rebuild dropping history loses the only record of failed runstime-traveltestRun history inherits the store’s transaction-time bound, so one as-of selector rewinds it with every other table.owner-leasetestArunningorwaitingrow carries an owner of process id, boot id and a lease expiry, renewed every 10 s with a 30 s time-to-live.Why
because a second process sharing the catalog needs a liveness signal to tell a live run from an orphanorphan-reaptestStartup marks only non-terminal rows with expired owner leases aspartial_failure. Startup leaves rows held by live processes unchanged.writing-sitetestEvery landed row carries its run’s site id in the_site_idcolumninjectedwrites, so a row names the site that wrote it.site-id-lengthtestA site id matches letters, digits, dot, underscore and hyphen, from 1 chars to 64 chars.Why
because it is interpolated into run directories and request-ledger filenamessite-id-unresolvedtestA site id comes from the manifest’ssite_id, or the variable itssite_id_envnames, and a run’s--site-idor--site-id-envreplaces that declaration; no declaration, both keys in one place, or an unset variable raisesSiteIdUnresolvedat startup.Why
P3history-windowtestHistory requests specify an inclusive start-instant lower bound and row ceiling, returning newest first. A SQL catalog applies the bound inWHEREover an indexed start column and the ceiling asLIMIT.Why
because filtering after the read scans every run a pipeline ever recorded, so a describe call grows with history rather than with its windowdescribe-windowtestThe describe surface answers with 5 rows by default and clamps a caller’s ceiling at 500 rows.bound-spellingtestA window lower bound isYYYY-MM-DDor a UTC RFC3339 instant endingZ; any other spelling raisesHistoryBoundSpellingat every caller-facing surface.Why
because a bound read under a guessed zone silently moves the windowtruncation-flagtesttestA history response echoes its window and flags truncation when the pipeline recorded more runs inside it than the ceiling returned; an empty window answers[]and raises nothing.history-exporttestHistory leaves the machine as a process listing and an NDJSON export over one run projection; the export’s first line carries store, window, count and truncation flag, then one run per line. It resolves no bucket.export-ceilingtestThe export returns 500 rows by default and clamps at 5000 rows. Each pipeline’s window takes the full ceiling; the export clips the merged result once.error-captestThe shared run projection masks credential-shaped spans in recorded errors and caps each error at 2 KiB with an ellipsis marker. Every surface serves that projection.failure-publishestestThe bucket push runs on the failure arm as on the success arm, for every failure past run open.skipped-counttesttestThe run row counts rows and bytes at the destination and sums each pull’s optionalskippedcount of inputs the source declined to land whole, replayed pulls included.Why
because a skipped input raises no error, and a count on the record surfaces it without reading the landed rowssweep-boundstesttestA store-driven run’s row carries its input asInputBounds: the resolvedas_of, snapshot ids and row count, and for a sweep origin the bounds it read under, the sweep id and the origin.Why
because a score reported per origin is auditable only when each run names the bounds it read underplan-identitytestA run row retains its admitted plan reference after owner retirement; a record lacking that field decodes with no plan identity.schema-diff-hometestA table’s run manifest holds its committed downgrade events with its parts and cursor; run history reads them under that run id.schema-diff-shapetestEach downgrade event names the table, column path, source type, landed type and reason; a stage that does not commit contributes none.
Scenarios
run.record.schema-diff-home: WHEN a nested column downgrades and its table commits, THEN run history and that table’s run manifest carry the same event; a failed stage carries none.
project
The runner publishes a best-effort live view of run progress to subscribers.
best-efforttesttestLive run state is a best-effort in-memory projection of events the runner emits after durable state changes; execution never reads it, and a restart discards it, so a subscriber recovers from the durable record.wire-snapshottestThe wire snapshot is aSnapshot, its status fromstatus-set. Every deploy target emits it identically.step-viewtestA step view is aStepView, its status aStepStatus.stopped-steptestA stopped run reports its executing stepfailedwith theCanceledtag and leaves the run-level error unset.emission-never-blockstestProgress emission is no journal step: events reach an in-process observer over a bounded channel that drops when full and never blocks the runner.terminal-slottestA terminal transition bypasses the lossy channel through a per-run slot that is never dropped; a projection reading non-terminal behind a terminal run record reconciles from the record.Why
because a dropped terminal event otherwise leaves a finished run reading running forevercoalescingtestThe hub coalesces snapshot broadcasts to at most one per 100 ms, keeping only the latest. A terminal transition flushes immediately. The hub uses an injected clock.metadata-too-largetestA metadata write whose serialized size exceeds 16384 B raisesRunMetadataTooLargenaming size and bound, and the snapshot stands unchanged.Why
because a silently clipped map reads as the complete metadataversiontestThe snapshot version is(epoch, counter), the epoch being the hub’s start instant, compared lexicographically; a client discards any delta at or below the highest version it holds.Why
because a bare counter restarts at zero with the process and freezes every client’s viewreducer-is-totaltestThe reducer is total: nothing overwrites a terminal step or run status, a new step label appends, an existing one merges field-wise, and a stale delta is a no-op.outputs-by-referencetestA step output appears as a post-redaction reference and a byte count, never inline; fetching the payload is a separately authorized request.connecttesttestA subscriber receives the folded snapshot and its update receiver under one lock, missing and duplicating no update; a late joiner may instead start from the oldest snapshotbroadcast-ringholds, reading each in order before live updates.broadcast-ringtestThe per-run broadcast holds 256 entries; a subscriber that overruns it resynchronizes to the latest snapshot.channel-evictiontestThe hub evicts a run’s channel 60 s after it folds the run’s terminal event; a later subscriber recovers from the durable record.Why
because a channel allocated per run otherwise grows a long-lived process by every run it has seenunauthenticated-upgradetestThe run-stream socket,GET /runs/:run_id/stream, authenticates on the HTTP request before the upgrade; a missing or invalid credential raisesRunStreamUnauthorizedwith401, and an unseen run answers404.read-only-sockettestA subscription permits reads only. Stop requests use an authenticated route.
sweep
The scheduler runs one store-driven job per origin across a declared span. Each origin reads data available at that origin under one frozen transaction-time bound.
originstestA store-driven[[job]]declaringSweepopens one run per origin from its start through its end at each step ingrammarunits, each reading withavailable_as_ofequal to its origin.Why
because an origin reading rows published after it scores a forecast against its own answersweep-idtestA sweep’s id is the execution id of the fire that opens it; every origin’s run records it, so run history groups one sweep’s origins.frozen-as-oftestEvery origin of one sweep reads at oneas_of: the job’s declaredas_of, else the instant the sweep opens, recorded once asopen-as-ofrecords it for resumes.origin-after-opentestA sweep whoseendis later than itsas_ofraisesRunSweepOriginAfterOpen, at validation for a declaredas_ofand at sweep open otherwise, and opens no origin.Why
because an origin past the frozen corpus reads nothing newer than it, and that empty answer looks like a forecastcarried-statetestUnderstate = "carried", origins run in ascending order one at a time, each reading the job’s outputtablesat the commit of the preceding origin and every other table at the sweep’sas_of.fresh-statetestUnderstate = "fresh", the default, every table an origin reads, its output tables included, resolves at the sweep’sas_of, so no origin reads a row another origin wrote.simulated-writetestA row an origin lands into a table declaringavailable_timecarries the origin as its availability instant, so a later origin reads it and an earlier origin never does.Why
because notes an agent wrote at a 2014 origin were available in 2014, not at the instant they committedorigin-pintestAn origin’s plan reference hashes the sweep declaration and its origin beside the fieldsinput-pinhashes, so a resume under a changed sweep meetspinned-plan-changed.no-collapsetestA sweep runs every declared origin, regardless of elapsed time.catch-upcollapses no origin. A failed origin in acarriedsweep prevents every later origin from opening.Why
because each origin is a distinct question, and a collapsed or skipped origin drops silently from the scorecache-keytestNo recorded or cached value crosses origins: each origin runs under its own execution id, which keysentry-key, and carries its origin asavailable_as_ofamong the boundsresult-keykeys on.Why
because a cache keyed without the bound passes one origin’s answer to another
Scenarios
run.sweep.origins: WHEN afreshsweep of 12 origins,7dapart from2014-03-05T00:00:00Z, reads a 2026-backfilled table holding a planted row available2014-04-30T00:00:00Z, THEN origins 1 to 8 return no planted row and origins 9 to 12 return it.
unsettled: Does a carried-state sweep resume from the last committed origin or replay from the start? owner: run-path affects: run.sweep
orchestrate
Each deployment supplies a durable orchestrator implementing the execution interface. This section specifies provider steps and the engine behavior inside each step.
roletestA target’s orchestrator implements every operation insubstrate-port. The engine runner overstorage-portsis the reference implementation.provider-steptestA provider adapter runs one provider step per table pull or backfill chunk, recording that table’s or chunk’s commit pointer. The engine’s journal entries underentry-keyremain inside the provider step.semanticstestEveryrun.journal,run.retry,run.suspendandrun.cancelclause holds inside a provider step. An adapter’s conformance suite demonstrates these clauses before a target profile names the adapter.resumetestA provider replay of a completed step returns its recorded commit pointer and re-enters nothing; a crash inside a step resumes under the engine’s entry keys, so the parts land byte-identical to an uninterrupted run underreplay-lands.Why
because a replay the provider owns below the commit pointer lands per-target bytes, and the proofs over the journal then cover no managed targetone-retry-layertestA provider adapter disables provider per-step retries. The schedule inone-layeris the only retry layer. A failed step closes its table or chunk as the engine closes it.Why
because a provider retry around an engine schedule multiplies attempts against a shared quotadelegate-insidetestEvery orchestrator runs a delegate underbindinginside the engine’s journaled step within a provider step. No adapter dispatches a delegate as a separate provider step.Why
because a delegate reached outside the engine’s step escapes the entry key, the fence and the item budgetstep-commandtestcontextful pipeline step <id> --run-id <step>runs a table pull or, with--chunk <n>, chunk n of the table’s seed plan, and prints its commit pointer; attempts name runs aspipeline rundoes under<step>, replays under<step>.replay-<k>.Why
because a replay named as the engine runner names its unit lands that runner’s bytesstep-leasetestA provider step opens its attempt’s owner lease and releases it on return, so no lease spans two steps; a replay while an earlier attempt’sowner-leaseruns refuses naming that attempt.Why
because no engine process runs between two steps to renew a lease
Scenarios
run.orchestrate.resume: WHEN a backfill chunk’s provider step crashes after half its pages and the provider replays the step, THEN the engine resumes from its recorded pages and the chunk’s parts match the uninterrupted run’s bytes.run.orchestrate.one-retry-layer: WHEN a provider step’s pull meets a throttle, THEN the engine’s schedule retries inside the step and the provider records one step attempt.run.orchestrate.step-lease: WHEN a chunk step’s process dies and the provider replays it before the dead attempt’s owner lease lapses, THEN the replay refuses naming that attempt, and a replay after the lapse resumes it.run.orchestrate.delegate-inside: WHEN a job step’s process dies while its second item’scallis unanswered and the provider replays the step, THEN the delegate receives that call alone again, under its first idempotency key.
unsettled: Where does a target profile record a provider’s step ceiling, so a backfill whose chunk plan passes it refuses at planning? owner: topology affects: run.orchestrate
Shapes
Run statuses: