Docs / Specification

contract run · 30-run.md · 31-pipeline.md · 32-derive.md · guide · card

The derive tier

Derive tasks process committed rows and append results through the ordinary write path. Implementers use this contract for task selection, engine bindings and output rows. Media tasks produce citable passages; link tasks extract document head fields and advertised images.

Derive tasks read parent rows, invoke an engine and write output rows:

select

The derive source selects eligible parent rows each tick and limits each run’s work.

• required-keytestMissing or blank source_table or parent_id_column raises DeriveConfigKeyMissing, naming the key and pipeline. For built-in tasks, missing or blank engine or media_column raises the same error.
  Why

  A-run#derivation-key-decides
• no-store-roottestA source built without a store root or without its pipeline id raises DeriveNoStoreRoot, naming which is absent.
  Why

  A-run#plan-compiled-content
• foreign-output-tabletestThe anti-join’s pipeline id comes from the build alone; a config key naming another pipeline’s output raises DeriveForeignOutputTable.
  Why

  A-run#derivation-key-decides
• rows-per-runtestmax_rows_per_run truncates the outstanding list after the anti-join, at 25 rows by default; zero or below resolves to that default.
• sampletesttestsample_rate r keeps a host unit before the anti-join when the first eight bytes of SHA-256 over its key then task name, read big-endian and divided by 2^64, fall below r; r outside (0, 1] refuses as spec-invalidrun.declare.spec-invalidA manifest file the canonical type cannot deserialize raises PipelineSpecInvalid, naming the file, the key path and the value found..
  Why

  because a hash of the unit and the task decides membership without stored state, so every pass and every resumed pass selects one set
• sincetesttestsince, an RFC 3339 instant, keeps a host task’s parent rows whose _ingested_at falls at or after it, before the anti-join; a row without a readable _ingested_at is dropped, and another since refuses as spec-invalidrun.declare.spec-invalidA manifest file the canonical type cannot deserialize raises PipelineSpecInvalid, naming the file, the key path and the value found..
  Why

  because a bound on landing time limits a pass to recent parents, and an unstamped row has no landing time to compare
• filtertesttesttestfilter keeps a host task’s parent rows meeting every condition, a scalar testing equality and a list membership, before sampling; a column outside the key and declared columns, or a condition filter-valuesread.retrieve.filter-valuesFilter values are strings, numbers or booleans; membership lists contain at least one value. Nulls, nested objects, nested lists and empty lists are malformed conditions. or filter-budgetread.retrieve.filter-budgetA filter holds at most 32 conditions, each membership list at most 256 entries, and its serialized JSON, kinds included, at most 16 KiB. refuses, refuses as spec-invalidrun.declare.spec-invalidA manifest file the canonical type cannot deserialize raises PipelineSpecInvalid, naming the file, the key path and the value found..
  Why

  because conditions over columns the unit already carries narrow a pass without another read, and a column the task never reads holds nothing to compare
• ordertesttesttesttestorder_by, the key or a declared column then asc or desc, orders a host task’s outstanding units before rows-per-runrun.select.rows-per-runmax_rows_per_run truncates the outstanding list after the anti-join, at 25 rows by default; zero or below resolves to that default., nulls last and ties by unit key; selection holds at most twice that budget in units.
  Why

  because a key tie-break gives every pass and every resumed pass one order, and cutting back whenever the buffer fills bounds memory below the parent count
• attempts-per-unittestmax_attempts defaults to 3 attempts per unit. A non-positive value selects that default.
• seconds-per-runtestmax_seconds_per_run bounds the serial unit loop at 300 s by default for link_preview and carries no default for transcribe.
• incomplete-unittestIf a parent row’s key or media column is null, empty or whitespace, the derive source raises DeriveUnitIncomplete. The derive source skips the row and counts the row on the run record.
  Why

  P4
• journaled-pulltestA derive pipeline configured to journal its pulls raises DeriveJournaledPull.
  Why

  A-authority#prepared-record
• unmetered-granttestA derive pipeline declaring a shared-quota grant for a task other than link_preview or a host task declaring pure false raises DeriveUnmeteredGrant.
  Why

  A-connector#host-access-quota
• metered-clienttestA link_preview pipeline opening a socket outside the mediated client raises DeriveMeteredClient; every request it makes enters the run’s request ledger.
  Why

  A-connector#host-access-quota
• anti-jointestEach tick recomputes the outstanding set: every parent row holding neither a passage nor a settled marker under its current derivation-keyrun.emit.derivation-keyEvery passage and marker row carries derivation_key: lowercase hex SHA-256 over the engine id, the binding less its bounds and zone, the output table's declared columns, and the parent row's id, media value and derivation_key. in the pipeline’s own output table, or, for a host task, its marker table.
  Why

  A-run#derive-work-set
• unit-parentstestA host task reads each joined table once, before the anti-join and before processing its first unit. Each unit carries every row whose join-column value equals that unit’s key. Rows landed after that read are absent.
  Why

  A-run#derivation-key-decides, A-run#selection-stays-bounded
• join-keytesttesttestA host task that joins tables also hashes each unit’s joined rows into its task-versionrun.emit.task-versionA host task's derivation_key hashes its registered name, its task_version and the parent row's id, declared columns and derivation_key, so raising the version re-selects every settled unit. key, each table’s rows sorted over its read columns, so a joined row landing, changing or leaving re-derives that unit alone.
  Why

  A-run#derivation-key-decides
• join-keyedtestA task reads joined tables declaring primary_key through dedup-viewstore.declare.dedup-viewA keyed table reads through ROW_NUMBER() OVER (PARTITION BY <pk> ORDER BY <order_by> DESC, _ingested_at DESC, _run_id DESC, _row_seq DESC) = 1 over the current snapshot, if any, unioned with the committed runs it omits.. Each unit joins one surviving row per key, excluding superseded versions.
  Why

  because a raw file read hands a re-landed row in both versions, and the admitted read hands one
• join-absenttestA joined table the store holds no schema for fails the tick with a Config failure, settling no unit, so the next tick selects every unit again.
  Why

  because an absent table read as empty settles each unit ok on a partial join that the anti-join never revisits
• latest-markertestA unit’s state under a key comes from its latest marker ordered by _ingested_at, _run_id and _row_seq. The highest attempt count does not determine that state.
  Why

  because an empty marker records 1 attempt, so ranking by count lets an older retry revive a settled unit
• key-changetestRows under the current key landed before the unit’s latest ok or empty landing under another key count for nothing, so a key changed and changed back derives the unit again.
  Why

  A-run#derivation-key-decides
• derive-ordertestA derive pipeline reading another derive pipeline’s output table runs after that parent on the parent’s tick, regardless of declaration order or its own schedule.
  Why

  A-run#chained-derive-pipelines
• parent-outcometesttestA derived child starts only once its parent produces a completed run outcome, a failed one included, and then reads only the parent’s committed rows; a launch, wait or signal failure, or a worker dispatch error, stops the unit first.
  Why

  A-run#chained-derive-pipelines
• join-ordertesttestA host task joining another derive pipeline’s output table runs after that pipeline on its tick, as derive-orderrun.select.derive-orderA derive pipeline reading another derive pipeline's output table runs after that parent on the parent's tick, regardless of declaration order or its own schedule. orders a source table’s reader; among several such parents on one chain, it runs after the last.
  Why

  A-run#chained-derive-pipelines
• parents-divergetestA derive pipeline whose source-table and joined parents lie on no one dependent run raises DeriveParentsDiverge, naming them, and arms none.
  Why

  because a dependent run orders one chain from its head, so parents on two chains share no tick
• derive-cycletesttesttestA build whose derive source-table or join dependencies return to a pipeline raises DeriveCycle, names every pipeline on the cycle and arms none.
  Why

  A-run#chained-derive-pipelines
• derive-after-conflicttestA derive child whose after names a pipeline other than the derive parent it runs after raises DeriveAfterConflict and arms none.
  Why

  A-run#chained-derive-pipelines
• parent-scantestThe derive source scans parent tables of at most 1000000 rows in memory. For larger tables, the source streams through the read engine in batches, holding one batch in memory.
  Why

  A-run#derive-work-set
• dry-runtestpipeline run --dry-run on a derive pipeline prints its eligible, already-derived and outstanding unit counts from one selection, calling no engine and landing no row.
  Why

  A-run#selection-stays-bounded
• available-selecttestA derive run carrying available_as_of selects only parent rows available-as-ofstore.available-time.available-as-ofavailable_as_of filters the inner source with <column> <= available_as_of under beneath-enforcement, combining with as-of and valid-as-of when the read supplies them. admits at that bound, and each derived row lands its parent row’s availability instant.
  Why

  because content derived from a later source record otherwise answers an earlier horizon

Scenarios

• run.select.derive-orderrun.select.derive-orderA derive pipeline reading another derive pipeline's output table runs after that parent on the parent's tick, regardless of declaration order or its own schedule.: WHEN configuration declares a scheduled child before its parent and the child reads the parent’s output, THEN the parent’s tick lands its new row before the child reads it.
• run.select.parent-outcomerun.select.parent-outcomeA derived child starts only once its parent produces a completed run outcome, a failed one included, and then reads only the parent's committed rows; a launch, wait or signal failure, or a worker dispatch error, stops the unit first.: WHEN a parent fails after earlier rows committed, THEN its child reads those committed rows in the same tick.
• run.select.join-keyrun.select.join-keyA host task that joins tables also hashes each unit's joined rows into its task-version key, each table's rows sorted over its read columns, so a joined row landing, changing or leaving re-derives that unit alone.: WHEN a keyed note one unit joins is re-landed with new text, THEN the next tick derives that unit under a new key and leaves the other units’ markers standing.
• run.select.join-orderrun.select.join-orderA host task joining another derive pipeline's output table runs after that pipeline on its tick, as derive-order orders a source table's reader; among several such parents on one chain, it runs after the last.: WHEN a scheduled host task declared first joins the words another derive pipeline lands, THEN one cycle hands it the words that pipeline landed in the same tick.

bind

Engine bindings select registered tasks, configure engine interfaces and constrain media access and returned values.

• unbound-enginetestAn engine with no [derive.<name>] block raises DeriveEngineUnbound, naming the pipeline, the engine, the file to edit and every bound engine.
  Why

  A-run#derive-engine-machine
• command-in-manifesttestA Binding key preprocess, env or allow_hosts, or a step’s command, inside [pipeline.source.config] raises DeriveCommandInManifest.
  Why

  A-run#derive-engine-machine
• unknown-tasktesttask names transcribe, the default, link_preview, or a task the embedding binary registered; any other value raises DeriveUnknownTask, listing the built-in and registered names.
  Why

  A-run#derive-engine-machine
• host-tasktestAn embedding binary registers each compiled derive task by name before build. Configuration selects that registered name without supplying a command. A duplicate built-in or registered task name raises DeriveTaskNameTaken.
  Why

  A-run#derive-engine-machine
• join-repeatedtestA host task whose joins name one table twice raises DeriveJoinRepeated at registration.
  Why

  because a unit holds one joined list per table, so the second join’s rows replace the first’s
• pure-tasktesttesttesttesttestA host task is pure unless it declares pure false. The engine hands a pure task no client, and its request for one refuses as metered-clientrun.select.metered-clientA link_preview pipeline opening a socket outside the mediated client raises DeriveMeteredClient; every request it makes enters the run's request ledger., failing the tick; the engine intercepts no socket the task’s own code opens.
  Why

  because compiled task code shares the engine’s process, so the client the engine hands it is the one reach the engine withholds
• reach-clienttesttesttestA host task declaring pure false receives the run’s metered client for every unit; a run binding none refuses its request as metered-clientrun.select.metered-clientA link_preview pipeline opening a socket outside the mediated client raises DeriveMeteredClient; every request it makes enters the run's request ledger..
  Why

  A-connector#host-access-quota
• reach-granttesttestUnder a grant, the metered client of a host task declaring pure false reserves one permit from the grant’s shared quota before each request and model call, so every pipeline naming that quota draws from one limit.
  Why

  A-connector#host-access-quota
• model-endpointtesttestA host task declaring pure false names its model-endpointconnector.infer.model-endpointEvery model call resolves to one operator-configured OpenAI-compatible HTTP endpoint declared as a capability. The host owns the credential, rate limit and span; the component owns the prompt template and response schema. with endpoint and model. The run’s metered client reaches that endpoint’s host alone, and each request it sends enters the run’s request ledger under the marker table’s run id.
  Why

  because a compiled task shares the engine’s process, so the client the engine hands it is the one route the engine meters
• model-keystesttestendpoint or model beside a built-in or pure task refuses as config-keyrun.declare.config-keyA source config key outside the set that source enumerates raises PipelineUnknownConfigKey before any I/O, naming the key and the accepted keys; request headers and the forwarded guest table are exempt., and either without the other refuses as spec-invalidrun.declare.spec-invalidA manifest file the canonical type cannot deserialize raises PipelineSpecInvalid, naming the file, the key path and the value found..
  Why

  because a task the engine hands no client reaches no endpoint, and an endpoint without a model names no call
• model-fencetesttestA host task’s model call sends the task’s template in the system role and each value it reads inside data-fenceconnector.infer.data-fenceNo ingested value reaches a model outside its data boundary markers: a pair whose opening marker carries the value's provenance label, a preamble declaring the blocks data, and a closing line restating the caller's rules. markers carrying its provenance label, answering the content beside its output-taintconnector.infer.output-taintModel output carries the least-trusted provenance label among its marked inputs, and lands under that label. label.
  Why

  because a parent row is third-party data, and a value outside the markers reads to a model as instruction
• off-endpointtesttestA host task’s request to an address off its endpoint’s origin refuses as metered-clientrun.select.metered-clientA link_preview pipeline opening a socket outside the mediated client raises DeriveMeteredClient; every request it makes enters the run's request ledger. before any byte leaves, failing the tick whether or not the task returns the error.
  Why

  because the refusal repeats on every retry, and a unit settling past it records a derivation its task never completed
• driver-mismatchtestdriver is exec or "none" behind transcribe and fetch behind link_preview, and a host task names no engine; any other pairing raises DeriveDriverMismatch.
  Why

  A-run#derive-engine-machine
• endpoint-host-baretestAn endpoint_host carrying a path, query, port or scheme raises DeriveEndpointHostNotBare.
  Why

  A-run#derive-engine-machine
• attach-unboundtestAn attach table on a fetch binding, or on a binding declaring no endpoint_host, raises DeriveAttachUnbound, naming the engine.
  Why

  because exec-proxyconnector.attach.exec-proxyCONTEXTFUL_EGRESS_URL names a host loopback proxy, http://127.0.0.1:<port>, open for one pull. The proxy sends each request's method, path, query, headers and body to https://<endpoint_host> through the mediated client, following no redirect. attaches a credential toward one declared vendor host, and a fetch reader reaches third-party hosts
• remote-url-unsupportedtestAn http or https media value for an engine declining remote addresses raises DeriveRemoteUrlUnsupported, naming the when = "media_is_url" preprocess step.
  Why

  A-run#derive-engine-machine
• media-unreadabletestA media value that is neither an address nor a readable local file raises DeriveMediaUnreadable, failing that unit alone.
  Why

  A-run#derive-engine-machine
• media-roottestThe engine canonicalizes local media paths under the binding’s media_root (declaration-basestore.init.declaration-baseA located project resolves relative declaration paths, a derive binding's media_root and path-form command[0] included, against its base directory rather than the working directory.), defaulting to that declaration base. A path outside the root raises DeriveMediaOutsideRoot and fails only that unit.
  Why

  because a parent row is third-party data, and an unconfined path reads any file the machine’s user can
• confidence-rangetestA confidence outside 0.0..=1.0 after adapter normalization raises DeriveConfidenceOutOfRange and lands null.
  Why

  A-run#derive-engine-machine
• upstream-excerpttestAn Upstream excerpt holds at most 4 KiB and no request body.
• engine-unavailabletestIf a transcriber returns EngineUnavailable, the run ends. If a LinkReader returns EngineUnavailable, the derive source raises DeriveLinkEngineUnavailable and fails only that unit.
  Why

  P6
• advisory-zonetestA row zone taken from the binding’s advisory zone key raises DeriveAdvisoryZone.
  Why

  A-run#derive-engine-machine
• locality-widertestAn engine whose declared locality is wider than the zones its source table admits raises DeriveLocalityWider at build, naming the engine, its locality and the table.
  Why

  because a derived passage carries its parent’s content, and an engine outside the parent’s zones moves that content where the parent cannot go

exec

The executor runs operator-declared argument arrays as local processes with pinned binaries, restricted environments and bounded runtime and output.

• shell-commandtestA string-valued command raises DeriveShellCommand; no-shellrun.exec.no-shellA step runs as its argument array with no shell, in a process group of its own, under a cleared environment holding only the binding's env table. runs the argument array.
  Why

  A-run#derive-engine-machine
• env-nametestAn allowlist name that is not ASCII alphanumeric or underscore, or that leads with a digit, raises DeriveEnvNameInvalid.
  Why

  A-run#derive-engine-machine
• env-credentialtesttestAn env value embedding a ${secret://<name>} placeholder raises DeriveEnvCredential, naming the engine and variable; a step reaches its vendor through exec-proxyconnector.attach.exec-proxyCONTEXTFUL_EGRESS_URL names a host loopback proxy, http://127.0.0.1:<port>, open for one pull. The proxy sends each request's method, path, query, headers and body to https://<endpoint_host> through the mediated client, following no redirect..
  Why

  A-connector#exec-egress-proxied
• chain-deadlinetesttimeout_secs bounds one unit’s whole chain at 1800 s by default.
• captured-outputtestmax_output_bytes bounds each step’s captured standard output and error at 8 MiB by default.
• audit-entriestestA chain contributes at most 64 entries of captured output to the run record.
• step-error-excerpttestA failing step’s error text carries at most 4 KiB of its standard error.
• non-zero-exittestA step exiting non-zero raises DeriveStepExit carrying its bounded error text into the run audit.
  Why

  A-run#derive-engine-machine
• deadline-elapsedtestIf a chain exceeds its deadline, the executor raises DeriveStepTimeout, naming the running step.
  Why

  A-run#derive-engine-machine
• output-captestThe executor counts and discards output past the bound without buffering. Crossing the bound kills the step and raises DeriveOutputCap with the byte count. The executor continues draining so the child does not block.
  Why

  A-run#derive-engine-machine
• silent-steptestA preprocess step exiting zero without writing its output file raises DeriveStepProducedNothing.
  Why

  A-run#derive-engine-machine
• missing-binarytestA command[0] that is not an executable file on this machine raises DeriveBinaryMissing, naming the binary and the step.
  Why

  A-connector#executables-content-pinned
• unpinned-pathtestA path-form command[0] without sha256 raises DeriveUnpinnedPath, quoting the computed digest. A bare search-path name carries no digest.
  Why

  A-connector#executables-content-pinned
• digest-mismatchtestA pinned file whose bytes differ from its recorded digest raises DeriveDigestMismatch.
  Why

  A-connector#executables-content-pinned
• no-shelltestA step runs as its argument array with no shell, in a process group of its own, under a cleared environment holding only the binding’s env table.
  Why

  A-run#derive-engine-machine
• step-conditiontestA preprocess step runs only while its when holds: media_is_url for an http or https input, engine_requires_pcm16_wav for an input other than 16-bit PCM WAV; another name raises DeriveStepConditionUnknown.
  Why

  because a misspelled condition otherwise skips or runs its step silently
• verified-spawntestBefore each spawn, the executor re-reads the step’s binary and compares its bytes with the digest resolved at run start. A mismatch follows digest-mismatchrun.exec.digest-mismatchA pinned file whose bytes differ from its recorded digest raises DeriveDigestMismatch.; matching bytes permit execution.
• engine-idtestAn exec engine id reads exec:<name>@<prefix>, the prefix being 12 chars of lowercase hex over every step’s binary digest and arguments.
• vendor-deadlinetestA vendor engine, a binding naming an endpoint_host, holds request_timeout_secs as its engine step’s deadline, clipped to the time chain-deadlinerun.exec.chain-deadlinetimeout_secs bounds one unit's whole chain at 1800 s by default. leaves, so a stalled vendor fails its unit before the chain elapses.
  Why

  because a chain deadline alone lets one stalled vendor call spend the unit’s whole budget with no step to name

fetch

The fetch driver follows third-party links with host and address checks, bounded redirects, document head scanning and image probes.

• schemetestAn address whose scheme is neither http nor https raises DeriveSchemeUnsupported before any socket opens.
  Why

  A-connector#outbound-transport-origin
• address-literaltestA host written as an address literal raises DeriveAddressLiteral.
  Why

  A-connector#outbound-transport-origin
• redirect-chaintestA redirect chain runs to at most 5 hops.
• binding-keytestA Binding key env, preprocess, engine or max_output_bytes on a fetch binding raises DeriveFetchBindingKey, naming the key.
  Why

  because a silently ignored key leaves an operator believing a fetch carries a credential or runs a process
• document-prefixtestmax_document_bytes bounds the scanned document prefix at 1 MiB by default, dropping the remainder.
• probe-prefixtestmax_probe_bytes bounds one picture range request at 64 KiB by default.
• hop-timeouttestrequest_timeout_secs bounds one hop at 20 s by default.
• retry-after-defaulttestA 429 without a usable Retry-After maps to RateLimited carrying 60 s.
• charsettestA declared character set other than UTF-8 raises DeriveCharsetUnsupported, naming the value.
  Why

  A-connector#untrusted-input-decodes
• not-utf8testIf a document declares no character set and fails UTF-8 validation, the fetch driver raises DeriveBytesNotUtf8. The validator tolerates a character split at the byte bound.
  Why

  A-connector#untrusted-input-decodes
• head-rowstestA link preview lands one passage for non-empty head title or description and one passage per distinct advertised head image, each carrying the scrubbed page URL and each image its probe status.
  Why

  because the head and its image claims remain separately citable, including an image whose probe fails

Scenarios

• run.fetch.head-rowsrun.fetch.head-rowsA link preview lands one passage for non-empty head title or description and one passage per distinct advertised head image, each carrying the scrubbed page URL and each image its probe status.: WHEN a page head declares a title, description and one image, THEN the output holds one text passage and one image passage with its probe status.

emit

Derive tasks emit content rows and completion markers with status, attempt counts, citation keys and provenance.

• failure-off-tabletestPer-unit failure state recorded outside the output table raises DeriveFailureOffTable.
  Why

  A-run#derive-work-set
• unit-statustestunit_status is ok, empty where the engine established nothing to derive, unavailable where it returned nothing and stated no reason, or failed on a typed error; any other value raises DeriveUnitStatusUnknown.
  Why

  A-run#derive-work-set
• reserved-discriminatortestA derive table declaring a column named kind raises DeriveReservedDiscriminator.
  Why

  A-run#derive-work-set
• settled-revivedtestA unit that settles under its current key while a tick derives it raises DeriveSettledUnitRevived, and that tick lands none of its rows; a changed key revives nothing.
  Why

  A-run#derivation-key-decides
• attemptstestattempts is the prior count under the row’s key plus one; unavailable and failed stop at attempts-per-unitrun.select.attempts-per-unitmax_attempts defaults to 3 attempts per unit. A non-positive value selects that default., and empty receives 1 attempts in total.
• unredacted-errortestA non-null last_error write that has not passed address redaction raises DeriveUnredactedError.
  Why

  A-authority#redaction-no-copy
• primary-keytestA derive output table or host marker table without primary_key ["unit_ref", "derivation_key", "cue_seq"], or a host content table whose key does not open with unit_ref and derivation_key, raises DerivePrimaryKeyMissing.
  Why

  because a re-derived unit otherwise lands its new passages over the originals, or beside them with no key telling them apart
• derivation-keytestEvery passage and marker row carries derivation_key: lowercase hex SHA-256 over the engine id, the binding less its bounds and zone, the output table’s declared columns, and the parent row’s id, media value and derivation_key.
  Why

  A-run#derivation-key-decides
• stale-supersedestestA unit’s latest ok or empty landing supersedes rows landed earlier under another key. Those rows answer until that landing. The next fold drops the superseded rows and rebuilds the table’s sidecars without them.
  Why

  A-run#derivation-key-decides, A-run#derived-unit-follows
• task-versiontestA host task’s derivation_key hashes its registered name, its task_version and the parent row’s id, declared columns and derivation_key, so raising the version re-selects every settled unit.
  Why

  A-run#host-registered-derive
• host-rowstestA host task’s content row carries its unit’s unit_ref, derivation_key and task_version, kind passage and unit_status ok; every unit lands one marker, ok over content rows, empty over none.
  Why

  A-run#derive-work-set
• content-emptytestA host unit landing no row in a content table lands one kind marker, unit_status empty row there under its key, so stale-supersedesrun.emit.stale-supersedesA unit's latest ok or empty landing supersedes rows landed earlier under another key. Those rows answer until that landing. The next fold drops the superseded rows and rebuilds the table's sidecars without them. holds in every content table.
  Why

  because a read and a fold supersede from one table’s rows alone, so a content table never seeing the unit’s newest key keeps its stale rows answering
• content-marker-retention-clocktestA generated host marker copies a custom retain-rowsstore.declare.retain-rowsretain_rows = { column = "<name>", age = "<n>d" } takes _ingested_at or a declared Timestamp column; another column or a null value raises StoreRetentionColumnInvalid before landing, and a malformed age refuses the declaration. timestamp from the parent row when the parent carries a same-named column.
  Why

  because a generated empty marker must carry the declared clock before the content table accepts its landing
• output-tablestestA host-task pipeline declares exactly its task’s marker and content tables, and a unit returns rows for its content tables alone; either breach raises DeriveOutputTablesMismatch, the second failing that unit alone.
  Why

  A-run#derive-work-set
• marker-lasttestA host-task run commits each content table under its own commit, then the marker table, and a content table failing stops the fire before its marker lands, so the unit re-runs and its rows collapse by key.
  Why

  A-run#derive-work-set
• version-retainedtestOn an output table declaring retain_versions, stale-supersedesrun.emit.stale-supersedesA unit's latest ok or empty landing supersedes rows landed earlier under another key. Those rows answer until that landing. The next fold drops the superseded rows and rebuilds the table's sidecars without them. holds per unit and task_version, so rows under an earlier task version stay current under that version beside the newer version’s rows.
  Why

  A-run#host-registered-derive
• marker-rowtestA unit yielding no passage lands one marker row, cue_seq -1 and kind marker, carrying its status, attempts, last error and whether it retries.
  Why

  A-run#derivation-key-decides
• empty-documenttestOnly a WebVTT document whose blocks are its header, notes and styles establishes nothing to derive; empty output, or blocks none of which parse, lands unavailable.
• canceled-unittestA unit whose chain a run stop interrupts lands no row and charges no attempt; the pull ends Canceled once child-reapedrun.cancel.child-reapedA stop signals a running subprocess chain's process group. The engine records canceled only after reaping the group..
• output-modalitytestA text passage extracted from a video parent carries _modality text, and an advertised image passage carries image, each under modalitystore.reserve.modality_modality takes text, image, audio, structured or mixed; another value fails its batch's validation..
  Why

  A-run#passage-s-modality
• local-content-keytestA local media unit’s derivation-keyrun.emit.derivation-keyEvery passage and marker row carries derivation_key: lowercase hex SHA-256 over the engine id, the binding less its bounds and zone, the output table's declared columns, and the parent row's id, media value and derivation_key. includes a SHA-256 digest of canonical file bytes, so changed bytes under an unchanged path reselect the unit.
  Why

  A-run#derived-unit-follows
• derived-idtestEvery passage and marker row keyed as primary-keyrun.emit.primary-keyA derive output table or host marker table without primary_key ["unit_ref", "derivation_key", "cue_seq"], or a host content table whose key does not open with unit_ref and derivation_key, raises DerivePrimaryKeyMissing. carries derived_id, lowercase hex SHA-256 over its unit_ref, derivation_key and cue_seq, and a sidecar over such a table defaults its id_column to derived_id.
  Why

  because a sidecar re-joins on one column, and a producer column outside the reserved underscore namespace needs no store change

Scenarios

• run.emit.output-modalityrun.emit.output-modalityA text passage extracted from a video parent carries _modality text, and an advertised image passage carries image, each under modality.: WHEN a video parent yields a transcript passage and an advertised picture, THEN their output modalities are text and image.
• run.emit.local-content-keyrun.emit.local-content-keyA local media unit's derivation-key includes a SHA-256 digest of canonical file bytes, so changed bytes under an unchanged path reselect the unit.: WHEN a local file changes bytes under the same path, THEN the next tick selects its unit under another key.

parse-cues

The caption parser reads SubRip and WebVTT through one grammar and groups cues into bounded passages.

• backward-cuetestIf a block starts before the last accepted block, the caption parser raises DeriveCueOutOfOrder and drops the block.
  Why

  P4
• passage-charstestA passage stays open while it holds fewer than 600 chars.
• passage-spantestA passage stays open while it spans less than 60 s.
• passage-bytestestOne passage’s text holds at most 8 KiB.
• passages-per-documenttestOne document yields at most 2000 rows of passages.
• grammartestSubRip and WebVTT read through one grammar: a block’s timing line, then its text lines joined by a space.

test-engine

The test-engine command builds one derive engine outside a pipeline and runs that engine against one file.

• unsupported-drivertestA fetch engine named to the verb raises DeriveTestEngineUnsupported.
  Why

  because the verb drives a transcriber over media, and a link engine takes no media

Shapes

Three engine bindings at the top level of contextful.toml:

[derive.local-asr]
driver           = "exec"
timeout_secs     = 1800
max_output_bytes = 8388608
zone             = "local:device"          # advisory; the adapter writes the row value

[[derive.local-asr.preprocess]]
when        = "engine_requires_pcm16_wav"
command     = ["ffmpeg", "-i", "{input}", "-ar", "16000", "-ac", "1", "{output}"]
output_path = "{output_stem}.wav"

[derive.local-asr.engine]
command       = ["speech-cli", "--model", "base", "--srt", "{input}"]
output_format = "srt"
output_path   = "{output_stem}.srt"

[derive.local-asr.env]
SPEECH_MODEL = "base"

[derive.cloud-asr]
driver        = "exec"
endpoint_host = "api.speech.example"       # steps reach it through CONTEXTFUL_EGRESS_URL

[derive.cloud-asr.attach]
Authorization = "Bearer ${secret://speech-vendor}"

[derive.cloud-asr.engine]
command       = ["speech-upload", "--server", "{input}"]
output_format = "srt"

[derive.card-reader]
driver               = "fetch"
allow_hosts          = ["example.com", "*.example.com"]
allow_image_hosts    = ["*.cdn.example.net"]
request_timeout_secs = 20