Contextfulv0.23.0

Docs / Specification

contract connector · 40-connector.md · 41-secrets.md · 42-drive-extensions.md · guide · card

Connectors and the built-in sources

A connector brings external data into a run. Connector implementers use this contract for component interfaces, host permissions, vendor quotas, model calls, artifact loading and compiled-in sources. spec/41-secrets.md specifies outbound credentials.

Component and compiled-in connectors use the host’s HTTP client for vendor and model requests:

Flowchart diagram Flowchart, top to bottom. 9 nodes, 10 edges. connector package → component guest [pinned install] component sandbox: component guest → mediated HTTP client [outgoing HTTP, logging, wall clock]; → runner [batches + position] engine process: compiled-in source → mediated HTTP client [outgoing requests]; → runner [batches + position] engine process: mediated HTTP client → limiter [acquire, report]; → vendor API [allowlist, attach credentials]; → model endpoint [infer on marked data] run contract: runner → land path [committed batches] run contract: land path limiter → mediated HTTP client [permits] topology contract: model endpoint componentsandbox component guest engine process outgoing requests compiled-insource mediated HTTPclient run contract committedbatches runner land path topology contract model endpoint pinned install outgoing HTTP,logging, wallclock acquire, report permits allowlist, attachcredentials infer on markeddata batches +position batches +position connectorpackage limiter vendor API

export

Connector interfaces: shared types, source and destination worlds, optional interfaces and compiled-in sources.

  • authoring-pathtestEvery connector implements the same trait as either a sandboxed component or a native source compiled into the binary. Dispatch does not branch on these implementation forms.
    WhyA-connector#guest-world-targets
  • type-taxonomytestThe shared types interface carries a schema (name, fields, primary key) and a field (name, data type, nullability) over boolean, int32, int64, float64, string, bytes, timestamp-millis and json.
  • batch-encodingtestA batch crosses the boundary as Arrow IPC bytes, and a position as opaque bytes beside a declared cursor kind.
  • arrow-typestestThe host reads Arrow Binary or FixedSizeBinary columns as bytes of that width and FixedSizeList of Float32 or Float16 as vectors. Each lands in its binary-and-vector type without text conversion.
  • read-handletestA source exports its cursor kind, discovery call and open call taking a table and optional position. The handle’s next yields one batch or exhaustion; position reports the current position.
  • optional-worldtestOperator configuration and failure attribution each use an additional world interface. The host probes instances by interface name. A base-world guest instantiates unchanged; neither interface changes the world version.
  • partition-attributiontesttestA guest exporting failure attribution answers with the byte-exact values of the outermost partition column a failed read belongs to. The host reads it on failure and success alike; an empty list keeps the unattributed refusal.
    Whybecause a read holding one partition back and landing the rest returns success
  • attribution-budgettestFailure attribution carries at most 512 entries per session, each value at most 512 B.
  • shared-failuretestA read failing under auth-expired, or during which the host failed to hydrate or lease a credential, fails unattributed; the host discards the guest’s failure attribution for that session.
    Whybecause every attach binding is session-wide, so a credential failure belongs to every partition the read could reach

A connector failure crosses as a variant of failure-taxonomy.

Scenarios

import

Host imports and the configuration table the host forwards to a guest.

  • empty-contexttestA guest standard library pulling broader interfaces links against an empty context: no preopens, no environment, no arguments.
  • forwarded-configtestThe pipeline’s guest configuration table is the one part of the source configuration crossing inward, as one JSON object delivered once per session ahead of discovery.
    WhyA-surface#configuration-reaches-engine
  • config-shapetestAhead of any I/O, a guest configuration value that is not a table, a serialization over 64 KiB, or a credential or environment reference inside it raises ConnectorConfigRejected.
    WhyA-connector#guest-world-targets
  • config-unclaimedtestWhere the artifact is local and probed, a guest table declared against a guest exporting no configuration interface raises ConnectorConfigUnclaimed.
    WhyP1
  • unusable-keytestA guest raises a permanent failure for an unsupported configuration key or value.
    WhyP1
  • config-hashingtestThe forwarded table folds into the connector’s content hash; with no table forwarded, the artifact digest is that hash verbatim.

The outer source configuration follows config-key.

declare-capability

Manifest-declared host access, the probe that checks a bound credential’s granted scopes, and the window a connector’s destination deduplicates a key for.

Flowchart diagram Flowchart, top to bottom. 9 nodes, 11 edges. connector manifest → allowlist shape valid? [declared capabilities] allowlist shape valid? → rejected manifest [no: ConnectorAllowlistRejected]; → only listed access? [yes] only listed access? → rejected manifest [no: ConnectorUndeclaredAccess]; → scope probe declared? [yes] scope probe declared? → connector session [no]; → identity endpoint [yes, probe] identity endpoint → scopes within expectation? [granted scopes] scopes within expectation? → refused session [no header: ConnectorScopeUnverified]; → refused session [exceeded: ConnectorScopeExceeded]; → connector session [yes] declared capabilities no:ConnectorAllowlistRejected yes no:ConnectorUndeclaredAccess yes no yes, probe granted scopes no header:ConnectorScopeUnverified exceeded:ConnectorScopeExceeded yes connector manifest allowlist shape valid? rejected manifest only listed access? scope probe declared? connector session identity endpoint scopes within expectation? refused session

unsettled: Which manifest keys declare environment names and a shared quota, and how does a component load bind each to the operator’s reference and [limiters.<quota>] table? owner: connector affects: connector.declare-capability

Scenarios

widen

Comparison of a candidate connector manifest with its admitted predecessor, and operator approval of additional host access.

unsettled: Which operator authorization action and durable approval store serve local reload and control-plane apply? owner: connector affects: connector.widen

Scenarios

meter

Vendor quota declarations, limiter bindings, pre-send checks, permit reservations, denials and usage reports.

  • limiter-declarationtestA connector declares a limiter capability naming the shared vendor quota, the traffic class of its requests, and the vendor quota-state response headers to forward.
  • forward-credentialtestA limiter declaration forwarding authorization, proxy-authorization, cookie or set-cookie raises ConnectorForwardRejected at load.
    Whybecause the report carries forwarded headers verbatim to a party outside the vendor credential’s scope
  • limiter-bindingtestThe operator binds that quota name to an endpoint, a bearer token held by reference, and a permit batch size.
  • quota-unboundtestA declared quota with no binding raises ConnectorQuotaUnbound at load.
    WhyA-connector#host-access-quota
  • binding-transporttestA limiter endpoint that is neither HTTPS nor loopback or carries a query, fragment or userinfo, and a limiter token that is not a secret:// reference, raise ConnectorLimiterBindingRejected at load.
    WhyA-connector#egress-passes-transport
  • limiter-addresstestA limiter endpoint may resolve to a private or loopback address; the engine’s check of vendor-host addresses does not apply to it.
    Whybecause the operator authors the limiter endpoint and no vendor answer supplies it
  • reservation-pointtesttestThe host reserves at the mediation point, the guest’s outgoing-HTTP import or the engine’s shared client. One permit covers one outbound request, and the limiter counts requests and prices none.
    Whybecause one call issues any number of requests for paging, retries and token refresh
  • permit-batchtestAn acquire may grant a batch of permits under a short TTL. Each request spends one, and the next report surrenders the unspent.
  • acquiretestAcquire is an authenticated POST of quota, class and permit count, answered granted with permits and a TTL, or denied with a retry-after. A bare 429 with Retry-After, and a zero-permit grant, read as denials.
  • acquire-calltestEach limiter call takes at most 5 s. A reservation does not sleep: denial returns RateLimited without waiting. The run’s retry schedule performs the specified wait.
    WhyA-run#meter-reservation-holds
  • unreadable-answertestAn unreadable limiter response raises ConnectorLimiterUnreadable and grants no permits.
    WhyA-connector#host-access-quota
  • denial-ceilingtestA limiter wait, whether a retry-after or a zero-permit grant’s TTL, reads as at most the ceiling of retry-after.
    Whybecause an unbounded wait named by the limiter parks the run past any retry its schedule allows
  • reporttestReport is an authenticated POST of quota, class, granted, spent, one entry per vendor response (status, retry-after, verbatim headers), an observation instant and the run id.
  • report-deliverytestThe run delivers reports at least once with bounded backoff and makes one delivery attempt before each refill. An undelivered report accompanies the next. The run audits a report that remains undelivered without failing anything.
  • unmetered-requesttestUnder a declared limiter, a vendor request without a granted reservation raises ConnectorUnmetered, including for an unreachable, unbound or failing limiter. These failures do not indicate expired vendor credentials.
    WhyA-connector#host-access-quota
  • pre-send-hooktestEach outbound hop passes one pre-send hook after the allowlist and before name resolution, carrying the hop’s intent: method, scrubbed URL, host, port, traffic class, run id and request body bytes.
    WhyA-connector#egress-passes-transport
  • hook-compositiontesttestThe limiter reservation is the pre-send hook’s innermost implementation, behind the allowlist and any operator hook, so a request either refuses makes no limiter call and spends no permit.
    WhyA-connector#egress-passes-transport
  • hook-refusaltestA hook refusing an intent raises ConnectorEgressRefused, naming the host and reason. The host resolves no name, and the retry schedule does not retry the request.
    WhyP6
  • hook-settletestThe hook receives each admitted hop’s outcome after the body read or the failure: the status or transport failure, the request body bytes sent, and the response body bytes the host read. A followed redirect is two intents.
    WhyA-connector#egress-passes-transport
  • held-back-requesttestA guest swallowing a held-back request still fails its call, and a fault inside the reservation machinery fails the request rather than passing it unmetered.
    WhyP2
  • synthesized-throttletestA denied reservation returns to the guest as a synthesized 429 carrying the retry-after. An unreachable limiter fails the request as a transport failure.
    Whybecause a connector mapping vendor throttles needs no second interface
Sequence diagram Sequence diagram. 4 participants, 11 messages. Participants: guest or compiled-in source (G), host mediation point (H), limiter (L), vendor (V). 1. guest or compiled-in source → host mediation point: outbound request 2. host mediation point → host mediation point: allowlist check, a refused request spends no permit 3. host mediation point → limiter: POST acquire · quota, class, permits alt granted: 4. limiter --> host mediation point: permits + TTL 5. host mediation point → vendor: request, one permit spent 6. vendor --> host mediation point: response + quota-state headers 7. host mediation point --> guest or compiled-in source: response else denied, bare 429 with Retry-After, or zero permits: 8. limiter --> host mediation point: retry-after 9. host mediation point --> guest or compiled-in source: synthesized 429 else limiter unreachable or unbound: 10. host mediation point --> guest or compiled-in source: ConnectorUnmetered 11. host mediation point → limiter: POST report: granted, spent, responses, run id engine limiter vendor altgranteddenied, bare 429with Retry-After,or zero permitslimiterunreachable orunbound guest orcompiled-insourceguest orcompiled-insource host mediationpointhost mediationpoint limiterlimiter vendorvendor outbound request allowlist check, arefused requestspends no permit POST acquire ·quota, class,permits permits + TTL request, onepermit spent response +quota-stateheaders response retry-after synthesized 429 ConnectorUnmetered POST report:granted, spent,responses, run id

unsettled: Where does an operator declare the pre-send hook a command-line run composes in front of a compiled-in source’s reservation? owner: connector affects: connector.meter

infer

Model requests: endpoint selection, data boundary markers and output provenance labels.

unsettled: Does a call to an endpoint naming an alias rather than a pinned snapshot refuse under an availability horizon? owner: connector affects: connector.infer

Scenarios

package

Distribution form, digest pinning, per-connector resource bounds and world versioning.

build

The connector pin verb builds a guest on one fixed platform from its manifest’s one build input.

components

A component source loads under its manifest’s grant, runs on the native or interpreted target, and holds to fixed resource bounds.

A connector striking a declared bound follows bound-hit-is-transient.

Flowchart diagram Flowchart, top to bottom. 9 nodes, 12 edges. connector name → on the built-in list? [lookup] on the built-in list? → compiled-in source [yes]; → distribution form? [no] distribution form? → refused install [http: ConnectorInsecureArtifact]; → 64-hex pin? [https or oci]; → pin switch set? [local path] 64-hex pin? → refused install [no: ConnectorRemoteUnpinned]; → re-hashed bytes match? [yes] pin switch set? → refused install [yes, no digest: ConnectorLocalUnpinned]; → re-hashed bytes match? [no] re-hashed bytes match? → refused install [no: ConnectorDigestMismatch]; → component instance [yes, instantiate] lookup yes no http:ConnectorInsecureArtifact https or oci local path no:ConnectorRemoteUnpinned yes yes, no digest:ConnectorLocalUnpinned no no:ConnectorDigestMismatch yes, instantiate connector name on the built-in list? compiled-in source distribution form? refused install 64-hex pin? pin switch set? re-hashed bytes match? component instance

Scenarios

source

The declared behavior of each source compiled into the engine.

http

The generic HTTP source binds credentials, walks its pages under one pagination shape, and expands rows by follow-up requests.

formats

A body’s declared format selects its decoder: JSON, delimited text, a feed, an office container or a workbook.

objects

The s3 source and directory walks address, decompress and dead-letter each object they read.

documents

The file source and the document decoder land one row per page or heading, keyed by slug and ordinal.

  • document-graintesttestA document source lands one row per page, or one row per heading past a length threshold, each identified by a slug plus an ordinal, chunked or not. With a base URL, each row links to its own page anchor.
  • document-unreadabletestAn encrypted document or a document with no extractable text on any page counts as unreadable input under unreadable-input. The source stores no empty pages for these documents.
    WhyP2
  • frontmatter-shapetestA nested map, a block scalar, a key carrying the reserved producer prefix, or, compared without case, a repeated key or a key other than title naming a column the source lands, in a note’s frontmatter raises ConnectorFrontmatterRejected.
    WhyA-connector#untrusted-input-decodes
  • conversion-requiredtestA compound-binary office container, detected by an office extension or by magic bytes under an extension the source reads, raises ConnectorConversionRequired naming the conversion command.
    WhyA-connector#untrusted-input-decodes
  • file-sourcetestThe file source resolves root against the project directory and walks into one documents table: .md and .markdown notes, .txt plain text, .pdf pages. The source declines other extensions without opening them.
  • file-globstestThe source reads a root-relative path if no exclude glob matches and either an include glob matches or none is declared. * and ? match within one segment; a ** segment matches any number.
  • file-no-followtestThe file walk follows no symbolic link, to a file or a directory.
    Whybecause a link is the one path out of the root that a sorted walk of names cannot see
  • document-slugtestA document’s slug is its root-relative path without extension, lowercased, each run of characters other than ASCII letters and digits in a segment folded to -. Two files folding to one slug fail the read naming both.
    Whybecause a slug is the identity a link and a citation carry, and two files under one identity overwrite each other’s rows
  • heading-thresholdtestA note whose body past its frontmatter exceeds 8192 B lands one row per heading section, the text before its first heading as its own row; a shorter note and a text file land one row.
  • document-columnstestA document row carries row_id, slug, ordinal, root-relative path, kind, title, heading, text, url, the file’s SHA-256 and a removed flag. The title is the frontmatter title, else the first level-one heading, else the file stem.
  • frontmatter-columnstestEach other frontmatter key lands lowercased as a string column on every row of its note: a scalar as its text, a list as a JSON array.
  • document-urltestWith base_url, a row’s url is the base joined with its slug, then #page=<n> on a PDF page or the heading’s anchor on a heading section; without base_url the url is null.
  • file-positiontestThe file position maps each landed path to its digest and row count. A matching digest lands nothing, a changed file re-lands whole and tombstones ordinals past its new count, and a vanished file tombstones every row.
  • file-unreadabletestAn unreadable file lands no data rows and one dead-letter row. The source retains earlier rows and continues the read.
    WhyA-connector#unreadable-objects-files
  • file-renametestA file renamed onto its old slug re-lands under its new path, and the old path tombstones only ordinals past the renamed file’s row count.
    Whybecause a tombstone on an identity the read lands live removes the renamed document
  • file-captestThe source declines a file over 64 MiB without reading past the leading bytes identifying its format.
    Whybecause a row-per-page document past that size is an export, not a note, and one file must not exhaust the read’s memory
  • file-pdf-absenttestA PDF the walk reaches in a build without the PDF decoder fails the read as a configuration fault naming the feature to rebuild with.

reports

Async vendor reports land through the report source, one watermark per partition.

  • report-id-committesttestAn async-report source commits the vendor run id when creation returns. Exhausting the polling budget ends the read successfully. The source recreates an expired vendor run for the same window.
    WhyP4
  • page-draintesttestReport paging drains a fixed number of pages per read, one batch per page, each carrying the position after it.
  • page-cursor-missingtestA response announcing another page while naming its cursor in neither the cursor field nor the next link raises ConnectorPageCursorMissing.
    WhyP4
  • report-watermarktesttestThe source holds report watermarks per partition: account, grain, breakdown and any split dimension. One position contains the versioned watermark map.
  • partition-isolationtesttesttestA retryable failure on one partition holds it at its committed position and keeps the pages other partitions drained.
    WhyP4
  • metric-graintestA metric read at a grain where the vendor estimates rather than computes it raises ConnectorMetricGrainRejected.
    WhyP4
  • restated-windowtestA caught-up partition re-opens its lookback window every run, the pull window is at least that wide, and each row carries a deterministic key over partition, entity, date and breakdown.
  • report-event-timetestA report row’s event time is an RFC 3339 instant at midnight UTC of the window-start day, null where the vendor gave no usable date.
  • ordering-columntestOrdering a report stream by the engine-injected ingest stamp raises ConnectorOrderingColumnRejected.
    Whybecause a late seed then outranks a live restatement
  • field-drifttestReport rows enter the batch as untyped JSON. The source recognizes throttling from a throttle status or an error body containing a throttle code.
  • report-authtestAn authentication failure on a report read raises ConnectorAuthTerminal, terminal against the retry schedule.
    WhyP6
  • report-declarationtesttestA report source declares its vendor as VendorDeclaration beside the keys of ReportConfig and lands one table; a missing, unknown or malformed key fails validation naming it.
    Whybecause a request shape found malformed at its first fire fails a scheduled run hours after the edit that broke it
  • report-templatestesttest{account}, {grain}, {breakdown}, {split.<key>}, {start}, {end} and {metrics}, comma-joined, fill every template, and {id} the poll and result URLs, percent-encoded in a URL; another placeholder, or one in a scheme or host, fails validation.
    Whybecause a partition value placed in a host chooses where a bound credential goes
  • report-origintestEvery report request passes the mediated client with the create URL’s host as its one allowlisted host; a poll or result URL on another origin fails validation, and a vendor-named rows URL off it refuses under weakened-hop.
    Whybecause a credential attaches to one exact host, and a URL the vendor names must not carry it elsewhere
  • report-polltestA poll reads the status at state.pointer: a done value readies the run, an expired value expires it, and any other value, or none, leaves it pending. Two polls of one run wait at least interval apart.
  • report-failedtestA poll reading a failed value raises ConnectorReportFailed naming the run id, and the partition’s committed position drops that run, so the next fire creates a new one.
    Whybecause a vendor never rebuilds a failed report, so polling its id again fails every later fire
  • report-resulttesttestA ready run’s pages come from result.url, or from the URL its done poll names at result.pointer; each page after the first carries its cursor as the cursor_param query parameter.

images, search and receive

Images, web search results and pushed receive batches each land through their own source.

drive

The drive source walks selected Drive folders and lands one row per file and one per PDF page.

unsettled: Does a job whose after names a receive pipeline fire after each delivered batch commits, or does validation refuse it? owner: connector affects: connector.source

Scenarios

unsettled: Does a next-URL walk resume mid-walk through a position holding the URL with every credential-bearing query parameter removed? owner: connector affects: connector.source

unsettled: Does an incremental HTTP pull carry its page token beside the watermark, so a watermarked walk resumes mid-walk rather than from the watermark? owner: connector affects: connector.source

unsettled: Does a conditional read also carry a watermark in one position, so a feed both skips an unchanged document and filters entries by published_at? owner: connector affects: connector.source

unsettled: Does a feed entry with an unreadable date land with a null published_at rather than refusing the whole document? owner: connector affects: connector.source

unsettled: Does the default build link the PDF decoder, so a document folder holding a PDF lands without a rebuild? owner: connector affects: connector.source

unsettled: Does an object read without skip_unchanged hold its dead letters at their SHA-256, so an unchanged unreadable object stops landing a dead letter every fire? owner: connector affects: connector.source

Shapes

A drive source, built with the drive feature, capturing metadata of two folders in one Shared Drive:

[[pipeline]]
id = "team-metadata"
tables = [{ name = "files", primary_key = ["file_id"] }]

[pipeline.source]
name = "drive"

[pipeline.source.config]
drive_id = "0AExampleDrive"
folder_ids = ["1FirstRoot", "2SecondRoot"]
mode = "metadata-only"

[pipeline.source.config.oauth]
refresh_token = "${secret://drive-refresh}"
client_id = "${secret://drive-client-id}"
client_secret = "${secret://drive-client-secret}"

A document folder, its PDF pages decoding in a build with the pdf feature:

[[pipeline]]
id = "handbook"
tables = [{ name = "documents", primary_key = ["slug", "ordinal"] }]

[pipeline.source]
name = "file"

[pipeline.source.config]
root     = "handbook"                       # resolved against the project directory
include  = ["**/*.md", "policies/**"]
exclude  = ["drafts/**"]
base_url = "https://handbook.example.org"

An async-report source over a declared HTTP vendor, one table holding every partition’s rows:

[[pipeline]]
id = "ads"
tables = ["insights"]

[pipeline.source]
name = "report"

[pipeline.source.config]
partitions    = [{ account = "a1", grain = "day", breakdown = "age" }]
metrics       = ["spend", "clicks"]
entity        = "ad_id"
date          = "date_start"
lookback_days = 3
headers       = { Authorization = "Bearer ${secret://vendor-token}" }

[pipeline.source.config.create]
url  = "https://api.vendor.example/v1/{account}/reports"
body = { since = "{start}", until = "{end}", fields = "{metrics}" }
id   = "/report/id"

[pipeline.source.config.poll]
url      = "https://api.vendor.example/v1/reports/{id}"
interval = "30s"

[pipeline.source.config.state]
pointer = "/status"
done    = "done"
failed  = ["failed", "error"]

[pipeline.source.config.result]
url = "https://api.vendor.example/v1/reports/{id}/rows"

A connector manifest in the adopted manifest posture:

name    = "vendor-metrics"
version = "1.4.0"
world   = "source-connector@1.2.0"

wasm             = "vendor-metrics.wasm"
wasm_sha256      = "9f2c1b7ad0e4…"         # 64 hex, written by `connector pin`
require_wasm_pin = true

[build]
source = "guest"                          # the guest crate and its lockfile, beside this file

[capabilities]
allow_hosts = ["api.vendor.example"]      # one exact host: the pipeline's attach binds a credential
clock       = true

[capabilities.scope_probe]
endpoint      = "https://api.vendor.example/v1/identity"
scopes_header = "X-Granted-Scopes"
expect        = ["reports.read", "accounts.read"]

One outbound request, end to end:

Flowchart diagram Flowchart, top to bottom. 10 nodes, 14 edges. component sandbox: guest source → host mediation point [outgoing-http] engine process: compiled-in source → host mediation point [outgoing request] engine process: host mediation point → host on allowlist? [request] engine process: host on allowlist? → blocked destination [no: SecretUnpermittedRequest]; → pre-send hook admits? [yes, intent] engine process: pre-send hook admits? → blocked destination [refused: ConnectorEgressRefused]; → throttled request [denied: synthesized 429]; → throttled request [unreachable: ConnectorUnmetered]; → address public? [admitted, resolve] engine process: address public? → blocked destination [no: ConnectorPrivateAddress]; → TLS or loopback? [yes, headers attached] engine process: TLS or loopback? → blocked destination [no: SecretCleartextEndpoint]; → vendor [yes, vetted address] vendor → blocked destination [off origin: SecretRedirectOffOrigin] component sandbox guest source engine process outgoing request request yes, intent admitted, resolve yes, headers attached compiled-in source host mediation point host on allowlist? pre-send hook admits? address public? TLS or loopback? outgoing-http no:SecretUnpermittedRequest refused:ConnectorEgressRefused denied: synthesized 429 unreachable:ConnectorUnmetered no: ConnectorPrivateAddress no: SecretCleartextEndpoint yes, vetted address off origin:SecretRedirectOffOrigin vendor blocked destination throttled request