Docs / Guides

contract topology · 01-topology.md · guide · card

System topology

The problem topology solves

An operator chooses a Contextful binary and a place to run it. Contextful runs connectors, stores their output as Parquet files and serves queries under access controls. The engine carries no business rules; applications supply domain meaning (domain-neutraltopology.bound-application.domain-neutralThe engine processes declared data, schemas, policies, time bounds and capabilities. Consumer configuration, connectors, components or explicitly selected evaluation profiles supply domain meanings, business rules and sector-specific metrics.).

Three components, three shared contracts

One Cargo workspace builds execution, storage and query code (workspacetopology.compose.workspaceOne Cargo workspace compiles execution, storage and query code. These components share one binary, package scope and release.). The components share exactly three contracts: the connector interface, the part and manifest layout, and the capability-token format (three-crossingstopology.compose.three-crossingsExecution, storage and query code share exactly three contracts: the connector interface world, columnar-part-plus-manifest layout and capability-token format. No general internal API connects these components.).

Because every binary shares the layout, a query-only binary reads what a full binary wrote (run-path-droppabletopology.compose.run-path-droppableA build without the execution engine, scheduler or component host reads data written by a full-profile build. Both builds use the same part-and-manifest layout and connector contract.). A function returning stored rows requires an access-control admission value, so a call omitting it fails to compile (mediationtopology.compose.mediationEach function returning or releasing stored rows requires an access-control admission value (row-token). A call omitting that value fails to compile. Access controls have no disable option.).

Build profiles

Each build profile is a Cargo feature bundle chosen at build time (profiletopology.package.profileThree profiles compile from the workspace, each a Cargo feature bundle selected at build time: contextful-edge, contextful-full, contextful-control. Each links the dependencies its role names and nothing else.). contextful-full runs connectors, schedules work and serves queries (full-profiletopology.package.full-profilecontextful-full includes the execution engine, in-process scheduler, component host, SQL queries, transforms, full-text and vector sidecars, and tool server.). contextful-edge syncs stored files and serves read-only SQL (edge-profiletopology.package.edge-profilecontextful-edge syncs parts and manifests from a bucket and serves read-only SQL queries. This profile links no scheduler, execution engine, script runtime or component host.).

contextful-control edits team configuration and applies canonical TOML (control-profiletopology.package.control-profilecontextful-control provides self-hosted team configuration and operator identity. Apply produces canonical TOML. This profile alone links the CRDT library.). No run-time detection widens a running binary into another profile (fixed-at-buildtopology.package.fixed-at-buildNo run-time detection widens a running binary into another profile's feature set. Build, release and automation tooling links into no profile.). A subcommand needing an unlinked capability fails, naming the subcommand and profile (capability-absenttopology.package.capability-absentA subcommand requiring an unlinked capability raises ProfileCapabilityAbsent, naming the subcommand and profile. The command performs no action. A subcommand absent from every profile is a usage error.).

Coordinating writers

Coordination rests on one property: a linearizable conditional write (primitivetopology.coordinate.primitiveCoordination rests on one property: a linearizable conditional write. No component depends on a stronger one, and the tree ships no consensus implementation or external coordination service.). Each new lease holder gets a higher fence, so a stale writer’s fence sits below the holder’s (fence-advancestopology.coordinate.fence-advancesEach acquisition of a lease row increments its fence, and release keeps the fence, so the fence never repeats for a key.). A single node uses a local catalog file, a cluster uses Postgres, and Cloudflare uses D1 (backendstopology.coordinate.backendsSingle-node self-hosting uses a local catalog file. Clusters use Postgres through pg-catalog. Cloudflare execution uses D1 through d1-catalog.).

One deployment, end to end

A Linux writer syncs corpus files to R2. A Cloudflare Linux Container reads a synchronized copy behind the gateway, the Worker that receives each caller’s request and forwards it with the caller’s credential. The deployment maps show every provider’s shapes.

One reconciler at a time holds the deployment’s cadence lease (cadence-lease-ttltopology.coordinate.cadence-lease-ttlThe catalog grants a cadence lease for 90 s.) and renews it while the applied document schedules work (cadence-lease-renewaltopology.coordinate.cadence-lease-renewalThe reconciler renews a cadence lease every 30 s while its applied document schedules a dispatchable unit.). A run child invokes the connector, and the writer advances its cursor by one conditional update (cursor-castopology.coordinate.cursor-casA cursor compare-and-swap is one conditional update predicated on the stored version, read back by its affected-row count.).

Bucket sync copies the corpus to R2, and the edge binary pulls it and serves queries. The gateway forwards a caller’s request to the query container, which reaches readiness before it finishes loading stored data (container-readinesstopology.publish-hostname.container-readinessA retrieval container reaches readiness within 8 s of a cold start, before its snapshot-set hydration completes.).

The deployment probe checks each hostname against its declared access policy. A public response on a private hostname fails the probe (posture-mismatchtopology.publish-hostname.posture-mismatchAn unreachable hostname or response outside its declared access policy raises HostnamePostureMismatch, naming the hostname, policy and observed response. contextful-ci deploy probe exits nonzero.). A parity check compares each target’s parts with the reference deployment’s parts (paritytopology.deploy.parityOne declaration produces byte-identical table parts on every control-plane target; contextful-ci deploy parity compares each target's object-store parts with the reference target's.).

Deploying on AWS

On AWS every process runs the contextful-full binary as a Fargate task, a container AWS runs with no server to manage. S3 holds the corpus and RDS PostgreSQL holds the catalog and execution state, so a task’s disk is a rebuildable cache (container-profilerun.journal.container-profileCONTEXTFUL_STORAGE_PROFILE=postgres selects postgres-stores with shared owners, cadence and remote-state; corpus objects use a remote S3-compatible bucket with pull-before-run and push-after-run enabled.).

Two kinds of task share that state. A scheduled task runs due pipelines and exits. A long-lived contextful serve task answers reads behind a load balancer, and its health check passes once stored data has loaded (health-hydratedread.register.health-hydratedGET /health?ready=hydrated answers as past-ceiling until snapshot-set hydration completes, then as health.). Tasks reach S3 through their IAM role rather than stored keys (role-credentialsstore.endpoint.role-credentials[sync] credentials = "role" on an s3:// or https:// endpoint signs with workload-role credentials in place of both keys, from the ECS container credential endpoint, then STS web identity, then IMDSv2.).

The deployment maps name the service filling each role, and the Fargate runbook provisions them step by step.