Docs / Deployment

AWS Fargate reference deployment

This OpenTofu configuration provisions the scheduled-fargate shape of spec/targets/aws.toml: every process runs the native contextful-full executable in a Fargate task. EventBridge Scheduler starts contextful pipeline serve --cycle on cycle_schedule; the cycle takes the cadence lease, runs every due unit and exits. An ECS service keeps serve_count tasks of contextful serve --http 0.0.0.0:<serve_port> behind an Application Load Balancer. RDS Postgres holds the catalog and the engine journal under CONTEXTFUL_STORAGE_PROFILE=postgres; S3 holds corpus objects. shape = "fargate-daemon" replaces the schedule with one always-on pipeline serve task.

Every identifier in this directory is a placeholder or a variable. terraform.tfvars.example lists the values a deployment supplies; the real terraform.tfvars, the state and .terraform/ stay untracked.

Resources

An internet-facing load balancer without certificate_arn fails the plan: bearer credentials never cross the internet in cleartext.

Roles

Both task roles trust ecs-tasks.amazonaws.com for this account’s ECS resources alone; the scheduler role trusts scheduler.amazonaws.com for this account alone. The sync adapter issues GET, PUT, DELETE and list requests, and no multipart upload, so the corpus policy grants nothing further.

Project image

The contextful-full image has no shell, and a Fargate task mounts no project directory, so the project travels in an image built on it. project.Dockerfile adds three files:

docker build --platform linux/amd64 -t contextful-full .          # repository root, see Dockerfile
tofu output -raw store_config > <project-dir>/config.toml
docker build --platform linux/amd64 -f apps/aws-fargate/project.Dockerfile \
  --build-arg BASE=contextful-full --build-arg PROJECT=<project> \
  -t <ecr-repository-url>:<tag> <project-dir>
docker push <ecr-repository-url>:<tag>

image_tag selects the pushed tag; tags are immutable, so each release pushes a new one and a tofu apply with the new image_tag registers new task definition revisions.

The rendered store configuration carries no [node] block. Each task starts on a fresh disk and resolves its own node id under store.lay-out.node-id-projectstore.lay-out.node-id-projectA project's default node id is node-<8 hex> of SHA-256 over the host id and absolute store root, so two checkouts on one host write as distinct nodes., so one task’s pull never skips run parts another task wrote under a shared id.

[sync]
endpoint        = "s3://<region>"
bucket          = "<corpus-bucket>"
prefix_from     = "env:CONTEXTFUL_SYNC_PREFIX"
coordination    = "cas"
pull_before_run = true
push_after_run  = true
credentials     = "role"

pull_before_run and push_after_run are mandatory under CONTEXTFUL_STORAGE_PROFILE=postgres (run.journal.container-profilerun.journal.container-profileCONTEXTFUL_STORAGE_PROFILE=postgres selects postgres-stores with shared owners, cadence and remote-state; corpus objects use a remote S3-compatible bucket with pull-before-run and push-after-run enabled.); a store configuration lacking either refuses with StorageCapabilityMissingrun.journal.storage-incompleteMissing profile bindings, corpus state, strict execution guards or a supported outbox raises StorageCapabilityMissing before execution admission; absent compiled adapters follow capability-absent..

Environment

serve takes --max-in-flight from serve_max_in_flight and --result-cache-bytes from serve_result_cache_bytes; project adds --project to both commands. A project whose applied control document schedules jobs that admit a credential needs CONTEXTFUL_TOKEN through secrets; without it, pipeline serve refuses with FireCredentialMissingsurface.dispatch.fire-credential-missingA serve dispatching to child processes, whose applied snapshot schedules a store-driven or synthesize job, holding neither --issuer-key nor CONTEXTFUL_TOKEN, raises FireCredentialMissing at start and fires nothing.. A source reaching a private VPC address lists that range in CONTEXTFUL_EGRESS_PRIVATE_RANGES through environment (connector.attach.private-range-admittedconnector.attach.private-range-admittedAn address inside a CIDR range listed in CONTEXTFUL_EGRESS_PRIVATE_RANGES passes private-address; the host name allowlist still applies.).

Every container runs under an init process, which reaps the child contextful processes a cycle dispatches. The pipeline container receives SIGTERM 120 s before SIGKILL; pipeline serve ends its children within 10 s (surface.dispatch.children-reapedsurface.dispatch.children-reapedA serve process starts each child run in its own process group; on SIGTERM, SIGINT, a --cycle exit or an unwind it signals each live group SIGTERM, SIGKILLs the remainder after 10 s, and exits once every child has.) and pushes for at most 15 s more (store.drain.deadlinestore.drain.deadlineOn a stop signal, pipeline serve with push_after_run ends its children, then pushes for its node's marks meeting before-read, for at most 15 s.). The serve container receives SIGTERM 30 s before SIGKILL (see Stopping).

Queue

Under queue_enabled, every task reaches the SQS queue as CONTEXTFUL_QUEUE_URL, signing with the task role (topology.compose.queue-sqstopology.compose.queue-sqsWith CONTEXTFUL_QUEUE_URL set, receive batches and object-arrival keys travel through that SQS standard queue, each request signed with Signature Version 4 under role-credentials; post-commit wake-ups stay in the local spool.). A POST /receive/:pipeline carrying Prefer: respond-async answers 202 once its batch is on the queue (run.land.receive-queuedrun.land.receive-queuedUnder queue-extension, a request asking for a queued answer receives 202 with its batch key after admission and queuing. 202 acknowledges queue custody, without claiming receive-ack's success.); a body past 1 MiB is written first to <corpus_prefix>/queue/spill/ in the corpus bucket, which the corpus policy already reaches (topology.compose.queue-sqs-captopology.compose.queue-sqs-capAn SQS message carries at most 1 MiB; a receive body past it spills under receive-queued-spill to the store's [sync] bucket beneath its prefix's queue/spill/.). Each pipeline serve --cycle drains the queue once after its units end: object-arrival keys wake the s3 reads covering them, and queued batches land and are deleted once their outcome records (surface.fire.cycle-drainsurface.fire.cycle-drainAfter its dispatched units end and before it releases the cadence lease, serve --cycle drains the queue's object-arrival keys and queued batches once, leaving post-commit wake-ups for contextful queue drain, and its answer counts what that drain settled.). A sender reads the outcome at GET /receive/:pipeline/:key. Post-commit wake-ups stay on each task’s local spool.

object_arrival_bucket names an existing bucket in this account and region; the notification resource replaces every other notification configured on it. The s3 source reading that bucket keeps its schedule, the backstop for an event S3 never delivers (run.land.object-arrival-wakerun.land.object-arrival-wakeAn object-arrival key only wakes its prefix read, which lists from its held position as a scheduled fire does; a duplicate or reordered key lands nothing twice, and the source's cadence still fires to catch a missed one.), and needs its own read access to the bucket. With kms_key_arn set, the key policy must let s3.amazonaws.com call kms:GenerateDataKey and kms:Decrypt, or S3 delivers no notification to the encrypted queue. Set queue_retention_seconds at or above the longest key_retention_secs of a receive source. A message on the dead-letter queue is one no consumer could read or settle; dead_letter_queue_url names it for inspection and redrive.

Catalog TLS

CONTEXTFUL_CATALOG_URL is postgres://contextful:<password>@<host>:5432/contextful?sslmode=require&sslrootcert=/etc/contextful/rds-global-bundle.pem. Remote PostgreSQL requires verified TLS with hostname checking (topology.coordinate.postgres-transporttopology.coordinate.postgres-transportBeyond loopback or Unix sockets, PostgreSQL requires TLS with WebPKI roots and hostname verification. Disabling TLS there raises DatabaseTransportUnprotected before DNS resolution. Preferred TLS mode permits no plaintext fallback there.). The RDS server certificate chains to the RDS CA named by db_ca_cert_identifier, which WebPKI omits; sslrootcert replaces the WebPKI roots for that connection with the bundle’s (topology.coordinate.postgres-ca-bundletopology.coordinate.postgres-ca-bundlesslrootcert=<path> in the connection string names a PEM bundle whose roots replace the WebPKI roots of postgres-transport for that connection; hostname verification still applies.). An unreadable or empty bundle refuses with DatabaseCaBundleUnreadabletopology.coordinate.postgres-ca-unreadableAn sslrootcert path that is unreadable or holds no certificate raises DatabaseCaBundleUnreadable, naming the path, before connecting. before connecting. rds_ca_bundle_path moves the bundle; the global-bundle.pem URL in project.Dockerfile serves every commercial region.

RDS Proxy also fronts the catalog. Its endpoint presents an ACM certificate chaining to Amazon Root CA 1, a WebPKI root. Set catalog_host to the proxy endpoint and rds_ca_bundle_path = "". The engine takes only transaction-scoped advisory locks, so proxy session pinning costs multiplexing and nothing else.

random_password generates the database password, and the catalog URL secret holds it; the OpenTofu state therefore holds both. Keep state in an encrypted backend or under OpenTofu state encryption.

Bucket credentials

sync_credentials = "role" renders credentials = "role" into the store configuration: requests sign with the task role’s credentials from the ECS container credential endpoint, refreshed 300 s before expiry (store.endpoint.role-credentialsstore.endpoint.role-credentials[sync] credentials = "role" on an s3:// or https:// endpoint signs with workload-role credentials in place of both keys, from the ECS container credential endpoint, then STS web identity, then IMDSv2.). v0.17.0 and earlier refuse the credentials key at parse.

sync_credentials = "keys" signs with a static key pair, for those releases. It creates the <name>/sync-keys secret, renders access_key_id = "env://CONTEXTFUL_SYNC_ACCESS_KEY_ID" and secret_access_key = "env://CONTEXTFUL_SYNC_SECRET_ACCESS_KEY", and the task definitions inject both from the secret’s JSON keys. Write the secret’s value as {"access_key_id": "...", "secret_access_key": "..."} for a principal holding the corpus_policy_arn policy alone. Injected keys do not rotate in a running task; a rotation takes effect at the next task start.

Secrets

secret://<name> references resolve through the Secrets Manager adapter as secret id <name>/secrets/<reference>, signed with the task role (connector.resolve.aws-secrets-managerconnector.resolve.aws-secrets-managerThe aws-secrets-manager backend answers secret://<name> with the AWSCURRENT value of secret id CONTEXTFUL_AWS_SECRET_PREFIX plus the name, signing under role-credentials; an absent secret is a miss.). For v0.17.0 and earlier, which lack that adapter, inject each credential through secrets as an environment variable and reference it by env://NAME; the execution role reads exactly the ARNs listed there, and the value does not rotate in a running task.

Health and warming

The target group checks GET /health?ready=hydrated every 15 s and expects 200. The probe admits no credential, answers 503 with Retry-After: 1 while the snapshot set hydrates, and once hydration completes answers 200 with the build identity and "hydration": "complete" (read.register.health-hydratedread.register.health-hydratedGET /health?ready=hydrated answers as past-ceiling until snapshot-set hydration completes, then as health.), so a rolling deployment routes no read to a warming task. Plain /health answers 200 from a cold start, before hydration completes (read.register.healthread.register.healthGET /health answers 200 with the build identity of build-identity, admitting no credential and reading no row., topology.publish-hostname.container-readinesstopology.publish-hostname.container-readinessA retrieval container reaches readiness within 8 s of a cold start, before its snapshot-set hydration completes.); until then every other route answers 503 with Retry-After: 1 (read.cache.warming-readread.cache.warming-readA request other than GET /health reaching a retrieval container whose snapshot-set hydration is incomplete answers as past-ceiling; /health names the hydration state.). serve_hydration_grace_seconds (default 300 s) plus three failed checks bounds how long a task may hydrate before ECS replaces it; size it to the store’s cold pull, or a large store replaces each task before it serves. serve pulls every bucket table before its first read (store.pull.before-runstore.pull.before-runWith [sync] pull_before_run = true, run start, pipeline run, mcp and serve pull the tables table-scope selects before their first read, a failed pull stopping the command.) and again every 30 s (store.pull.serve-refreshstore.pull.serve-refreshUnder before-run, serve pulls again every 30 s and maps the new generation for reads starting after that pull; a failed refresh keeps serving the held generation and logs the failure.). A deployment registers replacement tasks before it drains the old ones, and the circuit breaker rolls back a revision whose tasks fail to start.

Stopping

ECS stops a serve task in two steps. It first deregisters the task from the target group, which routes it no new request and holds its open connections for deregistration_delay (30 s). It then sends SIGTERM and, stopTimeout (30 s) later, SIGKILL. On SIGTERM, serve admits no new request, answers every connection it accepted, and exits 0 once none remains, at most 20 s after the signal (read.register.stop-drainread.register.stop-drainOn SIGTERM or SIGINT, contextful serve --http admits no new request, answers every connection it accepted before the signal, and exits 0 once none remains, at most 20 s after the signal.). Each new connection meanwhile, /health included, answers 503 with Retry-After: 1 (read.register.draining-healthread.register.draining-healthFrom the stop signal on, every new connection, GET /health included, answers as past-ceiling before its request is parsed.). A connection still unanswered at 20 s closes; serve names their count on stderr and exits non-zero (read.register.drain-unfinishedread.register.drain-unfinishedA connection still unanswered at the bound of stop-drain closes unanswered; the server names their count on stderr and exits non-zero.). Keep stopTimeout above 20 s, or SIGKILL cuts the drain short.

First deployment

1. Copy terraform.tfvars.example to terraform.tfvars and replace every placeholder.

2. tofu init, then tofu apply -target=aws_ecr_repository.project when the configuration creates the repository.

3. Build and push the project image as above, then tofu apply with its image_tag.

4. Bind the empty catalog to recovery_id once, with the pipeline task definition and a command override:

   aws ecs run-task --cluster "$(tofu output -raw cluster)" \
     --task-definition "$(tofu output -raw pipeline_task_definition)" --launch-type FARGATE \
     --network-configuration "awsvpcConfiguration={subnets=[<subnet-ids>],securityGroups=[<task-security-group>],assignPublicIp=DISABLED}" \
     --overrides '{"containerOverrides":[{"name":"pipeline","command":["storage","initialize"]}]}'

   task_network prints the subnet and security group ids. A later storage check override validates the catalog, the recovery identity and the corpus.

5. Run pipeline import, then pipeline apply, as further command overrides. In a project with [sync], each admits an admin capability and signs a control receipt (surface.apply.synced-attestationsurface.apply.synced-attestationIn a project with [sync], import and apply admit an admin capability, then sign a receipt over the project, version, predecessor receipt digest and snapshot digest through signing-port.), so the override task carries that credential through secrets. The next schedule tick dispatches the applied pipelines.

fargate-daemon

shape = "fargate-daemon" removes the schedule and the scheduler role and runs the pipeline task definition as a one-task service whose command is contextful pipeline serve: the in-process scheduler arms each unit under the cadence lease, for cadences below the one-minute floor of EventBridge Scheduler. A deployment stops the old task before starting its replacement, and the cadence lease fences any overlap. The serve service and every other resource stay identical. While it holds the cadence lease, the daemon drains the queue beside its ticks, starting each drain 5 s after the previous one ends, so queued batches land and object-arrival keys wake their reads between cadence fires (surface.fire.daemon-drainsurface.fire.daemon-drainWhile it holds the cadence lease, pipeline serve under the in-process trigger drains the queue as cycle-drain does, beside its ticks, starting each drain 5 s after the previous one ends.). A failed drain leaves its deliveries queued and the next pass retries them (surface.fire.daemon-drain-retrysurface.fire.daemon-drain-retryA failed daemon drain logs its error and leaves its deliveries queued; the daemon keeps serving and drains again on its next pass.).

Validation

cd apps/aws-fargate
tofu fmt -check -recursive
tofu init -backend=false
tofu validate

.terraform.lock.hcl pins the provider versions and their checksums for Linux, macOS and Windows.