Files
adr-sml/docs/09-cutover-runbook.md
T
Pouya LajevardiandClaude Opus 5 07a8ff6989
Build and deploy / build-and-deploy (push) Failing after 4s
fix: assert which aws binary runs, because this script rewrites the whole config
Round 2 returned an addendum after the previous commit, having re-checked its
own instrument. Two findings, one blocking on the next --apply.

This machine carries two AWS CLIs on PATH: /opt/homebrew/bin/aws 2.34.53 and
/usr/local/bin/aws 2.11.15 (April 2023). configure.mjs called bare `aws`, so
PATH decided. That matters because update-distribution is a full replace and
botocore parses a config against its own model, dropping members it does not
know — an old CLI reads a lossy config and writes the loss back, and --if-match
cannot catch it because the ETag is genuinely current.

Measured, which the review could not do: the older model is missing nine
members, and E1OK7G98KNKUTA carries two of them — GrpcConfig on the default
behaviour and on /api/*, both {Enabled: false}. So a round trip through the old
CLI would write back the same effective value and change nothing observable
today. That is precisely why it needed a guard rather than a look: nothing
reports it when that stops being true.

The script now resolves the binary, prints it and its version as the first line
of output, and exits 2 below a floor before any AWS call. Verified both ways.

Second finding: entry (av) claimed "no client-side validation of any kind"
stood between the 182-character Comment and the API, from reading one
validate.py without naming which CLI it came from — and the other install is
frozen, so it is unchecked rather than confirmed. That is CLAUDE.md's named
shape. Both records now name the instrument and version and lead with the claim
that needs no qualification: the Comment reached the API and came back
InvalidArgument, so nothing stopped it on the CLI that ran.

The second install's model independently confirms FunctionARN max 108, its
pattern, and the 128 on both Comment members.

Nothing was applied to the distribution and nothing was deployed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Md3GndFqWPzK78xAoebsg5
2026-09-04 14:32:11 -04:00

75 KiB
Raw Blame History

09 — Cutover runbook: the exact command sequence

Authority: AGENTS.md §3 D11 (build everything, one clean cutover) and §7 for every operational fact. docs/06-deployment.md owns the cutover checklist — what must be true. This file owns the commands — how to make it true. The checklist cites this file; this file does not restate the checklist.

Pouya runs everything here. Parts 17 need administrative credentials that no agent on this project holds, and Part 8 is a deploy that must not run as user/pouya (§10). Every command is followed by the command that verifies it and the output to expect. Where a verification can come back two ways, both are named and the sequence branches — a branch driven by a measurement rather than by a guess is the point.

Read docs/06's cutover checklist alongside this. The parts below discharge its Technical group. Its Content and compliance group — the claims-auditor pass and Pouya's own page-by-page read — is not commands and is not here.


Why the order is this order, and what is broken while it runs

The bucket currently holds the old single-file site: index.html (2,206,032 bytes) plus two logo PNGs [verified 2026-09-01 — list-objects-v2]. So the live site today is one page, and one page is all that works: measured on the live distribution the same day, / returns 200 while /about/ and /definitely-not-a-page/ both return 403 with an 111-byte application/xml body — S3's AccessDenied, served raw to the reader.

That is why the infrastructure goes first and the site goes last:

  1. Parts 14 (distribution) change nothing a visitor can see. The router rewrites / to /index.html, which is what the default root object already did, and the paths it newly handles do not exist on the old site either way.
  2. Parts 57 (intake) are inert until /contact/ exists.
  3. Part 8 is the cutover. The moment index.html syncs, the new site is live — and by then the router, the 404 mapping and /api/* are already in place, so there is no window in which 22 of 23 pages are broken.

Reversing this — site first, infrastructure second — would put every page except / behind a 403 for as long as the CloudFront deployment took.


Part 0 — Preflight

0.1 The substitution block

⚠️ THIS IS THE ONLY PLACE IN THIS FILE THAT NAMES A RESOURCE, AND THAT IS A DELIBERATE, BOUNDED EXCEPTION TO THE §7 RULE. AGENTS.md §7 is the single source of truth for operational facts and specs cite it rather than copying it — but a runbook that cannot be pasted into a terminal is not a runbook. So the values appear exactly once, and 0.2 is the mechanism that keeps the copy honest: it proves every one of them resolves to a real resource before anything is changed. A duplicated fact needs a mechanism, not a promise.

Confirm each against §7 before pasting. ./scripts/aws-discover.sh regenerates them from AWS if §7 itself needs re-verifying.

cd ~/Dev/Websites/adr-sml

export AWS_REGION=ca-central-1
export AWS_DEFAULT_REGION=ca-central-1
export ACCT=327082975128
export FN=adr-intake-handler
export TABLE=adr-intake-submissions
export API_ID=4tl0m5igkj
export API_DOMAIN=4tl0m5igkj.execute-api.ca-central-1.amazonaws.com
export DIST_ID=E1OK7G98KNKUTA
export BUCKET=adr-smlcompany-site
export SITE=https://adr.smlcompany.ca
export NOTIFY_TO=info@smlcompany.ca
export MAIL_FROM=intake@smlcompany.ca

NOTIFY_TO is where an inquiry lands and MAIL_FROM is the From on both messages. Both are choices rather than records: info@ is the address /contact/ publishes, so a reply threads where the inquirer expects; intake@ distinguishes form mail from correspondence. Both must be at smlcompany.ca, which is the verified sending identity — 0.2 checks that.

0.2 Prove every one of them exists

aws sts get-caller-identity
for v in AWS_REGION ACCT FN TABLE API_ID API_DOMAIN DIST_ID BUCKET SITE NOTIFY_TO MAIL_FROM; do
  eval "val=\$$v"
  [ -n "$val" ] || { echo "EMPTY: $v"; break; }
  printf '%-12s %s\n' "$v" "$val"
done
aws lambda get-function-configuration --function-name "$FN" --query 'FunctionName' --output text
aws dynamodb describe-table --table-name "$TABLE" --query 'Table.TableStatus' --output text
aws apigatewayv2 get-api --api-id "$API_ID" --query 'Name' --output text
aws cloudfront get-distribution --id "$DIST_ID" --query 'Distribution.Status' --output text
aws s3api head-bucket --bucket "$BUCKET" && echo "bucket ok"
aws sesv2 get-email-identity --email-identity "${MAIL_FROM#*@}" \
  --query 'VerifiedForSendingStatus' --output text

Expect: an identity with administrative rights; eleven non-empty values; adr-intake-handler; ACTIVE; adr-intake-api; Deployed; bucket ok; True.

A None, an empty value, or any non-zero exit stops the run here. That is the whole purpose of 0.2 — a stale identifier fails loudly at the top instead of halfway through Part 6.

0.3 State this run starts from

Recorded so a later reader can tell what this sequence changed from what it found. All [verified 2026-09-01], read-only:

Lambda nodejs24.x, arm64, handler index.handler, timeout 10 s, memory 128 MB, no environment variables, no DLQ, code 1,527 bytes (the hand-built inline function), last modified 2026-05-26
Lambda role adr-intake-lambda-roleAWSLambdaBasicExecutionRole plus an inline policy granting dynamodb:PutItem on the table and ses:SendEmail/ses:SendRawEmail. Sufficient for the new handler; no change needed
Lambda invoke permission one statement, apigateway.amazonaws.com, SourceArn scoped to …/4tl0m5igkj/*/*/submissions — the old route's path only
HTTP API one route, POST /submissions → integration 0ftgjgv (AWS_PROXY, payload format 2.0, the correct one for this handler). Stage $default, auto-deploy on, no throttling, no access log. CORS allows POST from the site origin
DynamoDB ACTIVE, on-demand, partition key submissionId (S), no sort key, 4 existing items, TTL ENABLED on attribute ttl, PITR ENABLED (35-day window), encryption at rest with the AWS-owned key (no CMK)
CloudFront one S3 origin with OAC E13GAFUL6UQP6R; default behaviour Managed-CachingOptimized + Managed-SecurityHeadersPolicy, methods HEAD/GET; no cache behaviours, no custom error responses, no function associations
S3 versioning Enabled; all four public-access blocks true; direct object GET on the S3 hostname returns 403, so OAC is in force
SES domain verified for sending, DKIM SUCCESS and signing enabled, no custom MAIL FROM
SNS ses-alerts email subscription to info@smlcompany.ca is CONFIRMED — it has a real subscription ARN, not PendingConfirmation. §7 recorded it as pending for six days after it stopped being pending; §12 R9 closes on this

Part 1 — Make S3 answer 404 instead of 403

Why this is first. Part 3 maps 404 to /404.html and deliberately does not map 403. With OAC and no s3:ListBucket, S3 answers a request for a missing key with 403 AccessDenied, so a 404-only mapping would never fire and every bad URL would keep returning raw XML. Granting the CloudFront service principal s3:ListBucket changes that answer to 404 NoSuchKey.

Mapping 403 as well would have been one command shorter and is the wrong trade: custom error responses are distribution-wide and cannot be scoped to one behaviour, so it would also convert a broken bucket policy or a broken OAC — a real outage on every URL at once — into a tidy "page not found", and it would turn the intake handler's Origin refusal into a 404 page. Keeping 403 meaning 403 is worth one statement.

aws s3api get-bucket-policy --bucket "$BUCKET" --query Policy --output text > /tmp/bucket-policy.before.json
python3 -m json.tool /tmp/bucket-policy.before.json

Expect: one statement, AllowCloudFrontServicePrincipal, s3:GetObject on arn:aws:s3:::$BUCKET/*, conditioned on the distribution ARN. Keep that file — Part 9 restores from it.

python3 - "$BUCKET" "$ACCT" "$DIST_ID" <<'PY' > /tmp/bucket-policy.after.json
import json, sys
bucket, acct, dist = sys.argv[1], sys.argv[2], sys.argv[3]
p = json.load(open('/tmp/bucket-policy.before.json'))
arn = f'arn:aws:cloudfront::{acct}:distribution/{dist}'
sid = 'AllowCloudFrontListForHonest404s'
p['Statement'] = [s for s in p['Statement'] if s.get('Sid') != sid]
p['Statement'].append({
    'Sid': sid,
    'Effect': 'Allow',
    'Principal': {'Service': 'cloudfront.amazonaws.com'},
    'Action': 's3:ListBucket',
    'Resource': f'arn:aws:s3:::{bucket}',
    'Condition': {'StringEquals': {'AWS:SourceArn': arn}},
})
json.dump(p, sys.stdout, indent=2)
PY
python3 -m json.tool /tmp/bucket-policy.after.json
aws s3api put-bucket-policy --bucket "$BUCKET" --policy "file:///tmp/bucket-policy.after.json"

Verify — three things, and the third is the one worth doing:

aws s3api get-bucket-policy --bucket "$BUCKET" --query Policy --output text | python3 -m json.tool
aws s3api get-public-access-block --bucket "$BUCKET" --query 'PublicAccessBlockConfiguration'
curl -s -o /dev/null -w 'direct S3 object: %{http_code}\n' \
  "https://${BUCKET}.s3.${AWS_REGION}.amazonaws.com/index.html"
aws cloudfront get-distribution-config --id "$DIST_ID" \
  --query 'DistributionConfig.DefaultCacheBehavior.{Cache:CachePolicyId,OriginRequest:OriginRequestPolicyId}'
aws cloudfront get-cache-policy --id 658327ea-f89d-4fab-a63d-7e88639e58f6 \
  --query 'CachePolicy.CachePolicyConfig.ParametersInCacheKeyAndForwardedToOrigin.QueryStringsConfig'

Expect: two statements; all four public-access blocks still true; direct S3 403; Cache = 658327ea-… (Managed-CachingOptimized) with OriginRequest: null; and QueryStringsConfig = {"QueryStringBehavior": "none"}".

⚠️ THOSE LAST TWO ARE THE CHECK, AND THE OBVIOUS ONE IS WORTHLESS. The tempting verification is curl "${SITE}/?list-type=2&max-keys=5" and confirming it returns HTML rather than an XML ListBucketResultit cannot return anything else, so it is evidence of nothing. The distribution has DefaultRootObject: index.html and, after Part 2, the router rewrites / to /index.html; the origin request is therefore a GetObject on a key, never a request for the bucket root, so list-type=2 could not be read as a list operation whether or not query strings reached S3. An operator running it sees HTML and ticks a control that never ran, which is the Q22 shape exactly. Found by adversarial-reviewer, 2026-09-01.

What actually makes the grant safe, and the two commands above assert both halves: no viewer path resolves to the bucket root, and the default behaviour forwards no query strings to the origin — with no origin request policy attached to override that. If either assertion fails, revert this part (Part 9.1) and map 403 instead.


Part 2 — The trailing-slash router function

Source and test are in the repo: infra/cloudfront/router.js and router.test.mjs. Read the header comment before publishing it — the reason it must not be associated with /api/* is in there, and it is the difference between a working form and a POST silently converted to a GET.

node infra/cloudfront/router.test.mjs

Expect: exit 0. The case count is deliberately not quoted here — it has already gone 15 → 22 → 30 in one day, and an Expect line that never matches teaches the operator to stop reading Expect lines. The script asserts its own count internally (case count != CASES.length), so exit 0 is the whole check.

aws cloudfront create-function \
  --name adr-sml-router \
  --function-config '{"Comment":"trailing-slash + index.html for the Astro directory build; docs/09","Runtime":"cloudfront-js-2.0"}' \
  --function-code "fileb://infra/cloudfront/router.js" \
  --query '{Name:FunctionSummary.FunctionMetadata.FunctionARN,Stage:FunctionSummary.FunctionMetadata.Stage,Status:FunctionSummary.Status}'

Expect: an ARN, Stage: DEVELOPMENT, Status: UNPUBLISHED.

Test it in the real runtime before publishing. The local test proves the branches; this proves the runtime accepts the code, which the local test cannot.

ETAG=$(aws cloudfront describe-function --name adr-sml-router --query ETag --output text)
for URI in / /about/ /about /robots.txt /_astro/x.css /api/intake \
           //evil.example.com/x '/\evil.example.com/x' //about/ //robots.txt; do
  printf '{"version":"1.0","request":{"method":"GET","uri":"%s","querystring":{},"headers":{},"cookies":{}}}' "$URI" \
    > /tmp/cf-event.json
  printf '%-16s ' "$URI"
  aws cloudfront test-function --name adr-sml-router --if-match "$ETAG" --stage DEVELOPMENT \
    --event-object fileb:///tmp/cf-event.json \
    --query 'TestResult.{Out:FunctionOutput,Err:FunctionErrorMessage}' --output text
done

fileb:// rather than a base64 argument: it hands the CLI raw bytes and sidesteps the question of whether the local base64 wraps its output, which would make the argument invalid in a way the error message would not explain. test-function does not change the function, so one ETAG read serves the whole loop.

Then the query-string path, which none of the URIs above exercises:

cat > /tmp/cf-event-qs.json <<'JSON'
{"version":"1.0","request":{"method":"GET","uri":"/fees","headers":{},"cookies":{},
 "querystring":{"utm":{"value":"a%20b"},"q":{"value":"x|y"},"bad":{"value":"z\"<>#&k=v"}}}}
JSON
aws cloudfront test-function --name adr-sml-router --if-match "$ETAG" --stage DEVELOPMENT \
  --event-object fileb:///tmp/cf-event-qs.json \
  --query 'TestResult.{Out:FunctionOutput,Err:FunctionErrorMessage}' --output text

Expect, in order for the ten URIs: uri /index.html; uri /about/index.html; a 301 to /about/; uri /robots.txt unchanged; uri /_astro/x.css unchanged; /api/intake a 301 to /api/intake/; and then four 301s that all stay on this origin with a single leading slash/evil.example.com/x/, /evil.example.com/x/, /about/, /robots.txt. No FunctionErrorMessage on any of the ten.

From the query-string call: a 301 to /fees/?utm=a%20b&q=x|y&bad=z&k=v. Three things are being checked there and each has been wrong once: %20 survives (the values arrive percent-encoded, so re-encoding would produce %2520); | survives (it is not in WHATWG's query percent-encode set, so browsers send it raw, and one revision of the function stripped it and silently corrupted campaign links); and ", <, > and # are gone (# is the one that changes the structure of the header — left in, &k=v lands in a fragment and the parameter is lost).

⚠️ THE FIRST VERSION OF THIS LOOP PASSED "querystring":{} ON EVERY CASE AND TRIED NEITHER REDIRECT PATH. So in the only environment that runs the real cloudfront-js-2.0 runtime — the environment this step exists to reach — none of the query-string handling and none of the normalisation was executed, while the prose called it "the authoritative check". Found by adversarial-reviewer round 2.

⚠️ That last row is the reason this function must not be associated with the /api/* behaviour, and it is in the list so you see it rather than read about it. A 301 turns a POST into a GET, so an intake submission routed through this function would arrive at the handler as a GET with no body — and the visitor would be told nothing was wrong. Part 3 associates the function with the default behaviour only; configure.mjs sets FunctionAssociations: { Quantity: 0 } on the /api/* behaviour explicitly rather than by omission.

aws cloudfront publish-function --name adr-sml-router --if-match "$ETAG"
export ROUTER_ARN=$(aws cloudfront describe-function --name adr-sml-router --stage LIVE \
  --query 'FunctionSummary.FunctionMetadata.FunctionARN' --output text)
echo "ROUTER_ARN=$ROUTER_ARN"

Expect: a non-empty ARN. It is empty if publish-function failed — read the status, not the absence of an error.


Part 3 — Apply the five distribution changes

One script, infra/cloudfront/configure.mjs, because the alternative is hand-editing a 300-line JSON document and posting it back with an IfMatch ETag. It is dry-run by default, idempotent, and it resolves both managed policy ids by name from the account rather than carrying them as literals.

node infra/cloudfront/configure.mjs --dist "$DIST_ID" --api-domain "$API_DOMAIN" \
  --function-arn "$ROUTER_ARN"

⚠️ THE BLOCK BELOW IS THE + CHANGE LINES AND THE TWO RESOLVED-POLICY LINES. IT IS NOT THE WHOLE OUTPUT, AND IT SAID "exactly" UNTIL 2026-09-04. Against the live distribution the dry run is 59 lines, and all 59 account for: 2 resolved-policy lines, 10 · lines, 3 + lines, a 40-line JSON dump of the *.pdf behaviour it would add, 2 blank lines, the N change(s) to distribution … header and the DRY RUN — nothing was sent. footer [measured 2026-09-04, exit 0, nothing written].

🛑 THE FIRST LINE OF OUTPUT NAMES THE aws BINARY AND ITS VERSION. READ IT. This script rewrites the whole distribution config, and botocore drops members its own model does not know — so an old CLI reads a lossy config and writes the loss back, on a distribution serving 23 pages and the intake form. --if-match cannot catch that: the ETag is genuinely current.

⚠️ THIS MACHINE HAS CARRIED TWO CLIs, 2.34.53 and 2.11.15 (April 2023), both on PATH [measured 2026-09-04]. The older model does not know GrpcConfig, and E1OK7G98KNKUTA carries it on two behaviours — both {Enabled: false}, so the round trip is lossless in effect today, and nothing would report it when that stops being true. It is also missing ConnectionMode, VpcOriginConfig, CacheTagConfig and five more. The script now refuses below a floor rather than leaving it to which:

resolved aws                                = /opt/homebrew/bin/aws (2.34.53)

If that line names /usr/local/bin/aws or a version below the floor, the run exits 2 before any AWS call. Run which -a aws and fix PATH — do not lower the floor to get through.

🛑 READ THE EXIT STATUS, AND IT HAS THREE VALUES. 0 — everything this script manages was applied or is already present. 2 — a usage error, before any AWS call. 3 — sections that could run did, and something was SKIPPED: read the ⚠ … SKIPPED, not changed block. Anything else is a throw. 3 exists because a skip used to exit 0, and this document uses exit 0 as its own success stamp throughout — so a partial run read as a complete one. A skip is never nothing: it means the *.pdf behaviour has no X-Robots-Tag, or /api/* is not forwarding CloudFront-Viewer-Address, and the skip line says which.

⚠️ THE SHAPE CHANGES WITH THE STATE, SO READ THE + LINES AND NOT THE TOTAL. Each item Parts 13 have already applied prints · when it is found and + when it is staged, so in the Part 0.3 state four lines cross from one column to the other and the totals move with them. The + lines are the check.

Expect — this is the dry run, and on a distribution in the state Part 0.3 records the change lines are:

resolved Managed-CachingDisabled            = 4135ea2d-6df8-44a3-9df3-4b5a84be39ad
resolved Managed-AllViewerExceptHostHeader  = b689b0a8-53d0-40ab-baf2-68738e2966ac

8 change(s) to distribution E1OK7G98KNKUTA (ETag …):
   + DefaultCacheBehavior.FunctionAssociations viewer-request -> arn:…:function/adr-sml-router
   + CustomErrorResponses += 404 -> /404.html with status 404
   + Origins += intake-api -> …execute-api… (https-only, TLSv1.2)
   + CacheBehaviors += /api/* -> intake-api, CachingDisabled, AllViewerExceptHostHeader, POST allowed
   + create response-headers policy adr-sml-pdf-noindex (SecurityHeadersConfig cloned from … + X-Robots-Tag: noindex)
   + CacheBehaviors += *.pdf -> <s3-origin>, default cache policy, adr-sml-pdf-noindex (policy id created in the same --apply pass)
   + create origin request policy adr-sml-api-viewer-address (whitelist: CloudFront-Viewer-Address, Content-Type, Origin, Referer, User-Agent; cookies all; query strings all)
   + /api/* OriginRequestPolicyId b689b0a8-… -> adr-sml-api-viewer-address

DRY RUN — nothing was sent. Re-run with --apply to write it.

⚠️ SECTION 4 CANNOT SHOW THE POLICY ID IN A DRY RUN, AND SAYS SO — IT IS STILL ONE --apply. The *.pdf behaviour has to carry the response-headers policy's id, and on a first run that policy does not exist yet, so the dry run prints the behaviour it would add with (policy id created in the same --apply pass) where the id goes. A single --apply creates the policy and adds the behaviour in one call — do not run it twice. The dry run reports both changes either way; one that listed only the policy would hide the half that touches a distribution serving 23 pages.

Fewer than eight changes means part of this is already done — read which lines are prefixed · — but READ THE WORDS, not the bullet: configure.mjs uses · for already present and for would CREATE / would SET / would ADD, so the prefix alone does not say whether a line is done or still pending. On the live distribution as at 2026-09-04, after the second --apply attempt, the dry run returns exactly three + lines — one for section 4 (the behaviour; its policy already exists, see the incident below) and the two for section 5[measured 2026-09-04, dry run against E1OK7G98KNKUTA, ETag E2EUQ1WTGCTBG2, exit 0, nothing written]. More than eight, or a different set, means the distribution is not in the state 0.3 recorded: stop and re-read it.

🛑 INCIDENT — TWO --apply ATTEMPTS FAILED ON 2026-09-04, FOR TWO DIFFERENT REASONS, AND THE SECOND ONE LEFT A POLICY BEHIND. The distribution is unchanged after both. Read both before the next attempt.

ATTEMPT 1 — nothing reached the distribution and nothing was created. Section 4's clone was sent to create-response-headers-policy verbatim and the AWS CLI rejected it client-side, before the call left the machine:

An error occurred (ParamValidation): Parameter validation failed:
Missing required parameter in ResponseHeadersPolicyConfig.SecurityHeadersConfig.ContentSecurityPolicy: "Override"
Missing required parameter in ResponseHeadersPolicyConfig.SecurityHeadersConfig.ContentSecurityPolicy: "ContentSecurityPolicy"

The cause, and it generalises past this script: a config AWS hands back is not necessarily a config AWS will accept. get-response-headers-policy on Managed-SecurityHeadersPolicy returns "ContentSecurityPolicy": {} — an empty object standing for a member the policy does not define — and ResponseHeadersPolicySecurityHeadersConfig has no required members while every one of its six members requires at least Override. So an empty member is always "undefined here" and is never a legal input.

⚠️ AND THE SAME IS TRUE ONE LEVEL UP, WHICH THE FIRST FIX MISSED. Every sibling member of ResponseHeadersPolicyConfig also declares required fields — CorsConfig five of them, RemoveHeadersConfig and CustomHeadersConfig a Quantity, ServerTimingHeadersConfig an Enabled — while the container itself requires only Name. So {} is a placeholder at both levels, and a fix covering only the inner one turns the outer placeholder into a hard abort instead of an omission. All of that is read out of the CLI's own service model, not inferred from the symptom.

State after the failure [verified 2026-09-04 — `get-distribution-config`, `list-response-headers-policies --type custom`, `list-origin-request-policies --type custom`]: /api/* still on b689b0a8-53d0-40ab-baf2-68738e2966ac, no *.pdf behaviour, zero custom response-headers policies, zero custom origin request policies.

⚠️ THE "ONE REVIEWABLE TRANSACTION" PROPERTY IS ABOUT THE DISTRIBUTION, NOT THE ACCOUNT — AND THIS RECORD ASSERTED THE WIDER VERSION FOR ONE ROUND. update-distribution is the script's last call, so a throw above it does leave the distribution untouched. But sections 4 and 5 each make their own write first — create-response-headers-policy and create-origin-request-policy — and the script's own comment on section 5's drift throw documents a reachable path where section 4 has already created adr-sml-pdf-noindex when section 5 aborts. So after any failed --apply, check for an orphaned policy as well as for a changed distribution, with both of these:

aws cloudfront list-response-headers-policies --type custom --output json \
  --query 'ResponseHeadersPolicyList.Quantity'
aws cloudfront list-origin-request-policies  --type custom --output json \
  --query 'OriginRequestPolicyList.Quantity'

An orphan is harmless and self-healing — the next run finds it by name, matches it and attaches it — so do not delete it by hand.

⚠️ THE EXPECTED VALUES DIFFER BY WHICH FAILURE YOU ARE RECOVERING FROM. After attempt 1 both returned 0 and nothing needed doing. After attempt 2 they return 1 and 0 — the response-headers policy is the orphan recorded below, and 1 is the correct reading, not a second problem [verified 2026-09-04].

Two changes came out of it. The clone now omits any empty member at either of those two levels — a ResponseHeadersPolicyConfig member, or a SecurityHeadersConfig member — and the dry run asserts that the generated config carries no empty object at any other level, naming the dotted path if it does. The asymmetry is deliberate: those two levels are where AWS is known to synthesise a placeholder, and anywhere else is unaccounted for and stops the run rather than being discarded quietly. The assertion runs before every branch, so the dry run now catches this class rather than an --apply discovering it — and if it ever does fire it skips section 4 rather than throwing, so router.js can still be re-applied.

The proof is a command rather than a session, which is the point of infra/cloudfront/policy-shapes.mjs existing as its own module — configure.mjs reads argv and calls AWS at import time, so the two functions could not otherwise be reached:

node infra/cloudfront/policy-shapes.test.mjs

Expect policy-shapes: 53 of 53 cases pass, exit 0. Its first case is this incident verbatim — the live SecurityHeadersConfig, empty ContentSecurityPolicy and all. The same suite covers attempt 2's limit checks, so this is the only command in this Part that proves both.


ATTEMPT 2 — section 4 SUCCEEDED, section 5 FAILED, and the run left an orphan. With the clone fixed, create-response-headers-policy created adr-sml-pdf-noindex = 51c4e79b-d9c6-4c6f-907c-dbb0e73dd374. Section 5 then failed:

An error occurred (InvalidArgument) when calling the CreateOriginRequestPolicy
operation: The parameter Comment is too big

Its Comment was 182 characters against a 128 cap. update-distribution never ran, so the distribution is untouched — but the account now holds a response-headers policy that no behaviour references.

🛑 DO NOT DELETE THAT POLICY BY HAND. This is the orphan case Part 3 and §7 predicted before it happened, and the recovery is measured, not asserted [measured 2026-09-04, dry run, exit 0, nothing written]: the next run finds it by name, matches it on every reconciled field, and stages it for attachment —

· response-headers policy adr-sml-pdf-noindex exists and matches the default behaviour
   + CacheBehaviors += *.pdf -> …, adr-sml-pdf-noindex (51c4e79b-d9c6-4c6f-907c-dbb0e73dd374)

— with the create line gone and the change count down from 4 to 3. No duplicate and no name collision. (A collision would not be silent either: a duplicate name returns ResponseHeadersPolicyAlreadyExists, which is a different error from the InvalidArgument above. The script never reaches it, because it looks the policy up by name first.)

⚠️ Comment IS NOT RECONCILED, so 51c4e79b keeps its original 118-character text while the script now carries a shorter one. Deliberate: adding Comment to the drift check would throw on this very policy and block the run that attaches it.

WHY NOTHING CAUGHT IT LOCALLY, AND THIS IS THE GENERAL LESSON. Measured 2026-09-04 against aws-cli/2.34.53's bundled botocore/validate.py: it checks neither max nor patternrange_check() reads only min, and the word pattern does not appear in the file. And the 128 cap is not modelled as a constraint at all: on both policy configs Comment is a bare string, and the number lives in the shape's documentation prose.

⚠️ THAT NAMES ONE VALIDATOR, DELIBERATELY. This machine also carries aws-cli/2.11.15, which is PyInstaller-frozen and whose validate.py cannot be read — unchecked, not confirmed. The claim that holds without qualification is the narrower and more useful one: the 182-character Comment reached the API and came back InvalidArgument, so nothing stopped it on the CLI that ran. That is why the pre-flight below had to be built rather than relied upon.

BOTH POLICY COMMENTS ARE NOW UNDER 80 CHARACTERS, and the dry run enforces the limits it knows about — infra/cloudfront/policy-shapes.mjs, PAYLOAD_LIMITS, one entry per limit with the source it came from. Every entry with a cited AWS source is enforced by the service and by nothing local — see below. Two entries are stamped [assumed] and are not: no AWS source states a policy name length, and those two constrain nothing this script sends (our names are 19 and 26 characters). A breach skips its section rather than throwing, and — the part that matters — skips the whole section, so a policy that is not created is never staged for attachment. The suite named earlier in this Part (node infra/cloudfront/policy-shapes.test.mjs) covers both the 182-character Comment that failed here and the 118-character one that did not.


One of the · lines carries a number worth reading, and it is not a warning:

· cloning 5 defined security header(s); omitting 1 the source does not define (ContentSecurityPolicy)

Five is the number to read. It is the count of security headers the PDF policy will carry, and the verification block at the end of this Part counts the same five arriving on the live PDF. A drop in this number is a partial clone announcing itself one step earlier than that curl would.

⚠️ RUN IT WITHOUT --function-arn ONLY IF THE ROUTER IS ALREADY ATTACHED. Omitting the flag prints · no --function-arn given, leaving FunctionAssociations alone and skips change 1 — which is right on a re-run and wrong on a first one, and the two look identical in a count.

🛑 SECTION 5 IS THE ONLY ONE THAT REPLACES SOMETHING, AND WHAT IT REPLACES IS ON THE INTAKE FORM'S PATH. Sections 14 add. Section 5 swaps the origin request policy on /api/* from Managed-AllViewerExceptHostHeader to a whitelist of five headers, because AWS has no behaviour meaning "all viewer headers except Host, plus a CloudFront header" — allExcept can only subtract, and allViewerAndWhitelistCloudFront drags Host along and 403s at API Gateway. Whitelisting is therefore forced, and the cost is that a header missing from that list is a header the handler never sees. The list is the handler's four headerOf reads plus CloudFront-Viewer-Address. The check prints the names, so they can be compared to the whitelist rather than counted:

grep -o "headerOf(event, '[a-z-]*'" backend/intake/handler.mjs \
  | sed "s/.*'\(.*\)'/\1/" | sort

Expect exactly content-type, origin, referer, user-agent. ⚠️ grep -n "headerOf(event" WAS PRESCRIBED HERE AND RETURNS FIVE — it matches function headerOf(event, name), the definition itself — so an operator comparing it against a documented "four" concludes the handler grew a read.

The failure mode is not an error. Every submission would validate short and redirect to /contact/could-not-send/ — a real inquirer would read it as their own browser misbehaving, and nothing would appear in a log as a fault. So the block below is not optional after an --apply that includes change 8, and there are three of them. The first is Part 7.1's probe with its output read differently — not "unchanged", which this said for one round: §7.1 pipes into head -12 and reads the status by eye, while these read curl's own exit status and count the location separately.

Run all three, in this order, and each answers a different question:

# probe what only it can tell you
1 Origin + body Origin is still forwarded — a 403 means it is not
2 Referer, no Origin the Firefox fallback still works — nothing else tests it
3 honeypot value the body parsed — probes 1 and 2 return the same 303 whether it did or not

PROBE 1 — is Origin still forwarded?

curl -si -X POST "$SITE/api/intake" \
  -H 'Origin: https://adr.smlcompany.ca' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data 'probe=1' -o /tmp/api.h
echo "curl_exit=$?"                       # curl's OWN status, on its own line
head -1 /tmp/api.h
grep -ic '^location: .*could-not-send' /tmp/api.h

Expect curl_exit=0, HTTP/2 303, and 1. A 403 here means the Origin header is no longer reaching the handler — i.e. the whitelist dropped it — and the form is broken for everyone.

PROBE 2 — the Referer fallback, which nothing else tests. The handler accepts Referer when Origin is absent (Firefox omits Origin on some same-origin form navigations), so a whitelist that forwarded Origin and dropped Referer passes probe 1 and fails for exactly those users:

curl -si -X POST "$SITE/api/intake" \
  -H 'Referer: https://adr.smlcompany.ca/contact/' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data 'company_website=probe' -o /tmp/api3.h
echo "curl_exit=$?"
head -1 /tmp/api3.h
grep -ic '^location: .*contact/received' /tmp/api3.h

Expect curl_exit=0, 303 and 1 [verified against production 2026-09-04 — it returns 303 today, on the managed policy]. A 403 means Referer is not being forwarded.

🛑 PROBE 3, AND NEITHER OF THE FIRST TWO CAN REPLACE IT: THEY CANNOT FAIL IN THE INTERESTING DIRECTION. 303 → could-not-send is what the handler returns both when it parsed the body and found an empty submission and when parseBody threw because Content-Type never arrived. Two opposite outcomes, one status, one location — so a dropped Content-Type reads as a pass. This probe separates them, and writes nothing and sends nothing:

curl -si -X POST "$SITE/api/intake" \
  -H 'Origin: https://adr.smlcompany.ca' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data 'company_website=probe' -o /tmp/api2.h
echo "curl_exit=$?"
head -1 /tmp/api2.h
grep -ic '^location: .*contact/received' /tmp/api2.h

Expect curl_exit=0, HTTP/2 303, and 1 — location /contact/received/, not could-not-send. That is the honeypot branch: it is reached only if the body parsed, and it returns before validation, before any DynamoDB write and before any SES send, so it leaves no record and sends no email. could-not-send here means the body did not parse — Content-Type is missing from the whitelist. Roll back.

⚠️ IT DEPENDS ON THE HONEYPOT'S NAME (company_website, fields.mjs). If that is ever renamed, this probe degrades to the could-not-send branch — which reads as a failure and starts an investigation, not as a pass. That direction is the safe one; keep it that way if you change the probe.

ROLLBACK, and it is one field. Do not debug a broken intake form in place:

# ⚠️ THIS RETURNS THE ID THE BEHAVIOUR HAS NOW — which, if change 8 applied, is
# the whitelist you are rolling back FROM, not the value to restore. The value to
# restore is the managed id on the line below. Run this to confirm which state
# you are in, then PUT the managed id back with
# update-distribution --if-match. ⚠️ NOT by re-running configure.mjs: section 5
# converges FORWARD and cannot tell a deliberate revert from a first run — the
# two are byte-identical in the config — so --apply would re-attach the
# whitelist and put the form back in the state you are rolling back from.
aws cloudfront get-distribution-config --id "$DIST_ID" \
  --query 'DistributionConfig.CacheBehaviors.Items[?PathPattern==`/api/*`].OriginRequestPolicyId'
# Managed-AllViewerExceptHostHeader = b689b0a8-53d0-40ab-baf2-68738e2966ac

Set that behaviour's OriginRequestPolicyId back to b689b0a8-53d0-40ab-baf2-68738e2966ac and update-distribution with the current ETag. configure.mjs prints the same id on the line it changes, prefixed , at the moment it changes it.

⚠️ THE HANDLER STILL STORES THE EDGE ADDRESS AFTER THIS. Forwarding the header does not change what is recorded, and it must not be made to as a follow-up edit: what the record holds is published field by field on /legal/privacy/, so storing CloudFront-Viewer-Address is a disclosure change governed by §7.2's decision table, not a code tidy. Pouya's ruling of 2026-09-04 is measured, not yet acted on.

⚠️ AND adr-sml-pdf-noindex IS RECONCILED ON EVERY RUN, NOT ONLY CREATED. A response-headers policy replaces rather than merges, so the PDF policy has to carry everything the default behaviour's policy carries. If they have diverged — someone adds the Content-Security-Policy or Permissions-Policy that docs/05 specifies to one and not the other — the script throws and names the diff rather than passing. That is deliberate: the failure it guards is the PDF being served different headers from the pages, which is silent.

node infra/cloudfront/configure.mjs --dist "$DIST_ID" --api-domain "$API_DOMAIN" \
  --function-arn "$ROUTER_ARN" --apply
aws cloudfront wait distribution-deployed --id "$DIST_ID"
echo "deployed: $?"

Verify the config landed:

aws cloudfront get-distribution-config --id "$DIST_ID" \
  --query 'DistributionConfig.{Fn:DefaultCacheBehavior.FunctionAssociations.Items[].EventType,Err:CustomErrorResponses.Items[].{Code:ErrorCode,Page:ResponsePagePath,Status:ResponseCode},Beh:CacheBehaviors.Items[].{P:PathPattern,O:TargetOriginId,RHP:ResponseHeadersPolicyId,Fn2:FunctionAssociations.Items[].EventType,Methods:AllowedMethods.Items},Origins:Origins.Items[].Id}'

Expect: Fn: ["viewer-request"]; one error response 404 → /404.html → 404; two behaviours — /api/*intake-api, POST in its method list, no RHP and Fn2: null (the association is withheld there deliberately: a 301 would turn the form's POST into a GET and drop the body), and *.pdf → the S3 origin with an RHP id and Fn2: ["viewer-request"]; two origins.

⚠️ THAT QUERY DOES NOT PROJECT OriginRequestPolicyId, SO IT CANNOT SEE CHANGE 8. Read it separately rather than concluding anything from its absence:

aws cloudfront get-distribution-config --id "$DIST_ID" \
  --query 'DistributionConfig.CacheBehaviors.Items[].{P:PathPattern,ORP:OriginRequestPolicyId}'

Expect /api/* carrying the adr-sml-api-viewer-address id — not b689b0a8-53d0-40ab-baf2-68738e2966ac, which is the managed policy it replaced and is what a rollback restores.

Then verify the header actually arrives, because the config landing is not the same fact:

curl -D /tmp/pdf.h -o /dev/null "$SITE/pouya-lajevardi-bio.pdf"
echo "curl_exit=$?"      # curl's OWN status, on its own line
for h in x-robots-tag strict-transport-security x-content-type-options \
         x-frame-options x-xss-protection referrer-policy; do
  printf '%-28s %s\n' "$h" "$(grep -ic "^$h:" /tmp/pdf.h)"
done

Expect curl_exit=0 and 1 against every one of the six — the five security headers and x-robots-tag.

⚠️ THE SHAPE OF THIS BLOCK IS THE POINT, and its first version got all three wrong. It piped curl -sI into one grep -E with six alternatives and read $?. That reports grep's status, not curl's, so a DNS failure, a TLS failure and a 5xx all read as exit=1 — indistinguishable from "the headers are missing", with -s deleting the message that would have told them apart. And an alternation exits 0 if ANY ONE matches, so exit=0 would not have meant the five arrived, which is the only regression the block exists to catch. Counting each header separately is what makes a partial clone visible. (CLAUDE.md: never suppress stderr, never read a pipeline's status as its first command's, and a uniform pass is the result that ends a check rather than starting one.)

⚠️ If any of the five security headers reads 0, the policy did not clone them and the PDF has LOST headers it had before this change.


Part 4 — Verify the distribution before the site exists

Run this now, against the old bucket contents. It is the check that the plumbing works independently of the deploy.

for p in / /index.html /about/ /nope-not-a-page/; do
  printf '%-22s ' "$p"
  curl -s -o /dev/null -w 'status=%{http_code} ct=%{content_type}\n' "${SITE}${p}"
done
printf 'no-slash redirect: '; curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' "${SITE}/about"

Expect: / and /index.html200 text/html (the old page still serves — the router did not break it); /about/ and /nope-not-a-page/404 rather than the 403 they returned before, because Part 1 changed S3's answer and Part 3 mapped it; /about301 → https://adr.smlcompany.ca/about/.

The body of those 404s is not the 404 page yet — 404.html is not in the bucket until Part 8. The status is what Part 4 proves; the page is what Part 8.4 proves. Do not tick docs/06's "404 returns a 404 status" item here; tick it after 8.4, when both halves are true at once.


Part 5 — Deploy the handler

5.1 Package

🛑 THREE FILES SINCE 2026-09-04, AND THE ZIP FOLLOWS NO IMPORT. handler.mjs imports both ./fields.mjs and ./spam-score.mjs; a zip missing either fails at cold start with Runtime.ImportModuleError and every submission then 500s. The list is now derived from the directoryls *.mjs minus the tests — in this step and in 5.5, so a new module is packaged without editing anything. It was typed out in both until 2026-09-04, and this banner still said so, fifteen lines above the paragraph that says otherwise.

rm -f /tmp/intake.zip
(cd backend/intake \
  && echo "packaging: $(ls *.mjs | grep -v '\.test\.' | tr '\n' ' ')" \
  && zip -q -X /tmp/intake.zip $(ls *.mjs | grep -v '\.test\.'))
unzip -l /tmp/intake.zip

⚠️ THE LIST IS SUBSTITUTED DIRECTLY, NOT HELD IN A VARIABLE, AND THAT IS NOT STYLE. A first version read MODULES=$(ls …) then zip … $MODULES. In zsh that packages ONE file whose name is all three joined by newlines — zsh does not word-split parameter expansions, only command substitutions — so it fails on the shell this project is actually operated from while working in bash. CLAUDE.md names this trap; it was reintroduced here and caught by running the block in both shells rather than by reading it.

Expect: exactly three entries — handler.mjs, fields.mjs, spam-score.mjs42,604 bytes uncompressed and 18,462 zipped [measured 2026-09-04]. All three at the zip root: the imports are ./-relative, so a nested directory breaks them at cold start. (This read "two entries, ≈ 25.7 KB / ≈ 10.8 KB" [measured 2026-09-01], before the scorer existed.)

⚠️ spam-score.test.mjs IS NOT IN THE ZIP AND MUST NOT BE. Run it at a keyboard — node backend/intake/spam-score.test.mjs, 39 of 39 — before packaging. It is the only check on the scorer, whose failure mode is labelling real inquiries rather than throwing.

5.2 Configuration first, code second

The handler calls requireEnv() at module scope and throws at cold start on any of six missing variables — deliberately, so a misconfigured function cannot accept a submission it will not store. Set the configuration before the code, so there is never a moment when the new code runs against no environment.

ENV_JSON=$(node scripts/intake-env.mjs --table "$TABLE" --notify "$NOTIFY_TO" --from "$MAIL_FROM")
echo "$ENV_JSON" | python3 -m json.tool

Expect six variables. ⚠️ RESPONSE_TIME and NO_RETAINER_NOTICE are read out of src/data/site.ts and must never be typed here. Both are published commitments — the two-business-day response (§4, Q27) and the no-retainer notice docs/01 §/contact/ requires, including its fourth clause about not itself creating a conflict check, which a hand-typed copy inside the handler had already dropped once. The notice also contains an en dash in "mediatorparty", which is exactly the character a retype loses. scripts/intake-env.mjs asserts both and exits non-zero rather than emitting a softened commitment.

aws lambda update-function-configuration --function-name "$FN" \
  --handler handler.handler --timeout 15 --memory-size 512 --environment "$ENV_JSON"
aws lambda wait function-updated --function-name "$FN"
aws lambda get-function-configuration --function-name "$FN" \
  --query '{Handler:Handler,Timeout:Timeout,Memory:MemorySize,Vars:sort(keys(Environment.Variables)),State:State,Update:LastUpdateStatus}'

Expect: Handler: handler.handler — the file is handler.mjs and the export is handler, where the hand-built function was index.handler; Timeout: 15; Memory: 512; the six names sorted; State: Active; Update: Successful.

Timeout 10 → 15 s and memory 128 → 512 MB are changes, not restatements. The handler does a DynamoDB write and two SES sends per invocation, and 128 MB is where an AWS SDK cold start is slowest — more memory shortens the billed duration enough that it is usually the cheaper setting, not the dearer one.

5.3 Code

aws lambda update-function-code --function-name "$FN" --zip-file fileb:///tmp/intake.zip
aws lambda wait function-updated --function-name "$FN"
aws lambda get-function-configuration --function-name "$FN" \
  --query '{CodeSize:CodeSize,Runtime:Runtime,Update:LastUpdateStatus,Modified:LastModified}'

Expect Update: Successful, and a CodeSize that says which path you took — it is the ZIP, not the source:

path expected CodeSize
5.1's plain three-file zip ≈ 18,462
5.5's bundled variant low single-digit MB — it was 3,307,021 on 2026-09-02 [measured 2026-09-04 — get-function-configuration]

🛑 5.5 IS THE PATH THAT WAS ACTUALLY TAKEN AT CUTOVER. The live function carries the bundled zip, so a redeploy that runs 5.1 and stops would replace it with an unbundled one and reintroduce the Runtime.ImportModuleError 5.5 exists to fix. Run 5.4 after 5.3, every time, and follow it to 5.5 if it fires.

(This line said "≈ 10,800", and before that "around 23,000" — 5.1's uncompressed figure applied to a different quantity. Both were written against the unbundled path, which is not the one in production.)

5.4 Prove it loads, without writing anything

cat > /tmp/probe-no-origin.json <<'JSON'
{"version":"2.0","routeKey":"POST /api/intake","rawPath":"/api/intake",
 "headers":{"content-type":"application/x-www-form-urlencoded"},
 "requestContext":{"http":{"method":"POST","path":"/api/intake","sourceIp":"127.0.0.1"}},
 "body":"probe=1","isBase64Encoded":false}
JSON
aws lambda invoke --function-name "$FN" --cli-binary-format raw-in-base64-out \
  --payload file:///tmp/probe-no-origin.json /tmp/probe-out.json
cat /tmp/probe-out.json; echo

Expect: the invoke prints "StatusCode": 200 with no FunctionError, and /tmp/probe-out.json is

{"statusCode":403,"headers":{"Cache-Control":"no-store"},"body":""}

That 403 is the handler's Origin check refusing a request with neither Origin nor Referer. It happens before any DynamoDB write and before any email, which is what makes this probe safe to run against production.

Two other outcomes, and each has one cause:

  • "FunctionError": "Unhandled" with Runtime.ImportModuleError in the body — the nodejs24.x runtime does not provide the AWS SDK v3 clients this handler imports. Run 5.5, then repeat 5.4.
  • A body naming intake handler: <NAME> is not set — that variable did not reach the function. Re-run 5.2 and read the Vars list.

5.5 Bundled variant — only if 5.4 said Runtime.ImportModuleError

Versions are resolved from the registry at run time rather than pinned in this file: CLAUDE.md's rule is that a version is checked against the registry and never recalled, and a literal here would be stale the week after it was written.

⚠️ THIS PATH WAS TAKEN — 2026-09-02, and the live function is the bundled zip. ⚠️ THE TWO RESOLVED VERSIONS AND THE COMMIT THE DEPLOYED SOURCES MATCH ARE IN AGENTS.md §7 AND ARE DELIBERATELY NOT REPEATED HERE. They were repeated here until 2026-09-04, and the paragraph directly above is the argument against it — the literals went stale in two days rather than a week: the 2026-09-04 redeploy moved both packages one patch and added a third source file, and this copy still named the old versions and a superseded commit while reading as a measurement. Read §7's two Lambda rows; this step's duty is to update them, not to mirror them.

THE cp AND zip LINES BELOW DERIVE THE FILE LIST THE SAME WAY 5.1 DOES. They were a second hand-typed copy until 2026-09-04, not derived from 5.1's and with nothing checking that the two agreed — so a module added to one and not the other would ship from whichever path the operator happened to take. Both now read the directory.

rm -rf /tmp/intake-bundle && mkdir -p /tmp/intake-bundle
echo "bundling: $(cd backend/intake && ls *.mjs | grep -v '\.test\.' | tr '\n' ' ')"
(cd backend/intake && cp $(ls *.mjs | grep -v '\.test\.') /tmp/intake-bundle/)
# ⚠️ ASSERT THE COPY LANDED. A glob that matches nothing makes `cp` fail, `zip`
# succeed on an empty set, and `update-function-code` upload a bundle with no
# handler — a silent failure that only shows up as 5xx on the live form.
test -f /tmp/intake-bundle/handler.mjs || { echo "FATAL: handler.mjs not copied"; exit 1; }
echo "copied: $(ls /tmp/intake-bundle/*.mjs | wc -l | tr -d ' ') module(s)"
( cd /tmp/intake-bundle \
  && npm init -y > /dev/null \
  && npm install --omit=dev --no-audit --no-fund \
       "@aws-sdk/client-dynamodb@$(npm view @aws-sdk/client-dynamodb version)" \
       "@aws-sdk/client-sesv2@$(npm view @aws-sdk/client-sesv2 version)" )
rm -f /tmp/intake.zip
( cd /tmp/intake-bundle && zip -qr -X /tmp/intake.zip $(ls *.mjs) node_modules package.json )
unzip -l /tmp/intake.zip | tail -1
aws lambda update-function-code --function-name "$FN" --zip-file fileb:///tmp/intake.zip
aws lambda wait function-updated --function-name "$FN"

Expect a zip in the low single-digit megabytes, well under the 50 MB direct- upload limit. If this path is taken, say so in the AGENTS.md Change Log and add both packages to §7 — they become pins this project maintains, and R11 covers them from that moment.


Part 6 — Route the API at /api/intake

6.1 The invoke permission, before the route

⚠️ THE EXISTING PERMISSION DOES NOT COVER THE NEW ROUTE, AND THIS IS THE STEP WHOSE OMISSION IS HARDEST TO DIAGNOSE. Part 0.3 records the one statement on the function: SourceArn …/4tl0m5igkj/*/*/submissions. Add a route at /api/intake without adding a permission for it and API Gateway is refused permission to invoke — the caller gets a 500, the Lambda logs nothing at all because it was never entered, and the only trace is an API Gateway metric.

aws lambda add-permission --function-name "$FN" \
  --statement-id apigw-post-api-intake \
  --action lambda:InvokeFunction \
  --principal apigateway.amazonaws.com \
  --source-arn "arn:aws:execute-api:${AWS_REGION}:${ACCT}:${API_ID}/*/POST/api/intake"
aws lambda get-policy --function-name "$FN" --query Policy --output text | python3 -m json.tool

Expect: two statements — the old one scoped to …/submissions and the new one to …/POST/api/intake.

6.2 The route

export INTEG_ID=$(aws apigatewayv2 get-integrations --api-id "$API_ID" \
  --query "Items[?IntegrationUri=='arn:aws:lambda:${AWS_REGION}:${ACCT}:function:${FN}'].IntegrationId | [0]" \
  --output text)
echo "INTEG_ID=$INTEG_ID"
[ -n "$INTEG_ID" ] && [ "$INTEG_ID" != "None" ] || echo "STOP — no integration points at $FN"
aws apigatewayv2 create-route --api-id "$API_ID" \
  --route-key 'POST /api/intake' --target "integrations/${INTEG_ID}" \
  --query '{RouteId:RouteId,RouteKey:RouteKey,Target:Target}'
aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].{Key:RouteKey,Target:Target}'

Expect: INTEG_ID non-empty and not None; then two routes, POST /submissions and POST /api/intake, both on the same integration. The stage is $default with auto-deploy on, so the route is live immediately — there is no deployment to create.

6.3 Throttling

⚠️ THIS IS NOT THE PER-IP LIMIT docs/05 ASKS FOR, AND IT CANNOT BE. That spec says "Rate limit by source IP at API Gateway: 5 requests / 5 minutes". API Gateway throttling is aggregate — per route and per stage — not per source IP. Per-IP rate limiting needs AWS WAF with a rate-based rule on the distribution, which is a paid service and therefore Pouya's decision, recorded on docs/06's checklist as explicitly not a launch blocker. What is set here is a real control over total volume; describing it as per-IP would be the Q22 shape again — a documented control that does not exist.

aws apigatewayv2 update-stage --api-id "$API_ID" --stage-name '$default' \
  --route-settings '{"POST /api/intake":{"ThrottlingRateLimit":1,"ThrottlingBurstLimit":5,"DetailedMetricsEnabled":true}}'
aws apigatewayv2 get-stage --api-id "$API_ID" --stage-name '$default' \
  --query 'RouteSettings'

Expect: the POST /api/intake entry with rate 1/s, burst 5, detailed metrics on. One per second sustained with a burst of five is far above any human filling this form and far below anything worth paying for.

6.4 Retire the old route

Nothing posts to /submissions any more — the form posts to /api/intake and the handler that answered the old contract is gone. An unused public route on the same function is surface with no purpose.

export OLD_ROUTE=$(aws apigatewayv2 get-routes --api-id "$API_ID" \
  --query "Items[?RouteKey=='POST /submissions'].RouteId | [0]" --output text)
echo "OLD_ROUTE=$OLD_ROUTE"
aws apigatewayv2 delete-route --api-id "$API_ID" --route-id "$OLD_ROUTE"
aws lambda remove-permission --function-name "$FN" \
  --statement-id 70aeb597-e4d8-5141-8fb1-6b6190f7b2ab
aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].RouteKey'
aws lambda get-policy --function-name "$FN" --query Policy --output text | python3 -m json.tool

Expect: one route, POST /api/intake; one permission statement, scoped to it. Confirm the statement id in the remove-permission call against the policy you printed in 6.1 before running it — it is an id, not a name, and it will differ if the permission was ever rebuilt.

The execute-api hostname stays reachable, so /api/intake can be reached without passing through CloudFront. That is not a new weakness and not a loose end: DisableExecuteApiEndpoint cannot be turned on, because the distribution's own origin is that hostname. The Origin check is a CSRF control rather than authentication either way, and 6.3's throttle applies at the stage, so it covers both paths.


Part 7 — End to end, with a real submission

7.1 The route answers

curl -si -X POST "${SITE}/api/intake" \
  -H 'Origin: https://adr.smlcompany.ca' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data 'deploy-route-probe=1' | head -12

Expect: HTTP/2 303 and location: https://adr.smlcompany.ca/contact/could-not-send/. The handler accepted the request, validated it, found an empty submission and redirected to the failure page — before any write and any email. This is the same probe scripts/deploy-local.sh runs at the end of every deploy.

⚠️ 404 IS AMBIGUOUS AND THE FIRST DRAFT OF THIS LINE WAS NOT. It means either the /api/* behaviour is missing (Part 3) or the POST /api/intake route is missing or misspelled (Part 6.2) — two different repairs. And because Part 3's custom error response is distribution-wide, a 404 from API Gateway is served as the styled /404.html body, so the one string that would have named the cause — API Gateway's {"message":"Not Found"} — is replaced before you see it. Check the route first; it is one command: aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].RouteKey'.

403 means the Origin header did not arrive, and as of 2026-09-04 there are two policies it could be — read which one the behaviour carries before repairing:

  • adr-sml-api-viewer-address (Part 3, change 8) — a whitelist. If Origin is missing from its Headers list, or the list drifted, every real submission 403s. Roll back by PUTting the managed id below with update-distribution --if-matchnot by re-running configure.mjs, which converges forward and would re-attach the whitelist.
  • Managed-AllViewerExceptHostHeader (b689b0a8-53d0-40ab-baf2-68738e2966ac) — what it replaced, and what a rollback restores.

Either way, a policy that drops Origin turns every real submission into a 403. 500 means Part 6.1 was skipped.

7.2 A real submission, from the real form

Do this in a browser at ${SITE}/contact/ after Part 8, because the form does not exist until the site is deployed. Fill it as an inquirer would, with Your name: Cutover test <date> so the record is identifiable, and a real address you can read.

Expect: the browser lands on /contact/received/, and two emails arrive — the notification at $NOTIFY_TO, replyable to the address you entered, and the confirmation at that address. Read the confirmation and check three things: the response-time sentence, the four-clause no-retainer notice, and that the field summary uses readable labels ("Your name", "Subject matter") rather than field names.

Then find the record. The notification email prints submissionId <uuid> — that value is the partition key, so it can be used directly:

export SUB_ID='<the uuid from the notification email>'
aws dynamodb get-item --table-name "$TABLE" \
  --key "{\"submissionId\":{\"S\":\"${SUB_ID}\"}}" \
  --query 'Item.{id:submissionId.S,at:submittedAt.S,ttl:ttl.N,name:name.S,area:practiceArea.S,sourceIp:sourceIp.S}'
curl -s https://checkip.amazonaws.com

⚠️ READ sourceIp AGAINST YOUR OWN ADDRESS — THIS IS THE ONLY PLACE THE PROXY CHAIN GETS MEASURED, AND IT HAS ALREADY BEEN REASONED ABOUT WRONGLY TWICE. The handler stores requestContext.http.sourceIp, which is the TCP peer. Behind the /api/* behaviour that peer is a CloudFront edge, so the expectation is an AWS address, not yours. Three outcomes and each has a different consequence:

what sourceIp holds what it means
an AWS address (not the checkip value) As designed. The field records the CDN, so it cannot serve abuse investigation, and /legal/privacy/'s "your IP address" is inaccurate — fold it into the Q62 edit on the same page rather than leaving two wrong sentences there
your address, matching checkip Better than expected, and worth knowing before anyone relies on it. Do not conclude it is trustworthy: verify it is not simply echoing a header by resubmitting with -H 'X-Forwarded-For: 8.8.8.8' and confirming 8.8.8.8 is not what lands. ⚠️ AND CORRECT /legal/privacy/ §What is collected, whose network-address paragraph says the address is "normally the network's own rather than your connection's" — wrong in this branch, and it understates what is held about the reader. This row carried no instruction at all until 2026-09-02, so two of these three outcomes had nothing reconciling the page with the measurement (adversarial-reviewer, round 2)
8.8.8.8 after that resubmission Stop. The field is client-controlled and a record can be made to name an uninvolved third party. Revert to storing nothing rather than storing that

⚠️ AN EARLIER REVISION OF THE HANDLER READ x-forwarded-for HERE, AND THAT WAS THE THIRD OUTCOME. CloudFront appends the viewer address to a client-supplied XFF rather than replacing it, so the leftmost entry is whatever the client sent. The fix, if a usable value is wanted, is a custom origin request policy on /api/* forwarding CloudFront-Viewer-Address, which CloudFront generates and overwrites — not the managed AllViewerAndCloudFrontHeaders, which forwards Host and would 403 every request at API Gateway. ⚠️ THAT CHANGE IS NOW IN THIS RUNBOOK — Part 3, change 8, written 2026-09-04 on Pouya's ruling and NOT YET APPLIED. This paragraph said it was "deliberately not in this runbook: measure first", which was true until the ruling and false afterwards. Forwarding the header does not change what is stored: viewerIp() still records requestContext.http.sourceIp, and the decision table above is still the procedure for changing that, because what the record holds is published field by field on /legal/privacy/. Measure first still governs the STORING, not the forwarding.

Expect the item, with ttl a 10-digit epoch-seconds value. Check it is 24 months out — read it, do not assume it:

python3 - <<PY
import datetime, os
ttl = int("${SUB_ID}" and os.popen(
  'aws dynamodb get-item --table-name ${TABLE} '
  '--key \'{"submissionId":{"S":"${SUB_ID}"}}\' --query Item.ttl.N --output text'
).read().strip())
d = datetime.datetime.fromtimestamp(ttl, datetime.timezone.utc)
now = datetime.datetime.now(datetime.timezone.utc)
months = (d.year - now.year) * 12 + (d.month - now.month)
print(f'ttl {ttl} -> {d.isoformat()}  ({months} months from now)')
print('PASS' if months == 24 else 'FAIL — /legal/privacy/ promises 24 months')
PY

Expect: 24 months from now and PASS. /legal/privacy/ states the period publicly, so a number that is not 24 is a false disclosure rather than a config error.

⚠️ DELETE THE TEST RECORD when you are done with it — it is a real row in a table whose contents are governed by a published privacy policy.

aws dynamodb delete-item --table-name "$TABLE" --key "{\"submissionId\":{\"S\":\"${SUB_ID}\"}}"

7.3 The four pre-existing items

The table held 4 items before any of this [verified 2026-09-01 — describe-table ItemCount]. They were written by the hand-built handler, whose code is in docs/reference/AWS-Hosting-Guide.md Part 8.3 and writes no ttl attribute — so DynamoDB will never expire them and they are retained indefinitely, against a policy that says otherwise.

This is Pouya's call and not a command in this runbook, because it depends on something no agent can determine: whether those four are the guide's own smoke test (Test User) or real inquiries that arrived through the old site. Read them in the console, then either delete the test rows or write a ttl onto the real ones. Tracked on docs/06's checklist.


Part 8 — The first production deploy

⚠️ DO NOT RUN 8.3 UNTIL POUYA HAS READ EVERY PAGE. docs/06's Content and compliance group — the claims-auditor pass and his own page-by-page read — is the gate on this part, and it is not a technical check.

8.1 Archive what is there

S3 versioning is Enabled, so pass 3's --delete is recoverable — but docs/06's post-cutover list says to archive the old build rather than rely on that, and a local copy costs one command.

mkdir -p _archive/pre-cutover-$(date +%F)
aws s3 sync "s3://${BUCKET}" "_archive/pre-cutover-$(date +%F)/" --no-progress
ls -la "_archive/pre-cutover-$(date +%F)/"
find "_archive/pre-cutover-$(date +%F)" -type f | wc -l

Expect: 3 files — index.html at 2,206,032 bytes and two PNGs under assets/. Do not commit them; docs/06 wants them kept, not versioned.

8.2 Dry run — the three passes, in order, writing nothing

npm run check && npm run build && npm run check:claims && npm run og:proof && npm run check:intake
echo "gates exit=$?"
npm run lighthouse; echo "lighthouse exit=$?"

Expect: every one exit 0. lighthouse is local-only and cannot run in CI; it is a keyboard gate and this is the keyboard.

aws s3 sync ./dist "s3://${BUCKET}" --dryrun \
  --exclude "*" --include "_astro/*" --include "fonts/*" \
  --cache-control "public, max-age=31536000, immutable" --no-progress | tail -5
aws s3 sync ./dist "s3://${BUCKET}" --dryrun \
  --exclude "*" --include "*.avif" --include "*.webp" --include "*.jpg" \
  --include "*.png" --include "*.svg" \
  --cache-control "public, max-age=604800" --no-progress | tail -5
aws s3 sync ./dist "s3://${BUCKET}" --dryrun \
  --exclude "_astro/*" --exclude "fonts/*" \
  --cache-control "public, max-age=0, must-revalidate" --delete --no-progress | tail -8

Expect, on the bucket state Part 0.3 records — measured 2026-09-01, so these are counts rather than shapes: 51 (dryrun) upload: lines in pass 1, 56 in pass 2, and 52 lines in pass 3 of which two are (dryrun) delete:assets/sml-logo-full.png and assets/sml-logo-mark.png, the old site's two logos. 404.html appears in pass 3. index.html is not deleted; it is overwritten.

⚠️ PASS 2's DRY RUN IS A SUPERSET OF WHAT PASS 2 ACTUALLY UPLOADS, AND THE REAL RUN WILL PRINT FEWER LINES. 34 of those 56 are files under _astro/, which pass 1 uploads first. Each --dryrun above is compared against the bucket as it is now, so it cannot see the uploads the preceding pass would have made; in the real sequential run aws s3 sync skips them as already in sync. That skipping is exactly what preserves pass 1's immutable header on those files rather than overwriting it with pass 2's week-long one — docs/06 §Cache policy says the same thing about pass 3, and this is why the pass order is load-bearing. Do not reorder the passes to make this output tidier, and do not read the smaller real counts as a failed upload.

⚠️ Check the largest file in ./dist while you are here. The deploy user has no s3:AbortMultipartUpload, which is safe only while nothing approaches aws s3 sync's 8 MB multipart threshold (docs/06).

find ./dist -type f -exec stat -f '%z %N' {} + | sort -rn | head -3

Expect the largest well under 8,388,608 bytes — it was 171,876 on 2026-09-01. GNU find -printf is not available here; this is the BSD/macOS form, and writing it as find -printf … || find -exec … would not have fallen back, because the exit status of that pipeline belongs to head.

8.3 The deploy — on Pouya's word only

⚠️ NOT AS user/pouya. The script refuses that identity and §10 is why: the account is shared with unrelated production systems. Use the adr-sml-deploy key, which §7 records as provisioned.

AWS_ACCESS_KEY_ID=AWS_SECRET_ACCESS_KEY=\
AWS_REGION="$AWS_REGION" S3_BUCKET="$BUCKET" CLOUDFRONT_DISTRIBUTION_ID="$DIST_ID" \
  npm run deploy

The script runs npm run check, the build, check:claims, the three passes, the /* invalidation, and then the intake route check. Read its last line. It warns rather than fails on a bad intake route, because by then the site is already published and failing the job would not un-publish it.

git tag "v$(date +%Y).1" && git push --tags

8.4 Verify the site, not the deploy

for p in / /about/ /mediation/ /arbitration/ /med-arb/ /practice/ /practice/construction/ \
         /process/ /for-parties/ /fees/ /insights/ /contact/ /bio/ /legal/privacy/ \
         /legal/terms/ /robots.txt /sitemap-index.xml /pouya-lajevardi-bio.pdf; do
  printf '%-32s ' "$p"
  curl -s -o /dev/null -w 'status=%{http_code} bytes=%{size_download}\n' "${SITE}${p}"
done
printf '%-32s ' "404 body"; curl -s "${SITE}/nope-not-a-page/" -o /tmp/404.html -w 'status=%{http_code}\n'
grep -c 'That page is not here' /tmp/404.html
printf '%-32s ' "cache header on HTML"; curl -sI "${SITE}/about/" | grep -i '^cache-control'
printf '%-32s ' "cache header on asset"
curl -s "${SITE}/about/" | grep -o '/_astro/[^"]*\.css' | head -1 \
  | xargs -I{} curl -sI "${SITE}{}" | grep -i '^cache-control'

Expect: every page 200 with real bytes; the 404 path 404 and the phrase That page is not here found once; max-age=0, must-revalidate on HTML; max-age=31536000, immutable on the hashed CSS. Only now is docs/06's "404 returns a 404 status" item true on both halves.


Part 9 — Rollback, per part

Each of these is independent. None of them needs the others undone first.

9.1 Part 1aws s3api put-bucket-policy --bucket "$BUCKET" --policy "file:///tmp/bucket-policy.before.json". Missing keys go back to 403 and the 404 mapping stops firing; nothing else changes.

9.2 Parts 23 — re-run configure.mjs is not a rollback; it is idempotent forward-only, and that now matters most for change 8: section 5 re-attaches the /api/* whitelist on the next --apply, because a deliberately reverted behaviour and a never-configured one are byte-identical in the config and no detector can separate them. To undo, get-distribution-config, remove the FunctionAssociations entry / the 404 custom error response / the /api/* behaviour and the intake-api origin, and update-distribution --if-match.

Sections 4 and 5 were added after this paragraph and undo the same way: put /api/*'s OriginRequestPolicyId back to b689b0a8-53d0-40ab-baf2-68738e2966ac (the managed policy) and/or remove the *.pdf behaviour, with update-distribution --if-match. The two custom policies adr-sml-api-viewer-address and adr-sml-pdf-noindex can then be deleted with delete-origin-request-policy / delete-response-headers-policy, each of which fails while still attached — the same ordering feature as the function below. ⚠️ Deleting the policies does not prevent re-attachment either — the next --apply simply creates them again by name and attaches them. Nothing in this script can be made to remember a deliberate revert, because a reverted behaviour and a never-configured one are byte-identical in the config. The rollback holds only until someone runs configure.mjs --apply again; that is a property of a forward-converging script, and the fix if it ever matters is a flag, not a deletion. Then aws cloudfront delete-function --name adr-sml-router --if-match <etag>, which fails while the function is still associated — that ordering is a feature.

9.3 Part 5 — the previous code is not recoverable from Lambda: the hand-built function was edited in the console and its 1,527 bytes exist only in docs/reference/AWS-Hosting-Guide.md Part 8.3. That is the rollback source, and it answers a different contract (JSON in, {"ok":true} out) that the current /contact/ form does not speak. Forward is the only real fix here — which is worth knowing before 5.3, not after.

9.4 Part 6delete-route the new route, remove-permission the new statement, and re-add the /submissions route and its permission if anything still needs it. Nothing does.

9.5 Part 8 — S3 versioning is Enabled. Restore the previous version of index.html, aws s3 rm what the new build added, invalidate /*. Or git revert and re-deploy, which is cleaner and is what the tag in 8.3 is for.


Part 10 — Q60: prove TTL deletes, not just that it is enabled

AGENTS.md §7 records TimeToLiveStatus: ENABLED on attribute ttl [re-verified 2026-09-01], and that proves the setting and not the behaviour. /legal/privacy/ tells the public that a record is "deleted automatically by the database rather than by someone remembering to do it" — an assertion about the mechanism. Q60 closes when a record has been watched to disappear, and not before. §12 R19 keeps it surfacing until then.

10.1 Write one record with a near-future ttl

Not through the form: the handler writes a 24-month ttl and waiting two years is not a test. Write it directly, with a shape that cannot be mistaken for an inquiry.

export TTL_ID="q60-ttl-probe-$(date +%Y%m%dT%H%M%SZ)"
export TTL_AT=$(python3 -c 'import time; print(int(time.time()) + 300)')
python3 -c "import datetime,os; print('ttl', os.environ['TTL_AT'], '->', datetime.datetime.fromtimestamp(int(os.environ['TTL_AT']), datetime.timezone.utc).isoformat())"
aws dynamodb put-item --table-name "$TABLE" --item "$(cat <<JSON
{"submissionId":{"S":"${TTL_ID}"},
 "submittedAt":{"S":"$(date -u +%Y-%m-%dT%H:%M:%SZ)"},
 "ttl":{"N":"${TTL_AT}"},
 "note":{"S":"AGENTS.md Q60 — TTL behaviour probe, no personal information, safe to delete"}}
JSON
)"
aws dynamodb get-item --table-name "$TABLE" --key "{\"submissionId\":{\"S\":\"${TTL_ID}\"}}" \
  --query 'Item.{id:submissionId.S,ttl:ttl.N,note:note.S}'
echo "$TTL_ID" > ~/q60-ttl-probe-id.txt
echo "$TTL_AT"  >> ~/q60-ttl-probe-id.txt

Expect: the item, with ttl five minutes in the future. The id is written to a file because the check below happens on a different day and this is the only thing that connects the two.

10.2 The check, and it is not five minutes later

⚠️ DYNAMODB'S TTL SWEEPER IS NOT PROMPT AND AWS DOES NOT PROMISE THAT IT IS. Deletion typically happens within a couple of days of the timestamp passing, and up to 48 hours or more is normal and not a fault. A check run at ttl + 5 minutes that finds the item present has measured nothing — it is the same mistake as reading a value before the transition finishes. So:

  • Earliest useful check: 48 hours after TTL_AT.
  • Do not conclude a failure before 7 days.
export TTL_ID=$(sed -n 1p ~/q60-ttl-probe-id.txt)
export TTL_AT=$(sed -n 2p ~/q60-ttl-probe-id.txt)
python3 -c "import time,os; d=time.time()-int(os.environ['TTL_AT']); print(f'{d/3600:.1f} hours since the ttl passed')"
aws dynamodb get-item --table-name "$TABLE" \
  --key "{\"submissionId\":{\"S\":\"${TTL_ID}\"}}" --output json
echo "get-item exit=$?"

Read both the output and the exit status. get-item on a missing key exits 0 with an empty response body — it is not an error, and a script that only checked the exit status would report the record present and absent identically.

result meaning
{"Item": {…}}, under 48 h since TTL_AT inconclusive. Wait.
{} (empty), any time after TTL_AT Q60 closes. The mechanism works.
{"Item": {…}}, more than 7 days after Q60 fails. TTL is enabled and not deleting. /legal/privacy/ is then making a claim the database does not honour, and the page has to change or the mechanism does.

10.3 Record it

On the {} result, and only then:

  • Stamp §7's Intake table row: the behaviour is observed, with the date and the elapsed time, alongside the ENABLED reading it already carries.
  • Close Q60 in §9, quoting the two timestamps.
  • Retire R19 from §12 — its whole purpose was to keep this surfacing.
  • Tick the TTL item on docs/06's checklist. It has two halves and this is the second: ENABLED proved the setting, this proves the behaviour.
  • Delete the probe row if it somehow survives, and delete ~/q60-ttl-probe-id.txt.

If it fails, that is a published-disclosure defect, not a backlog item: src/pages/legal/privacy.astro carries the matching TODO(pouya) and the page asserts the mechanism in as many words.