# 09 — Cutover runbook: the exact command sequence Authority: `AGENTS.md` §3 D11 (build everything, one clean cutover) and §7 for every operational fact. `docs/06-deployment.md` owns the cutover **checklist** — what must be true. This file owns the **commands** — how to make it true. The checklist cites this file; this file does not restate the checklist. **Pouya runs everything here.** Parts 1–7 need administrative credentials that no agent on this project holds, and Part 8 is a deploy that must not run as `user/pouya` (§10). Every command is followed by the command that verifies it and the output to expect. Where a verification can come back two ways, both are named and the sequence branches — a branch driven by a measurement rather than by a guess is the point. > **Read `docs/06`'s cutover checklist alongside this.** The parts below discharge > its Technical group. Its Content and compliance group — the `claims-auditor` > pass and Pouya's own page-by-page read — is not commands and is not here. --- ## Why the order is this order, and what is broken while it runs The bucket currently holds the **old** single-file site: `index.html` (2,206,032 bytes) plus two logo PNGs `[verified 2026-09-01 — list-objects-v2]`. So the live site today is one page, and one page is all that works: measured on the live distribution the same day, `/` returns **200** while `/about/` and `/definitely-not-a-page/` both return **403 with an 111-byte `application/xml` body** — S3's `AccessDenied`, served raw to the reader. That is why the infrastructure goes first and the site goes last: 1. **Parts 1–4 (distribution) change nothing a visitor can see.** The router rewrites `/` to `/index.html`, which is what the default root object already did, and the paths it newly handles do not exist on the old site either way. 2. **Parts 5–7 (intake) are inert until `/contact/` exists.** 3. **Part 8 is the cutover.** The moment `index.html` syncs, the new site is live — and by then the router, the 404 mapping and `/api/*` are already in place, so there is no window in which 22 of 23 pages are broken. Reversing this — site first, infrastructure second — would put every page except `/` behind a 403 for as long as the CloudFront deployment took. --- ## Part 0 — Preflight ### 0.1 The substitution block ⚠️ **THIS IS THE ONLY PLACE IN THIS FILE THAT NAMES A RESOURCE, AND THAT IS A DELIBERATE, BOUNDED EXCEPTION TO THE §7 RULE.** `AGENTS.md` §7 is the single source of truth for operational facts and specs cite it rather than copying it — but a runbook that cannot be pasted into a terminal is not a runbook. So the values appear exactly once, and **0.2 is the mechanism that keeps the copy honest**: it proves every one of them resolves to a real resource before anything is changed. A duplicated fact needs a mechanism, not a promise. Confirm each against §7 before pasting. `./scripts/aws-discover.sh` regenerates them from AWS if §7 itself needs re-verifying. ```bash cd ~/Dev/Websites/adr-sml export AWS_REGION=ca-central-1 export AWS_DEFAULT_REGION=ca-central-1 export ACCT=327082975128 export FN=adr-intake-handler export TABLE=adr-intake-submissions export API_ID=4tl0m5igkj export API_DOMAIN=4tl0m5igkj.execute-api.ca-central-1.amazonaws.com export DIST_ID=E1OK7G98KNKUTA export BUCKET=adr-smlcompany-site export SITE=https://adr.smlcompany.ca export NOTIFY_TO=info@smlcompany.ca export MAIL_FROM=intake@smlcompany.ca ``` `NOTIFY_TO` is where an inquiry lands and `MAIL_FROM` is the From on both messages. Both are choices rather than records: `info@` is the address `/contact/` publishes, so a reply threads where the inquirer expects; `intake@` distinguishes form mail from correspondence. Both must be at `smlcompany.ca`, which is the verified sending identity — 0.2 checks that. ### 0.2 Prove every one of them exists ```bash aws sts get-caller-identity for v in AWS_REGION ACCT FN TABLE API_ID API_DOMAIN DIST_ID BUCKET SITE NOTIFY_TO MAIL_FROM; do eval "val=\$$v" [ -n "$val" ] || { echo "EMPTY: $v"; break; } printf '%-12s %s\n' "$v" "$val" done aws lambda get-function-configuration --function-name "$FN" --query 'FunctionName' --output text aws dynamodb describe-table --table-name "$TABLE" --query 'Table.TableStatus' --output text aws apigatewayv2 get-api --api-id "$API_ID" --query 'Name' --output text aws cloudfront get-distribution --id "$DIST_ID" --query 'Distribution.Status' --output text aws s3api head-bucket --bucket "$BUCKET" && echo "bucket ok" aws sesv2 get-email-identity --email-identity "${MAIL_FROM#*@}" \ --query 'VerifiedForSendingStatus' --output text ``` **Expect:** an identity with administrative rights; eleven non-empty values; `adr-intake-handler`; `ACTIVE`; `adr-intake-api`; `Deployed`; `bucket ok`; `True`. A `None`, an empty value, or any non-zero exit stops the run here. That is the whole purpose of 0.2 — a stale identifier fails loudly at the top instead of halfway through Part 6. ### 0.3 State this run starts from Recorded so a later reader can tell what this sequence changed from what it found. All `[verified 2026-09-01]`, read-only: | | | |---|---| | Lambda | `nodejs24.x`, arm64, handler `index.handler`, timeout 10 s, memory 128 MB, **no environment variables**, no DLQ, code 1,527 bytes (the hand-built inline function), last modified 2026-05-26 | | Lambda role | `adr-intake-lambda-role` — `AWSLambdaBasicExecutionRole` plus an inline policy granting `dynamodb:PutItem` on the table and `ses:SendEmail`/`ses:SendRawEmail`. **Sufficient for the new handler; no change needed** | | Lambda invoke permission | one statement, `apigateway.amazonaws.com`, **`SourceArn` scoped to `…/4tl0m5igkj/*/*/submissions`** — the old route's path only | | HTTP API | one route, `POST /submissions` → integration `0ftgjgv` (`AWS_PROXY`, payload format **2.0**, the correct one for this handler). Stage `$default`, auto-deploy on, **no throttling**, no access log. CORS allows `POST` from the site origin | | DynamoDB | `ACTIVE`, on-demand, **partition key `submissionId` (S), no sort key**, 4 existing items, TTL **ENABLED** on attribute `ttl`, PITR **ENABLED** (35-day window), encryption at rest with the **AWS-owned** key (no CMK) | | CloudFront | one S3 origin with OAC `E13GAFUL6UQP6R`; default behaviour `Managed-CachingOptimized` + `Managed-SecurityHeadersPolicy`, methods HEAD/GET; **no cache behaviours**, **no custom error responses**, **no function associations** | | S3 | versioning `Enabled`; all four public-access blocks `true`; direct object GET on the S3 hostname returns 403, so OAC is in force | | SES | domain verified for sending, DKIM `SUCCESS` and signing enabled, no custom MAIL FROM | | SNS `ses-alerts` | email subscription to `info@smlcompany.ca` is **CONFIRMED** — it has a real subscription ARN, not `PendingConfirmation`. §7 recorded it as pending for six days after it stopped being pending; §12 R9 closes on this | --- ## Part 1 — Make S3 answer 404 instead of 403 **Why this is first.** Part 3 maps **404** to `/404.html` and deliberately does **not** map 403. With OAC and no `s3:ListBucket`, S3 answers a request for a missing key with `403 AccessDenied`, so a 404-only mapping would never fire and every bad URL would keep returning raw XML. Granting the CloudFront service principal `s3:ListBucket` changes that answer to `404 NoSuchKey`. Mapping 403 as well would have been one command shorter and is the wrong trade: custom error responses are distribution-wide and cannot be scoped to one behaviour, so it would also convert a broken bucket policy or a broken OAC — a real outage on every URL at once — into a tidy "page not found", and it would turn the intake handler's `Origin` refusal into a 404 page. Keeping 403 meaning 403 is worth one statement. ```bash aws s3api get-bucket-policy --bucket "$BUCKET" --query Policy --output text > /tmp/bucket-policy.before.json python3 -m json.tool /tmp/bucket-policy.before.json ``` **Expect:** one statement, `AllowCloudFrontServicePrincipal`, `s3:GetObject` on `arn:aws:s3:::$BUCKET/*`, conditioned on the distribution ARN. Keep that file — Part 9 restores from it. ```bash python3 - "$BUCKET" "$ACCT" "$DIST_ID" <<'PY' > /tmp/bucket-policy.after.json import json, sys bucket, acct, dist = sys.argv[1], sys.argv[2], sys.argv[3] p = json.load(open('/tmp/bucket-policy.before.json')) arn = f'arn:aws:cloudfront::{acct}:distribution/{dist}' sid = 'AllowCloudFrontListForHonest404s' p['Statement'] = [s for s in p['Statement'] if s.get('Sid') != sid] p['Statement'].append({ 'Sid': sid, 'Effect': 'Allow', 'Principal': {'Service': 'cloudfront.amazonaws.com'}, 'Action': 's3:ListBucket', 'Resource': f'arn:aws:s3:::{bucket}', 'Condition': {'StringEquals': {'AWS:SourceArn': arn}}, }) json.dump(p, sys.stdout, indent=2) PY python3 -m json.tool /tmp/bucket-policy.after.json aws s3api put-bucket-policy --bucket "$BUCKET" --policy "file:///tmp/bucket-policy.after.json" ``` **Verify — three things, and the third is the one worth doing:** ```bash aws s3api get-bucket-policy --bucket "$BUCKET" --query Policy --output text | python3 -m json.tool aws s3api get-public-access-block --bucket "$BUCKET" --query 'PublicAccessBlockConfiguration' curl -s -o /dev/null -w 'direct S3 object: %{http_code}\n' \ "https://${BUCKET}.s3.${AWS_REGION}.amazonaws.com/index.html" aws cloudfront get-distribution-config --id "$DIST_ID" \ --query 'DistributionConfig.DefaultCacheBehavior.{Cache:CachePolicyId,OriginRequest:OriginRequestPolicyId}' aws cloudfront get-cache-policy --id 658327ea-f89d-4fab-a63d-7e88639e58f6 \ --query 'CachePolicy.CachePolicyConfig.ParametersInCacheKeyAndForwardedToOrigin.QueryStringsConfig' ``` **Expect:** two statements; all four public-access blocks still `true`; direct S3 `403`; `Cache` = `658327ea-…` (`Managed-CachingOptimized`) with **`OriginRequest: null`**; and `QueryStringsConfig` = **`{"QueryStringBehavior": "none"}"`**. ⚠️ **THOSE LAST TWO ARE THE CHECK, AND THE OBVIOUS ONE IS WORTHLESS.** The tempting verification is `curl "${SITE}/?list-type=2&max-keys=5"` and confirming it returns HTML rather than an XML `ListBucketResult` — **it cannot return anything else, so it is evidence of nothing.** The distribution has `DefaultRootObject: index.html` and, after Part 2, the router rewrites `/` to `/index.html`; the origin request is therefore a `GetObject` on a key, never a request for the bucket root, so `list-type=2` could not be read as a list operation whether or not query strings reached S3. An operator running it sees HTML and ticks a control that never ran, which is the Q22 shape exactly. Found by `adversarial-reviewer`, 2026-09-01. **What actually makes the grant safe, and the two commands above assert both halves:** no viewer path resolves to the bucket root, and the default behaviour forwards **no query strings** to the origin — with no origin request policy attached to override that. If either assertion fails, revert this part (Part 9.1) and map 403 instead. --- ## Part 2 — The trailing-slash router function Source and test are in the repo: `infra/cloudfront/router.js` and `router.test.mjs`. Read the header comment before publishing it — the reason it must not be associated with `/api/*` is in there, and it is the difference between a working form and a POST silently converted to a GET. ```bash node infra/cloudfront/router.test.mjs ``` **Expect:** exit **0**. The case count is deliberately not quoted here — it has already gone 15 → 22 → 30 in one day, and an `Expect` line that never matches teaches the operator to stop reading `Expect` lines. The script asserts its own count internally (`case count != CASES.length`), so exit 0 is the whole check. ```bash aws cloudfront create-function \ --name adr-sml-router \ --function-config '{"Comment":"trailing-slash + index.html for the Astro directory build; docs/09","Runtime":"cloudfront-js-2.0"}' \ --function-code "fileb://infra/cloudfront/router.js" \ --query '{Name:FunctionSummary.FunctionMetadata.FunctionARN,Stage:FunctionSummary.FunctionMetadata.Stage,Status:FunctionSummary.Status}' ``` **Expect:** an ARN, `Stage: DEVELOPMENT`, `Status: UNPUBLISHED`. **Test it in the real runtime before publishing.** The local test proves the branches; this proves the runtime accepts the code, which the local test cannot. ```bash ETAG=$(aws cloudfront describe-function --name adr-sml-router --query ETag --output text) for URI in / /about/ /about /robots.txt /_astro/x.css /api/intake \ //evil.example.com/x '/\evil.example.com/x' //about/ //robots.txt; do printf '{"version":"1.0","request":{"method":"GET","uri":"%s","querystring":{},"headers":{},"cookies":{}}}' "$URI" \ > /tmp/cf-event.json printf '%-16s ' "$URI" aws cloudfront test-function --name adr-sml-router --if-match "$ETAG" --stage DEVELOPMENT \ --event-object fileb:///tmp/cf-event.json \ --query 'TestResult.{Out:FunctionOutput,Err:FunctionErrorMessage}' --output text done ``` `fileb://` rather than a base64 argument: it hands the CLI raw bytes and sidesteps the question of whether the local `base64` wraps its output, which would make the argument invalid in a way the error message would not explain. `test-function` does not change the function, so one `ETAG` read serves the whole loop. Then the query-string path, which none of the URIs above exercises: ```bash cat > /tmp/cf-event-qs.json <<'JSON' {"version":"1.0","request":{"method":"GET","uri":"/fees","headers":{},"cookies":{}, "querystring":{"utm":{"value":"a%20b"},"q":{"value":"x|y"},"bad":{"value":"z\"<>#&k=v"}}}} JSON aws cloudfront test-function --name adr-sml-router --if-match "$ETAG" --stage DEVELOPMENT \ --event-object fileb:///tmp/cf-event-qs.json \ --query 'TestResult.{Out:FunctionOutput,Err:FunctionErrorMessage}' --output text ``` **Expect,** in order for the ten URIs: `uri /index.html`; `uri /about/index.html`; a **301 to `/about/`**; `uri /robots.txt` unchanged; `uri /_astro/x.css` unchanged; `/api/intake` a **301 to `/api/intake/`**; and then **four 301s that all stay on this origin with a single leading slash** — `/evil.example.com/x/`, `/evil.example.com/x/`, `/about/`, `/robots.txt`. No `FunctionErrorMessage` on any of the ten. From the query-string call: a **301 to `/fees/?utm=a%20b&q=x|y&bad=z&k=v`**. Three things are being checked there and each has been wrong once: `%20` survives (the values arrive percent-encoded, so re-encoding would produce `%2520`); `|` survives (it is not in WHATWG's query percent-encode set, so browsers send it raw, and one revision of the function stripped it and silently corrupted campaign links); and `"`, `<`, `>` and `#` are gone (`#` is the one that changes the *structure* of the header — left in, `&k=v` lands in a fragment and the parameter is lost). ⚠️ **THE FIRST VERSION OF THIS LOOP PASSED `"querystring":{}` ON EVERY CASE AND TRIED NEITHER REDIRECT PATH.** So in the only environment that runs the real `cloudfront-js-2.0` runtime — the environment this step exists to reach — none of the query-string handling and none of the normalisation was executed, while the prose called it "the authoritative check". Found by `adversarial-reviewer` round 2. ⚠️ **That last row is the reason this function must not be associated with the `/api/*` behaviour, and it is in the list so you see it rather than read about it.** A 301 turns a POST into a GET, so an intake submission routed through this function would arrive at the handler as a GET with no body — and the visitor would be told nothing was wrong. Part 3 associates the function with the **default** behaviour only; `configure.mjs` sets `FunctionAssociations: { Quantity: 0 }` on the `/api/*` behaviour explicitly rather than by omission. ```bash aws cloudfront publish-function --name adr-sml-router --if-match "$ETAG" export ROUTER_ARN=$(aws cloudfront describe-function --name adr-sml-router --stage LIVE \ --query 'FunctionSummary.FunctionMetadata.FunctionARN' --output text) echo "ROUTER_ARN=$ROUTER_ARN" ``` **Expect:** a non-empty ARN. It is empty if `publish-function` failed — read the status, not the absence of an error. --- ## Part 3 — Apply the five distribution changes One script, `infra/cloudfront/configure.mjs`, because the alternative is hand-editing a 300-line JSON document and posting it back with an `IfMatch` ETag. It is **dry-run by default**, **idempotent**, and it resolves both managed policy ids **by name from the account** rather than carrying them as literals. ```bash node infra/cloudfront/configure.mjs --dist "$DIST_ID" --api-domain "$API_DOMAIN" \ --function-arn "$ROUTER_ARN" ``` ⚠️ **THE BLOCK BELOW IS THE `+` CHANGE LINES AND THE TWO RESOLVED-POLICY LINES. IT IS NOT THE WHOLE OUTPUT, AND IT SAID "exactly" UNTIL 2026-09-04.** Against the **live** distribution the dry run is **59 lines**, and all 59 account for: 2 resolved-policy lines, **10** `·` lines, **3** `+` lines, a **40-line JSON dump** of the `*.pdf` behaviour it would add, 2 blank lines, the `N change(s) to distribution …` header and the `DRY RUN — nothing was sent.` footer `[measured 2026-09-04, exit 0, nothing written]`. 🛑 **READ THE EXIT STATUS, AND IT HAS THREE VALUES.** `0` — everything this script manages was applied or is already present. `2` — a usage error, before any AWS call. **`3` — sections that could run did, and something was SKIPPED: read the `⚠ … SKIPPED, not changed` block.** Anything else is a throw. `3` exists because a skip used to exit `0`, and this document uses `exit 0` as its own success stamp throughout — so a partial run read as a complete one. A skip is never nothing: it means the `*.pdf` behaviour has no `X-Robots-Tag`, or `/api/*` is not forwarding `CloudFront-Viewer-Address`, and the skip line says which. ⚠️ **THE SHAPE CHANGES WITH THE STATE, SO READ THE `+` LINES AND NOT THE TOTAL.** Each item Parts 1–3 have already applied prints `·` when it is found and `+` when it is staged, so in the Part 0.3 state four lines cross from one column to the other and the totals move with them. **The `+` lines are the check.** **Expect** — this is the dry run, and on a distribution in the state Part 0.3 records the change lines are: ``` resolved Managed-CachingDisabled = 4135ea2d-6df8-44a3-9df3-4b5a84be39ad resolved Managed-AllViewerExceptHostHeader = b689b0a8-53d0-40ab-baf2-68738e2966ac 8 change(s) to distribution E1OK7G98KNKUTA (ETag …): + DefaultCacheBehavior.FunctionAssociations viewer-request -> arn:…:function/adr-sml-router + CustomErrorResponses += 404 -> /404.html with status 404 + Origins += intake-api -> …execute-api… (https-only, TLSv1.2) + CacheBehaviors += /api/* -> intake-api, CachingDisabled, AllViewerExceptHostHeader, POST allowed + create response-headers policy adr-sml-pdf-noindex (SecurityHeadersConfig cloned from … + X-Robots-Tag: noindex) + CacheBehaviors += *.pdf -> , default cache policy, adr-sml-pdf-noindex (policy id created in the same --apply pass) + create origin request policy adr-sml-api-viewer-address (whitelist: CloudFront-Viewer-Address, Content-Type, Origin, Referer, User-Agent; cookies all; query strings all) + /api/* OriginRequestPolicyId b689b0a8-… -> adr-sml-api-viewer-address DRY RUN — nothing was sent. Re-run with --apply to write it. ``` ⚠️ **SECTION 4 CANNOT SHOW THE POLICY ID IN A DRY RUN, AND SAYS SO — IT IS STILL ONE `--apply`.** The `*.pdf` behaviour has to carry the response-headers policy's id, and on a first run that policy does not exist yet, so the dry run prints the behaviour it *would* add with `(policy id created in the same --apply pass)` where the id goes. **A single `--apply` creates the policy and adds the behaviour in one call — do not run it twice.** The dry run reports both changes either way; one that listed only the policy would hide the half that touches a distribution serving 23 pages. Fewer than eight changes means part of this is already done — read which lines are prefixed `·` — but READ THE WORDS, not the bullet: `configure.mjs` uses `·` for *already present* **and** for *would CREATE / would SET / would ADD*, so the prefix alone does not say whether a line is done or still pending. **On the live distribution as at 2026-09-04, after the second `--apply` attempt, the dry run returns exactly three `+` lines — **one** for section 4 (the behaviour; its policy already exists, see the incident below) and the two for section 5** — `[measured 2026-09-04, dry run against `E1OK7G98KNKUTA`, ETag `E2EUQ1WTGCTBG2`, exit 0, nothing written]`. More than eight, or a different set, means the distribution is not in the state 0.3 recorded: stop and re-read it. 🛑 **INCIDENT — TWO `--apply` ATTEMPTS FAILED ON 2026-09-04, FOR TWO DIFFERENT REASONS, AND THE SECOND ONE LEFT A POLICY BEHIND.** The distribution is unchanged after both. Read both before the next attempt. **ATTEMPT 1 — nothing reached the distribution and nothing was created.** Section 4's clone was sent to `create-response-headers-policy` verbatim and the AWS CLI rejected it **client-side**, before the call left the machine: ``` An error occurred (ParamValidation): Parameter validation failed: Missing required parameter in ResponseHeadersPolicyConfig.SecurityHeadersConfig.ContentSecurityPolicy: "Override" Missing required parameter in ResponseHeadersPolicyConfig.SecurityHeadersConfig.ContentSecurityPolicy: "ContentSecurityPolicy" ``` **The cause, and it generalises past this script: a config AWS hands back is not necessarily a config AWS will accept.** `get-response-headers-policy` on `Managed-SecurityHeadersPolicy` returns `"ContentSecurityPolicy": {}` — an empty object standing for a member the policy does not define — and `ResponseHeadersPolicySecurityHeadersConfig` has **no required members** while **every one of its six members requires at least `Override`**. So an empty member is always "undefined here" and is **never** a legal input. ⚠️ **AND THE SAME IS TRUE ONE LEVEL UP, WHICH THE FIRST FIX MISSED.** Every sibling member of `ResponseHeadersPolicyConfig` also declares required fields — `CorsConfig` five of them, `RemoveHeadersConfig` and `CustomHeadersConfig` a `Quantity`, `ServerTimingHeadersConfig` an `Enabled` — while the container itself requires only `Name`. So `{}` is a placeholder at **both** levels, and a fix covering only the inner one turns the outer placeholder into a hard abort instead of an omission. All of that is read out of the CLI's own service model, not inferred from the symptom. **State after the failure** ``[verified 2026-09-04 — `get-distribution-config`, `list-response-headers-policies --type custom`, `list-origin-request-policies --type custom`]``**:** `/api/*` still on `b689b0a8-53d0-40ab-baf2-68738e2966ac`, **no** `*.pdf` behaviour, **zero** custom response-headers policies, **zero** custom origin request policies. ⚠️ **THE "ONE REVIEWABLE TRANSACTION" PROPERTY IS ABOUT THE DISTRIBUTION, NOT THE ACCOUNT — AND THIS RECORD ASSERTED THE WIDER VERSION FOR ONE ROUND.** `update-distribution` is the script's last call, so a throw above it does leave the **distribution** untouched. But sections 4 and 5 each make their own write first — `create-response-headers-policy` and `create-origin-request-policy` — and the script's own comment on section 5's drift throw documents a reachable path where section 4 has **already created `adr-sml-pdf-noindex`** when section 5 aborts. **So after any failed `--apply`, check for an orphaned policy as well as for a changed distribution**, with both of these: ```bash aws cloudfront list-response-headers-policies --type custom --output json \ --query 'ResponseHeadersPolicyList.Quantity' aws cloudfront list-origin-request-policies --type custom --output json \ --query 'OriginRequestPolicyList.Quantity' ``` An orphan is harmless and self-healing — the next run finds it by name, matches it and attaches it — **so do not delete it by hand.** ⚠️ **THE EXPECTED VALUES DIFFER BY WHICH FAILURE YOU ARE RECOVERING FROM.** After **attempt 1** both returned `0` and nothing needed doing. After **attempt 2** they return **`1` and `0`** — the response-headers policy is the orphan recorded below, and `1` is the correct reading, not a second problem `[verified 2026-09-04]`. **Two changes came out of it.** The clone now **omits** any empty member at either of those two levels — a `ResponseHeadersPolicyConfig` member, or a `SecurityHeadersConfig` member — and the dry run **asserts** that the generated config carries no empty object at any *other* level, naming the dotted path if it does. The asymmetry is deliberate: those two levels are where AWS is known to synthesise a placeholder, and anywhere else is unaccounted for and stops the run rather than being discarded quietly. The assertion runs before every branch, so **the dry run now catches this class** rather than an `--apply` discovering it — and if it ever does fire it **skips section 4** rather than throwing, so `router.js` can still be re-applied. **The proof is a command rather than a session**, which is the point of `infra/cloudfront/policy-shapes.mjs` existing as its own module — `configure.mjs` reads argv and calls AWS at import time, so the two functions could not otherwise be reached: ```bash node infra/cloudfront/policy-shapes.test.mjs ``` **Expect** `policy-shapes: 53 of 53 cases pass`, exit 0. Its first case is this incident verbatim — the live `SecurityHeadersConfig`, empty `ContentSecurityPolicy` and all. The same suite covers attempt 2's limit checks, so this is the only command in this Part that proves both. --- **ATTEMPT 2 — section 4 SUCCEEDED, section 5 FAILED, and the run left an orphan.** With the clone fixed, `create-response-headers-policy` created **`adr-sml-pdf-noindex` = `51c4e79b-d9c6-4c6f-907c-dbb0e73dd374`**. Section 5 then failed: ``` An error occurred (InvalidArgument) when calling the CreateOriginRequestPolicy operation: The parameter Comment is too big ``` Its `Comment` was **182 characters** against a **128** cap. `update-distribution` never ran, so the distribution is untouched — **but the account now holds a response-headers policy that no behaviour references.** 🛑 **DO NOT DELETE THAT POLICY BY HAND.** This is the orphan case Part 3 and §7 predicted before it happened, and the recovery is **measured, not asserted** `[measured 2026-09-04, dry run, exit 0, nothing written]`: the next run finds it **by name**, matches it on every reconciled field, and stages it for attachment — ``` · response-headers policy adr-sml-pdf-noindex exists and matches the default behaviour + CacheBehaviors += *.pdf -> …, adr-sml-pdf-noindex (51c4e79b-d9c6-4c6f-907c-dbb0e73dd374) ``` — with the **create line gone** and the change count down from 4 to 3. **No duplicate and no name collision.** (A collision would not be silent either: a duplicate name returns `ResponseHeadersPolicyAlreadyExists`, which is a different error from the `InvalidArgument` above. The script never reaches it, because it looks the policy up by name first.) ⚠️ **`Comment` IS NOT RECONCILED**, so `51c4e79b` keeps its original 118-character text while the script now carries a shorter one. Deliberate: adding `Comment` to the drift check would throw on this very policy and block the run that attaches it. **WHY NOTHING CAUGHT IT LOCALLY, AND THIS IS THE GENERAL LESSON.** Measured 2026-09-04: **`botocore/validate.py` checks neither `max` nor `pattern`** — its `range_check()` reads only `min`, and the word `pattern` does not appear in the file. And the 128 cap is not modelled as a constraint at all: on both policy configs `Comment` is a bare `string`, and the number lives in the shape's **`documentation` prose**. So **no client-side validation of any kind stood between that 182-character string and the CloudFront API**, which is exactly why the pre-flight below had to be built rather than relied upon. **BOTH POLICY COMMENTS ARE NOW UNDER 80 CHARACTERS**, and the dry run enforces the limits it knows about — `infra/cloudfront/policy-shapes.mjs`, `PAYLOAD_LIMITS`, one entry per limit with the source it came from. **Every entry with a cited AWS source is enforced by the service and by nothing local** — see below. Two entries are stamped `[assumed]` and are not: no AWS source states a policy **name** length, and those two constrain nothing this script sends (our names are 19 and 26 characters). A breach **skips its section** rather than throwing, and — the part that matters — **skips the whole section**, so a policy that is not created is never staged for attachment. The suite named earlier in this Part (`node infra/cloudfront/policy-shapes.test.mjs`) covers both the 182-character `Comment` that failed here and the 118-character one that did not. --- **One of the `·` lines carries a number worth reading**, and it is not a warning: ``` · cloning 5 defined security header(s); omitting 1 the source does not define (ContentSecurityPolicy) ``` **Five is the number to read.** It is the count of security headers the PDF policy will carry, and the verification block at the end of this Part counts the same five arriving on the live PDF. A drop in this number is a partial clone announcing itself one step earlier than that `curl` would. ⚠️ **RUN IT WITHOUT `--function-arn` ONLY IF THE ROUTER IS ALREADY ATTACHED.** Omitting the flag prints `· no --function-arn given, leaving FunctionAssociations alone` and skips change 1 — which is right on a re-run and wrong on a first one, and the two look identical in a count. 🛑 **SECTION 5 IS THE ONLY ONE THAT REPLACES SOMETHING, AND WHAT IT REPLACES IS ON THE INTAKE FORM'S PATH.** Sections 1–4 add. Section 5 swaps the origin request policy on `/api/*` from `Managed-AllViewerExceptHostHeader` to a **whitelist** of five headers, because AWS has no behaviour meaning "all viewer headers except Host, plus a CloudFront header" — `allExcept` can only subtract, and `allViewerAndWhitelistCloudFront` drags `Host` along and 403s at API Gateway. Whitelisting is therefore forced, and the cost is that **a header missing from that list is a header the handler never sees.** The list is the handler's four `headerOf` reads plus `CloudFront-Viewer-Address`. The check prints the names, so they can be compared to the whitelist rather than counted: ```bash grep -o "headerOf(event, '[a-z-]*'" backend/intake/handler.mjs \ | sed "s/.*'\(.*\)'/\1/" | sort ``` **Expect** exactly `content-type`, `origin`, `referer`, `user-agent`. ⚠️ **`grep -n "headerOf(event"` WAS PRESCRIBED HERE AND RETURNS FIVE** — it matches `function headerOf(event, name)`, the definition itself — so an operator comparing it against a documented "four" concludes the handler grew a read. **The failure mode is not an error.** Every submission would validate short and redirect to `/contact/could-not-send/` — a real inquirer would read it as their own browser misbehaving, and nothing would appear in a log as a fault. So the block below is **not optional after an `--apply` that includes change 8**, and there are **three** of them. The first is Part 7.1's probe with its output read differently — **not "unchanged", which this said for one round**: §7.1 pipes into `head -12` and reads the status by eye, while these read curl's own exit status and count the `location` separately. **Run all three, in this order, and each answers a different question:** | # | probe | what only it can tell you | |---|---|---| | 1 | `Origin` + body | `Origin` is still forwarded — a **403** means it is not | | 2 | `Referer`, no `Origin` | the Firefox fallback still works — nothing else tests it | | 3 | honeypot value | the **body parsed** — probes 1 and 2 return the same 303 whether it did or not | **PROBE 1 — is `Origin` still forwarded?** ```bash curl -si -X POST "$SITE/api/intake" \ -H 'Origin: https://adr.smlcompany.ca' \ -H 'Content-Type: application/x-www-form-urlencoded' \ --data 'probe=1' -o /tmp/api.h echo "curl_exit=$?" # curl's OWN status, on its own line head -1 /tmp/api.h grep -ic '^location: .*could-not-send' /tmp/api.h ``` **Expect** `curl_exit=0`, `HTTP/2 303`, and `1`. A **403** here means the `Origin` header is no longer reaching the handler — i.e. the whitelist dropped it — and the form is broken for everyone. **PROBE 2 — the `Referer` fallback, which nothing else tests.** The handler accepts `Referer` when `Origin` is absent (Firefox omits `Origin` on some same-origin form navigations), so a whitelist that forwarded `Origin` and dropped `Referer` passes probe 1 and fails for exactly those users: ```bash curl -si -X POST "$SITE/api/intake" \ -H 'Referer: https://adr.smlcompany.ca/contact/' \ -H 'Content-Type: application/x-www-form-urlencoded' \ --data 'company_website=probe' -o /tmp/api3.h echo "curl_exit=$?" head -1 /tmp/api3.h grep -ic '^location: .*contact/received' /tmp/api3.h ``` **Expect** `curl_exit=0`, `303` and `1` `[verified against production 2026-09-04 — it returns 303 today, on the managed policy]`. A **403** means `Referer` is not being forwarded. 🛑 **PROBE 3, AND NEITHER OF THE FIRST TWO CAN REPLACE IT: THEY CANNOT FAIL IN THE INTERESTING DIRECTION.** `303 → could-not-send` is what the handler returns **both** when it parsed the body and found an empty submission **and** when `parseBody` threw because `Content-Type` never arrived. Two opposite outcomes, one status, one location — so a dropped `Content-Type` reads as a pass. This probe separates them, and **writes nothing and sends nothing**: ```bash curl -si -X POST "$SITE/api/intake" \ -H 'Origin: https://adr.smlcompany.ca' \ -H 'Content-Type: application/x-www-form-urlencoded' \ --data 'company_website=probe' -o /tmp/api2.h echo "curl_exit=$?" head -1 /tmp/api2.h grep -ic '^location: .*contact/received' /tmp/api2.h ``` **Expect** `curl_exit=0`, `HTTP/2 303`, and `1` — location `/contact/received/`, **not** `could-not-send`. That is the honeypot branch: it is reached **only if the body parsed**, and it returns before validation, before any DynamoDB write and before any SES send, so it leaves no record and sends no email. `could-not-send` here means the body did not parse — `Content-Type` is missing from the whitelist. **Roll back.** ⚠️ **IT DEPENDS ON THE HONEYPOT'S NAME** (`company_website`, `fields.mjs`). If that is ever renamed, this probe degrades to the `could-not-send` branch — which reads as a failure and starts an investigation, not as a pass. That direction is the safe one; keep it that way if you change the probe. **ROLLBACK, and it is one field.** Do not debug a broken intake form in place: ```bash # ⚠️ THIS RETURNS THE ID THE BEHAVIOUR HAS NOW — which, if change 8 applied, is # the whitelist you are rolling back FROM, not the value to restore. The value to # restore is the managed id on the line below. Run this to confirm which state # you are in, then PUT the managed id back with # update-distribution --if-match. ⚠️ NOT by re-running configure.mjs: section 5 # converges FORWARD and cannot tell a deliberate revert from a first run — the # two are byte-identical in the config — so --apply would re-attach the # whitelist and put the form back in the state you are rolling back from. aws cloudfront get-distribution-config --id "$DIST_ID" \ --query 'DistributionConfig.CacheBehaviors.Items[?PathPattern==`/api/*`].OriginRequestPolicyId' # Managed-AllViewerExceptHostHeader = b689b0a8-53d0-40ab-baf2-68738e2966ac ``` Set that behaviour's `OriginRequestPolicyId` back to `b689b0a8-53d0-40ab-baf2-68738e2966ac` and `update-distribution` with the current ETag. `configure.mjs` prints the same id on the line it changes, prefixed `↩`, at the moment it changes it. ⚠️ **THE HANDLER STILL STORES THE EDGE ADDRESS AFTER THIS.** Forwarding the header does not change what is recorded, and it must not be made to as a follow-up edit: what the record holds is published field by field on `/legal/privacy/`, so storing `CloudFront-Viewer-Address` is a **disclosure** change governed by §7.2's decision table, not a code tidy. Pouya's ruling of 2026-09-04 is *measured, not yet acted on*. ⚠️ **AND `adr-sml-pdf-noindex` IS RECONCILED ON EVERY RUN, NOT ONLY CREATED.** A response-headers policy **replaces** rather than merges, so the PDF policy has to carry everything the default behaviour's policy carries. If they have diverged — someone adds the `Content-Security-Policy` or `Permissions-Policy` that `docs/05` specifies to one and not the other — the script **throws and names the diff** rather than passing. That is deliberate: the failure it guards is the PDF being served different headers from the pages, which is silent. ```bash node infra/cloudfront/configure.mjs --dist "$DIST_ID" --api-domain "$API_DOMAIN" \ --function-arn "$ROUTER_ARN" --apply aws cloudfront wait distribution-deployed --id "$DIST_ID" echo "deployed: $?" ``` **Verify the config landed:** ```bash aws cloudfront get-distribution-config --id "$DIST_ID" \ --query 'DistributionConfig.{Fn:DefaultCacheBehavior.FunctionAssociations.Items[].EventType,Err:CustomErrorResponses.Items[].{Code:ErrorCode,Page:ResponsePagePath,Status:ResponseCode},Beh:CacheBehaviors.Items[].{P:PathPattern,O:TargetOriginId,RHP:ResponseHeadersPolicyId,Fn2:FunctionAssociations.Items[].EventType,Methods:AllowedMethods.Items},Origins:Origins.Items[].Id}' ``` **Expect:** `Fn: ["viewer-request"]`; one error response `404 → /404.html → 404`; **two** behaviours — `/api/*` → `intake-api`, POST in its method list, **no `RHP` and `Fn2: null`** (the association is withheld there deliberately: a 301 would turn the form's POST into a GET and drop the body), and `*.pdf` → the S3 origin **with an `RHP` id and `Fn2: ["viewer-request"]`**; two origins. ⚠️ **THAT QUERY DOES NOT PROJECT `OriginRequestPolicyId`, SO IT CANNOT SEE CHANGE 8.** Read it separately rather than concluding anything from its absence: ```bash aws cloudfront get-distribution-config --id "$DIST_ID" \ --query 'DistributionConfig.CacheBehaviors.Items[].{P:PathPattern,ORP:OriginRequestPolicyId}' ``` **Expect** `/api/*` carrying the **`adr-sml-api-viewer-address`** id — *not* `b689b0a8-53d0-40ab-baf2-68738e2966ac`, which is the managed policy it replaced and is what a rollback restores. **Then verify the header actually arrives, because the config landing is not the same fact:** ```bash curl -D /tmp/pdf.h -o /dev/null "$SITE/pouya-lajevardi-bio.pdf" echo "curl_exit=$?" # curl's OWN status, on its own line for h in x-robots-tag strict-transport-security x-content-type-options \ x-frame-options x-xss-protection referrer-policy; do printf '%-28s %s\n' "$h" "$(grep -ic "^$h:" /tmp/pdf.h)" done ``` **Expect** `curl_exit=0` and **`1` against every one of the six** — the five security headers *and* `x-robots-tag`. ⚠️ **THE SHAPE OF THIS BLOCK IS THE POINT, and its first version got all three wrong.** It piped `curl -sI` into one `grep -E` with six alternatives and read `$?`. That reports **grep's** status, not curl's, so a DNS failure, a TLS failure and a 5xx all read as `exit=1` — indistinguishable from "the headers are missing", with `-s` deleting the message that would have told them apart. And an alternation exits **0 if ANY ONE** matches, so `exit=0` would not have meant the five arrived, which is the only regression the block exists to catch. Counting each header separately is what makes a partial clone visible. (`CLAUDE.md`: never suppress stderr, never read a pipeline's status as its first command's, and a uniform pass is the result that ends a check rather than starting one.) ⚠️ **If any of the five security headers reads `0`, the policy did not clone them and the PDF has LOST headers it had before this change.** --- ## Part 4 — Verify the distribution before the site exists Run this now, against the **old** bucket contents. It is the check that the plumbing works independently of the deploy. ```bash for p in / /index.html /about/ /nope-not-a-page/; do printf '%-22s ' "$p" curl -s -o /dev/null -w 'status=%{http_code} ct=%{content_type}\n' "${SITE}${p}" done printf 'no-slash redirect: '; curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' "${SITE}/about" ``` **Expect:** `/` and `/index.html` → **200 text/html** (the old page still serves — the router did not break it); `/about/` and `/nope-not-a-page/` → **404** rather than the 403 they returned before, because Part 1 changed S3's answer and Part 3 mapped it; `/about` → **301 → `https://adr.smlcompany.ca/about/`**. The body of those 404s is not the 404 page yet — `404.html` is not in the bucket until Part 8. **The status is what Part 4 proves; the page is what Part 8.4 proves.** Do not tick `docs/06`'s "404 returns a 404 status" item here; tick it after 8.4, when both halves are true at once. --- ## Part 5 — Deploy the handler ### 5.1 Package 🛑 **THREE FILES SINCE 2026-09-04, AND THE ZIP FOLLOWS NO IMPORT.** `handler.mjs` imports both `./fields.mjs` and `./spam-score.mjs`; a zip missing either fails at cold start with `Runtime.ImportModuleError` and every submission then 500s. **The list is now derived from the directory** — `ls *.mjs` minus the tests — in this step and in 5.5, so a new module is packaged without editing anything. It was typed out in both until 2026-09-04, and this banner still said so, fifteen lines above the paragraph that says otherwise. ```bash rm -f /tmp/intake.zip (cd backend/intake \ && echo "packaging: $(ls *.mjs | grep -v '\.test\.' | tr '\n' ' ')" \ && zip -q -X /tmp/intake.zip $(ls *.mjs | grep -v '\.test\.')) unzip -l /tmp/intake.zip ``` ⚠️ **THE LIST IS SUBSTITUTED DIRECTLY, NOT HELD IN A VARIABLE, AND THAT IS NOT STYLE.** A first version read `MODULES=$(ls …)` then `zip … $MODULES`. **In zsh that packages ONE file whose name is all three joined by newlines** — zsh does not word-split parameter expansions, only command substitutions — so it fails on the shell this project is actually operated from while working in bash. `CLAUDE.md` names this trap; it was reintroduced here and caught by running the block in both shells rather than by reading it. **Expect:** exactly three entries — `handler.mjs`, `fields.mjs`, `spam-score.mjs` — **42,604 bytes uncompressed and 18,462 zipped** `[measured 2026-09-04]`. All three at the zip root: the imports are `./`-relative, so a nested directory breaks them at cold start. *(This read "two entries, ≈ 25.7 KB / ≈ 10.8 KB" `[measured 2026-09-01]`, before the scorer existed.)* ⚠️ **`spam-score.test.mjs` IS NOT IN THE ZIP AND MUST NOT BE.** Run it at a keyboard — `node backend/intake/spam-score.test.mjs`, **39 of 39** — before packaging. It is the only check on the scorer, whose failure mode is labelling real inquiries rather than throwing. ### 5.2 Configuration first, code second The handler calls `requireEnv()` at **module scope** and throws at cold start on any of six missing variables — deliberately, so a misconfigured function cannot accept a submission it will not store. Set the configuration before the code, so there is never a moment when the new code runs against no environment. ```bash ENV_JSON=$(node scripts/intake-env.mjs --table "$TABLE" --notify "$NOTIFY_TO" --from "$MAIL_FROM") echo "$ENV_JSON" | python3 -m json.tool ``` **Expect** six variables. ⚠️ **`RESPONSE_TIME` and `NO_RETAINER_NOTICE` are read out of `src/data/site.ts` and must never be typed here.** Both are published commitments — the two-business-day response (§4, Q27) and the no-retainer notice `docs/01` §`/contact/` requires, **including its fourth clause about not itself creating a conflict check**, which a hand-typed copy inside the handler had already dropped once. The notice also contains an en dash in "mediator–party", which is exactly the character a retype loses. `scripts/intake-env.mjs` asserts both and exits non-zero rather than emitting a softened commitment. ```bash aws lambda update-function-configuration --function-name "$FN" \ --handler handler.handler --timeout 15 --memory-size 512 --environment "$ENV_JSON" aws lambda wait function-updated --function-name "$FN" aws lambda get-function-configuration --function-name "$FN" \ --query '{Handler:Handler,Timeout:Timeout,Memory:MemorySize,Vars:sort(keys(Environment.Variables)),State:State,Update:LastUpdateStatus}' ``` **Expect:** `Handler: handler.handler` — the file is `handler.mjs` and the export is `handler`, where the hand-built function was `index.handler`; `Timeout: 15`; `Memory: 512`; the six names sorted; `State: Active`; `Update: Successful`. Timeout 10 → 15 s and memory 128 → 512 MB are changes, not restatements. The handler does a DynamoDB write and two SES sends per invocation, and 128 MB is where an AWS SDK cold start is slowest — more memory shortens the billed duration enough that it is usually the cheaper setting, not the dearer one. ### 5.3 Code ```bash aws lambda update-function-code --function-name "$FN" --zip-file fileb:///tmp/intake.zip aws lambda wait function-updated --function-name "$FN" aws lambda get-function-configuration --function-name "$FN" \ --query '{CodeSize:CodeSize,Runtime:Runtime,Update:LastUpdateStatus,Modified:LastModified}' ``` **Expect** `Update: Successful`, and a `CodeSize` that says **which path you took** — it is the ZIP, not the source: | path | expected `CodeSize` | |---|---| | 5.1's plain three-file zip | **≈ 18,462** | | 5.5's bundled variant | **low single-digit MB** — it was **3,307,021** on 2026-09-02 `[measured 2026-09-04 — get-function-configuration]` | 🛑 **5.5 IS THE PATH THAT WAS ACTUALLY TAKEN AT CUTOVER.** The live function carries the bundled zip, so **a redeploy that runs 5.1 and stops would replace it with an unbundled one and reintroduce the `Runtime.ImportModuleError` 5.5 exists to fix.** Run 5.4 after 5.3, every time, and follow it to 5.5 if it fires. *(This line said "≈ 10,800", and before that "around 23,000" — 5.1's uncompressed figure applied to a different quantity. Both were written against the unbundled path, which is not the one in production.)* ### 5.4 Prove it loads, without writing anything ```bash cat > /tmp/probe-no-origin.json <<'JSON' {"version":"2.0","routeKey":"POST /api/intake","rawPath":"/api/intake", "headers":{"content-type":"application/x-www-form-urlencoded"}, "requestContext":{"http":{"method":"POST","path":"/api/intake","sourceIp":"127.0.0.1"}}, "body":"probe=1","isBase64Encoded":false} JSON aws lambda invoke --function-name "$FN" --cli-binary-format raw-in-base64-out \ --payload file:///tmp/probe-no-origin.json /tmp/probe-out.json cat /tmp/probe-out.json; echo ``` **Expect:** the invoke prints `"StatusCode": 200` with **no `FunctionError`**, and `/tmp/probe-out.json` is ```json {"statusCode":403,"headers":{"Cache-Control":"no-store"},"body":""} ``` That 403 is the handler's `Origin` check refusing a request with neither `Origin` nor `Referer`. It happens **before** any DynamoDB write and before any email, which is what makes this probe safe to run against production. **Two other outcomes, and each has one cause:** - `"FunctionError": "Unhandled"` with `Runtime.ImportModuleError` in the body — the `nodejs24.x` runtime does not provide the AWS SDK v3 clients this handler imports. Run 5.5, then repeat 5.4. - A body naming `intake handler: is not set` — that variable did not reach the function. Re-run 5.2 and read the `Vars` list. ### 5.5 Bundled variant — only if 5.4 said `Runtime.ImportModuleError` Versions are resolved from the registry at run time rather than pinned in this file: `CLAUDE.md`'s rule is that a version is checked against the registry and never recalled, and a literal here would be stale the week after it was written. ⚠️ **THIS PATH WAS TAKEN — 2026-09-02, and the live function is the bundled zip.** ⚠️ **THE TWO RESOLVED VERSIONS AND THE COMMIT THE DEPLOYED SOURCES MATCH ARE IN `AGENTS.md` §7 AND ARE DELIBERATELY NOT REPEATED HERE.** They were repeated here until 2026-09-04, and the paragraph directly above is the argument against it — **the literals went stale in two days rather than a week**: the 2026-09-04 redeploy moved both packages one patch and added a third source file, and this copy still named the old versions and a superseded commit while reading as a measurement. Read §7's two Lambda rows; this step's duty is to **update** them, not to mirror them. ✅ **THE `cp` AND `zip` LINES BELOW DERIVE THE FILE LIST THE SAME WAY 5.1 DOES.** They were a second hand-typed copy until 2026-09-04, not derived from 5.1's and with nothing checking that the two agreed — so a module added to one and not the other would ship from whichever path the operator happened to take. Both now read the directory. ```bash rm -rf /tmp/intake-bundle && mkdir -p /tmp/intake-bundle echo "bundling: $(cd backend/intake && ls *.mjs | grep -v '\.test\.' | tr '\n' ' ')" (cd backend/intake && cp $(ls *.mjs | grep -v '\.test\.') /tmp/intake-bundle/) # ⚠️ ASSERT THE COPY LANDED. A glob that matches nothing makes `cp` fail, `zip` # succeed on an empty set, and `update-function-code` upload a bundle with no # handler — a silent failure that only shows up as 5xx on the live form. test -f /tmp/intake-bundle/handler.mjs || { echo "FATAL: handler.mjs not copied"; exit 1; } echo "copied: $(ls /tmp/intake-bundle/*.mjs | wc -l | tr -d ' ') module(s)" ( cd /tmp/intake-bundle \ && npm init -y > /dev/null \ && npm install --omit=dev --no-audit --no-fund \ "@aws-sdk/client-dynamodb@$(npm view @aws-sdk/client-dynamodb version)" \ "@aws-sdk/client-sesv2@$(npm view @aws-sdk/client-sesv2 version)" ) rm -f /tmp/intake.zip ( cd /tmp/intake-bundle && zip -qr -X /tmp/intake.zip $(ls *.mjs) node_modules package.json ) unzip -l /tmp/intake.zip | tail -1 aws lambda update-function-code --function-name "$FN" --zip-file fileb:///tmp/intake.zip aws lambda wait function-updated --function-name "$FN" ``` **Expect** a zip in the low single-digit megabytes, well under the 50 MB direct- upload limit. **If this path is taken, say so in the `AGENTS.md` Change Log and add both packages to §7** — they become pins this project maintains, and `R11` covers them from that moment. --- ## Part 6 — Route the API at `/api/intake` ### 6.1 The invoke permission, before the route ⚠️ **THE EXISTING PERMISSION DOES NOT COVER THE NEW ROUTE, AND THIS IS THE STEP WHOSE OMISSION IS HARDEST TO DIAGNOSE.** Part 0.3 records the one statement on the function: `SourceArn` `…/4tl0m5igkj/*/*/submissions`. Add a route at `/api/intake` without adding a permission for it and API Gateway is refused permission to invoke — the caller gets a **500**, the Lambda logs nothing at all because it was never entered, and the only trace is an API Gateway metric. ```bash aws lambda add-permission --function-name "$FN" \ --statement-id apigw-post-api-intake \ --action lambda:InvokeFunction \ --principal apigateway.amazonaws.com \ --source-arn "arn:aws:execute-api:${AWS_REGION}:${ACCT}:${API_ID}/*/POST/api/intake" aws lambda get-policy --function-name "$FN" --query Policy --output text | python3 -m json.tool ``` **Expect:** two statements — the old one scoped to `…/submissions` and the new one to `…/POST/api/intake`. ### 6.2 The route ```bash export INTEG_ID=$(aws apigatewayv2 get-integrations --api-id "$API_ID" \ --query "Items[?IntegrationUri=='arn:aws:lambda:${AWS_REGION}:${ACCT}:function:${FN}'].IntegrationId | [0]" \ --output text) echo "INTEG_ID=$INTEG_ID" [ -n "$INTEG_ID" ] && [ "$INTEG_ID" != "None" ] || echo "STOP — no integration points at $FN" aws apigatewayv2 create-route --api-id "$API_ID" \ --route-key 'POST /api/intake' --target "integrations/${INTEG_ID}" \ --query '{RouteId:RouteId,RouteKey:RouteKey,Target:Target}' aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].{Key:RouteKey,Target:Target}' ``` **Expect:** `INTEG_ID` non-empty and not `None`; then two routes, `POST /submissions` and `POST /api/intake`, both on the same integration. The stage is `$default` with auto-deploy on, so the route is live immediately — there is no deployment to create. ### 6.3 Throttling ⚠️ **THIS IS NOT THE PER-IP LIMIT `docs/05` ASKS FOR, AND IT CANNOT BE.** That spec says "Rate limit by source IP at API Gateway: 5 requests / 5 minutes". **API Gateway throttling is aggregate — per route and per stage — not per source IP.** Per-IP rate limiting needs AWS WAF with a rate-based rule on the distribution, which is a paid service and therefore Pouya's decision, recorded on `docs/06`'s checklist as explicitly **not** a launch blocker. What is set here is a real control over total volume; describing it as per-IP would be the Q22 shape again — a documented control that does not exist. ```bash aws apigatewayv2 update-stage --api-id "$API_ID" --stage-name '$default' \ --route-settings '{"POST /api/intake":{"ThrottlingRateLimit":1,"ThrottlingBurstLimit":5,"DetailedMetricsEnabled":true}}' aws apigatewayv2 get-stage --api-id "$API_ID" --stage-name '$default' \ --query 'RouteSettings' ``` **Expect:** the `POST /api/intake` entry with rate 1/s, burst 5, detailed metrics on. One per second sustained with a burst of five is far above any human filling this form and far below anything worth paying for. ### 6.4 Retire the old route Nothing posts to `/submissions` any more — the form posts to `/api/intake` and the handler that answered the old contract is gone. An unused public route on the same function is surface with no purpose. ```bash export OLD_ROUTE=$(aws apigatewayv2 get-routes --api-id "$API_ID" \ --query "Items[?RouteKey=='POST /submissions'].RouteId | [0]" --output text) echo "OLD_ROUTE=$OLD_ROUTE" aws apigatewayv2 delete-route --api-id "$API_ID" --route-id "$OLD_ROUTE" aws lambda remove-permission --function-name "$FN" \ --statement-id 70aeb597-e4d8-5141-8fb1-6b6190f7b2ab aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].RouteKey' aws lambda get-policy --function-name "$FN" --query Policy --output text | python3 -m json.tool ``` **Expect:** one route, `POST /api/intake`; one permission statement, scoped to it. Confirm the statement id in the `remove-permission` call against the policy you printed in 6.1 before running it — it is an id, not a name, and it will differ if the permission was ever rebuilt. > **The `execute-api` hostname stays reachable**, so `/api/intake` can be reached > without passing through CloudFront. That is not a new weakness and not a > loose end: `DisableExecuteApiEndpoint` cannot be turned on, because the > distribution's own origin **is** that hostname. The `Origin` check is a CSRF > control rather than authentication either way, and 6.3's throttle applies at > the stage, so it covers both paths. --- ## Part 7 — End to end, with a real submission ### 7.1 The route answers ```bash curl -si -X POST "${SITE}/api/intake" \ -H 'Origin: https://adr.smlcompany.ca' \ -H 'Content-Type: application/x-www-form-urlencoded' \ --data 'deploy-route-probe=1' | head -12 ``` **Expect:** `HTTP/2 303` and `location: https://adr.smlcompany.ca/contact/could-not-send/`. The handler accepted the request, validated it, found an empty submission and redirected to the failure page — **before** any write and any email. This is the same probe `scripts/deploy-local.sh` runs at the end of every deploy. ⚠️ **404 IS AMBIGUOUS AND THE FIRST DRAFT OF THIS LINE WAS NOT.** It means **either** the `/api/*` behaviour is missing (Part 3) **or** the `POST /api/intake` route is missing or misspelled (Part 6.2) — two different repairs. And because Part 3's custom error response is **distribution-wide**, a 404 from API Gateway is served as the styled `/404.html` body, so the one string that would have named the cause — API Gateway's `{"message":"Not Found"}` — is replaced before you see it. **Check the route first; it is one command:** `aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].RouteKey'`. **403** means the `Origin` header did not arrive, and **as of 2026-09-04 there are two policies it could be** — read which one the behaviour carries before repairing: - **`adr-sml-api-viewer-address`** (Part 3, change 8) — a **whitelist**. If `Origin` is missing from its Headers list, or the list drifted, every real submission 403s. Roll back by PUTting the managed id below with `update-distribution --if-match` — **not** by re-running `configure.mjs`, which converges forward and would re-attach the whitelist. - **`Managed-AllViewerExceptHostHeader`** (`b689b0a8-53d0-40ab-baf2-68738e2966ac`) — what it replaced, and what a rollback restores. Either way, a policy that drops `Origin` turns every real submission into a 403. **500** means Part 6.1 was skipped. ### 7.2 A real submission, from the real form Do this in a browser at `${SITE}/contact/` **after Part 8**, because the form does not exist until the site is deployed. Fill it as an inquirer would, with `Your name: Cutover test ` so the record is identifiable, and a real address you can read. **Expect:** the browser lands on `/contact/received/`, and two emails arrive — the notification at `$NOTIFY_TO`, replyable to the address you entered, and the confirmation at that address. Read the confirmation and check three things: the response-time sentence, the **four**-clause no-retainer notice, and that the field summary uses readable labels ("Your name", "Subject matter") rather than field names. Then find the record. The notification email prints `submissionId ` — that value **is** the partition key, so it can be used directly: ```bash export SUB_ID='' aws dynamodb get-item --table-name "$TABLE" \ --key "{\"submissionId\":{\"S\":\"${SUB_ID}\"}}" \ --query 'Item.{id:submissionId.S,at:submittedAt.S,ttl:ttl.N,name:name.S,area:practiceArea.S,sourceIp:sourceIp.S}' curl -s https://checkip.amazonaws.com ``` ⚠️ **READ `sourceIp` AGAINST YOUR OWN ADDRESS — THIS IS THE ONLY PLACE THE PROXY CHAIN GETS MEASURED, AND IT HAS ALREADY BEEN REASONED ABOUT WRONGLY TWICE.** The handler stores `requestContext.http.sourceIp`, which is the TCP peer. Behind the `/api/*` behaviour that peer is a **CloudFront edge**, so the expectation is an AWS address, not yours. Three outcomes and each has a different consequence: | what `sourceIp` holds | what it means | |---|---| | an **AWS** address (not the `checkip` value) | As designed. The field records the CDN, so it **cannot serve abuse investigation**, and `/legal/privacy/`'s *"your IP address"* is inaccurate — fold it into the Q62 edit on the same page rather than leaving two wrong sentences there | | **your** address, matching `checkip` | Better than expected, and worth knowing before anyone relies on it. Do not conclude it is trustworthy: verify it is not simply echoing a header by resubmitting with `-H 'X-Forwarded-For: 8.8.8.8'` and confirming `8.8.8.8` is **not** what lands. ⚠️ **AND CORRECT `/legal/privacy/` §What is collected**, whose network-address paragraph says the address is *"normally the network's own rather than your connection's"* — wrong in this branch, and it understates what is held about the reader. This row carried **no instruction at all** until 2026-09-02, so two of these three outcomes had nothing reconciling the page with the measurement (`adversarial-reviewer`, round 2) | | `8.8.8.8` after that resubmission | **Stop.** The field is client-controlled and a record can be made to name an uninvolved third party. Revert to storing nothing rather than storing that | ⚠️ **AN EARLIER REVISION OF THE HANDLER READ `x-forwarded-for` HERE, AND THAT WAS THE THIRD OUTCOME.** CloudFront **appends** the viewer address to a client-supplied XFF rather than replacing it, so the leftmost entry is whatever the client sent. The fix, if a usable value is wanted, is a **custom** origin request policy on `/api/*` forwarding `CloudFront-Viewer-Address`, which CloudFront generates and overwrites — not the managed `AllViewerAndCloudFrontHeaders`, which forwards `Host` and would 403 every request at API Gateway. ⚠️ **THAT CHANGE IS NOW IN THIS RUNBOOK — Part 3, change 8, written 2026-09-04 on Pouya's ruling and NOT YET APPLIED.** This paragraph said it was *"deliberately not in this runbook: measure first"*, which was true until the ruling and false afterwards. **Forwarding the header does not change what is stored:** `viewerIp()` still records `requestContext.http.sourceIp`, and the decision table above is still the procedure for changing that, because what the record holds is published field by field on `/legal/privacy/`. Measure first still governs the STORING, not the forwarding. **Expect** the item, with `ttl` a 10-digit epoch-seconds value. Check it is 24 months out — read it, do not assume it: ```bash python3 - < {d.isoformat()} ({months} months from now)') print('PASS' if months == 24 else 'FAIL — /legal/privacy/ promises 24 months') PY ``` **Expect:** `24 months from now` and `PASS`. `/legal/privacy/` states the period publicly, so a number that is not 24 is a false disclosure rather than a config error. ⚠️ **DELETE THE TEST RECORD when you are done with it** — it is a real row in a table whose contents are governed by a published privacy policy. ```bash aws dynamodb delete-item --table-name "$TABLE" --key "{\"submissionId\":{\"S\":\"${SUB_ID}\"}}" ``` ### 7.3 The four pre-existing items The table held **4 items** before any of this `[verified 2026-09-01 — describe-table ItemCount]`. They were written by the hand-built handler, whose code is in `docs/reference/AWS-Hosting-Guide.md` Part 8.3 and **writes no `ttl` attribute** — so DynamoDB will never expire them and they are retained indefinitely, against a policy that says otherwise. **This is Pouya's call and not a command in this runbook**, because it depends on something no agent can determine: whether those four are the guide's own smoke test (`Test User`) or real inquiries that arrived through the old site. Read them in the console, then either delete the test rows or write a `ttl` onto the real ones. Tracked on `docs/06`'s checklist. --- ## Part 8 — The first production deploy ⚠️ **DO NOT RUN 8.3 UNTIL POUYA HAS READ EVERY PAGE.** `docs/06`'s Content and compliance group — the `claims-auditor` pass and his own page-by-page read — is the gate on this part, and it is not a technical check. ### 8.1 Archive what is there S3 versioning is `Enabled`, so pass 3's `--delete` is recoverable — but `docs/06`'s post-cutover list says to archive the old build rather than rely on that, and a local copy costs one command. ```bash mkdir -p _archive/pre-cutover-$(date +%F) aws s3 sync "s3://${BUCKET}" "_archive/pre-cutover-$(date +%F)/" --no-progress ls -la "_archive/pre-cutover-$(date +%F)/" find "_archive/pre-cutover-$(date +%F)" -type f | wc -l ``` **Expect:** 3 files — `index.html` at 2,206,032 bytes and two PNGs under `assets/`. Do not commit them; `docs/06` wants them kept, not versioned. ### 8.2 Dry run — the three passes, in order, writing nothing ```bash npm run check && npm run build && npm run check:claims && npm run og:proof && npm run check:intake echo "gates exit=$?" npm run lighthouse; echo "lighthouse exit=$?" ``` **Expect:** every one exit 0. `lighthouse` is local-only and cannot run in CI; it is a keyboard gate and this is the keyboard. ```bash aws s3 sync ./dist "s3://${BUCKET}" --dryrun \ --exclude "*" --include "_astro/*" --include "fonts/*" \ --cache-control "public, max-age=31536000, immutable" --no-progress | tail -5 aws s3 sync ./dist "s3://${BUCKET}" --dryrun \ --exclude "*" --include "*.avif" --include "*.webp" --include "*.jpg" \ --include "*.png" --include "*.svg" \ --cache-control "public, max-age=604800" --no-progress | tail -5 aws s3 sync ./dist "s3://${BUCKET}" --dryrun \ --exclude "_astro/*" --exclude "fonts/*" \ --cache-control "public, max-age=0, must-revalidate" --delete --no-progress | tail -8 ``` **Expect,** on the bucket state Part 0.3 records — measured 2026-09-01, so these are counts rather than shapes: **51** `(dryrun) upload:` lines in pass 1, **56** in pass 2, and **52** lines in pass 3 of which **two are `(dryrun) delete:`** — `assets/sml-logo-full.png` and `assets/sml-logo-mark.png`, the old site's two logos. `404.html` appears in pass 3. `index.html` is not deleted; it is overwritten. ⚠️ **PASS 2's DRY RUN IS A SUPERSET OF WHAT PASS 2 ACTUALLY UPLOADS, AND THE REAL RUN WILL PRINT FEWER LINES.** 34 of those 56 are files under `_astro/`, which pass 1 uploads first. Each `--dryrun` above is compared against the bucket **as it is now**, so it cannot see the uploads the preceding pass would have made; in the real sequential run `aws s3 sync` skips them as already in sync. That skipping is exactly what preserves pass 1's `immutable` header on those files rather than overwriting it with pass 2's week-long one — `docs/06` §Cache policy says the same thing about pass 3, and this is why the pass order is load-bearing. **Do not reorder the passes to make this output tidier, and do not read the smaller real counts as a failed upload.** ⚠️ **Check the largest file in `./dist` while you are here.** The deploy user has no `s3:AbortMultipartUpload`, which is safe only while nothing approaches `aws s3 sync`'s 8 MB multipart threshold (`docs/06`). ```bash find ./dist -type f -exec stat -f '%z %N' {} + | sort -rn | head -3 ``` **Expect** the largest well under 8,388,608 bytes — it was **171,876** on 2026-09-01. GNU `find -printf` is not available here; this is the BSD/macOS form, and writing it as `find -printf … || find -exec …` would not have fallen back, because the exit status of that pipeline belongs to `head`. ### 8.3 The deploy — on Pouya's word only ⚠️ **NOT AS `user/pouya`.** The script refuses that identity and §10 is why: the account is shared with unrelated production systems. Use the `adr-sml-deploy` key, which §7 records as provisioned. ```bash AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… \ AWS_REGION="$AWS_REGION" S3_BUCKET="$BUCKET" CLOUDFRONT_DISTRIBUTION_ID="$DIST_ID" \ npm run deploy ``` The script runs `npm run check`, the build, `check:claims`, the three passes, the `/*` invalidation, and then the intake route check. **Read its last line.** It warns rather than fails on a bad intake route, because by then the site is already published and failing the job would not un-publish it. ```bash git tag "v$(date +%Y).1" && git push --tags ``` ### 8.4 Verify the site, not the deploy ```bash for p in / /about/ /mediation/ /arbitration/ /med-arb/ /practice/ /practice/construction/ \ /process/ /for-parties/ /fees/ /insights/ /contact/ /bio/ /legal/privacy/ \ /legal/terms/ /robots.txt /sitemap-index.xml /pouya-lajevardi-bio.pdf; do printf '%-32s ' "$p" curl -s -o /dev/null -w 'status=%{http_code} bytes=%{size_download}\n' "${SITE}${p}" done printf '%-32s ' "404 body"; curl -s "${SITE}/nope-not-a-page/" -o /tmp/404.html -w 'status=%{http_code}\n' grep -c 'That page is not here' /tmp/404.html printf '%-32s ' "cache header on HTML"; curl -sI "${SITE}/about/" | grep -i '^cache-control' printf '%-32s ' "cache header on asset" curl -s "${SITE}/about/" | grep -o '/_astro/[^"]*\.css' | head -1 \ | xargs -I{} curl -sI "${SITE}{}" | grep -i '^cache-control' ``` **Expect:** every page 200 with real bytes; the 404 path **404** *and* the phrase `That page is not here` found once; `max-age=0, must-revalidate` on HTML; `max-age=31536000, immutable` on the hashed CSS. Only now is `docs/06`'s "404 returns a 404 status" item true on both halves. --- ## Part 9 — Rollback, per part Each of these is independent. None of them needs the others undone first. **9.1 Part 1** — `aws s3api put-bucket-policy --bucket "$BUCKET" --policy "file:///tmp/bucket-policy.before.json"`. Missing keys go back to 403 and the 404 mapping stops firing; nothing else changes. **9.2 Parts 2–3** — re-run `configure.mjs` is *not* a rollback; it is idempotent forward-only, **and that now matters most for change 8**: section 5 re-attaches the `/api/*` whitelist on the next `--apply`, because a deliberately reverted behaviour and a never-configured one are byte-identical in the config and no detector can separate them. To undo, `get-distribution-config`, remove the `FunctionAssociations` entry / the `404` custom error response / the `/api/*` behaviour and the `intake-api` origin, and `update-distribution --if-match`. **Sections 4 and 5 were added after this paragraph and undo the same way:** put `/api/*`'s `OriginRequestPolicyId` back to `b689b0a8-53d0-40ab-baf2-68738e2966ac` (the managed policy) and/or remove the `*.pdf` behaviour, with `update-distribution --if-match`. The two custom policies `adr-sml-api-viewer-address` and `adr-sml-pdf-noindex` can then be deleted with `delete-origin-request-policy` / `delete-response-headers-policy`, each of which **fails while still attached** — the same ordering feature as the function below. ⚠️ **Deleting the policies does not prevent re-attachment either** — the next `--apply` simply creates them again by name and attaches them. Nothing in this script can be made to remember a deliberate revert, because a reverted behaviour and a never-configured one are byte-identical in the config. **The rollback holds only until someone runs `configure.mjs --apply` again**; that is a property of a forward-converging script, and the fix if it ever matters is a flag, not a deletion. Then `aws cloudfront delete-function --name adr-sml-router --if-match `, which fails while the function is still associated — that ordering is a feature. **9.3 Part 5** — the previous code is not recoverable from Lambda: the hand-built function was edited in the console and its 1,527 bytes exist only in `docs/reference/AWS-Hosting-Guide.md` Part 8.3. That is the rollback source, and it answers a different contract (JSON in, `{"ok":true}` out) that the current `/contact/` form does not speak. **Forward is the only real fix here** — which is worth knowing before 5.3, not after. **9.4 Part 6** — `delete-route` the new route, `remove-permission` the new statement, and re-add the `/submissions` route and its permission if anything still needs it. Nothing does. **9.5 Part 8** — S3 versioning is `Enabled`. Restore the previous version of `index.html`, `aws s3 rm` what the new build added, invalidate `/*`. Or `git revert` and re-deploy, which is cleaner and is what the tag in 8.3 is for. --- ## Part 10 — Q60: prove TTL deletes, not just that it is enabled `AGENTS.md` §7 records `TimeToLiveStatus: ENABLED` on attribute `ttl` `[re-verified 2026-09-01]`, and **that proves the setting and not the behaviour**. `/legal/privacy/` tells the public that a record is *"deleted automatically by the database rather than by someone remembering to do it"* — an assertion about the mechanism. Q60 closes when a record has been watched to disappear, and not before. §12 R19 keeps it surfacing until then. ### 10.1 Write one record with a near-future `ttl` Not through the form: the handler writes a 24-month `ttl` and waiting two years is not a test. Write it directly, with a shape that cannot be mistaken for an inquiry. ```bash export TTL_ID="q60-ttl-probe-$(date +%Y%m%dT%H%M%SZ)" export TTL_AT=$(python3 -c 'import time; print(int(time.time()) + 300)') python3 -c "import datetime,os; print('ttl', os.environ['TTL_AT'], '->', datetime.datetime.fromtimestamp(int(os.environ['TTL_AT']), datetime.timezone.utc).isoformat())" aws dynamodb put-item --table-name "$TABLE" --item "$(cat < ~/q60-ttl-probe-id.txt echo "$TTL_AT" >> ~/q60-ttl-probe-id.txt ``` **Expect:** the item, with `ttl` five minutes in the future. The id is written to a file because the check below happens on a different day and this is the only thing that connects the two. ### 10.2 The check, and it is not five minutes later ⚠️ **DYNAMODB'S TTL SWEEPER IS NOT PROMPT AND AWS DOES NOT PROMISE THAT IT IS.** Deletion typically happens within a couple of days of the timestamp passing, and **up to 48 hours or more is normal and not a fault.** A check run at `ttl + 5 minutes` that finds the item present has measured nothing — it is the same mistake as reading a value before the transition finishes. So: - **Earliest useful check: 48 hours after `TTL_AT`.** - **Do not conclude a failure before 7 days.** ```bash export TTL_ID=$(sed -n 1p ~/q60-ttl-probe-id.txt) export TTL_AT=$(sed -n 2p ~/q60-ttl-probe-id.txt) python3 -c "import time,os; d=time.time()-int(os.environ['TTL_AT']); print(f'{d/3600:.1f} hours since the ttl passed')" aws dynamodb get-item --table-name "$TABLE" \ --key "{\"submissionId\":{\"S\":\"${TTL_ID}\"}}" --output json echo "get-item exit=$?" ``` **Read both the output and the exit status.** `get-item` on a missing key exits **0** with an **empty response body** — it is not an error, and a script that only checked the exit status would report the record present and absent identically. | result | meaning | |---|---| | `{"Item": {…}}`, under 48 h since `TTL_AT` | inconclusive. Wait. | | `{}` (empty), any time after `TTL_AT` | **Q60 closes.** The mechanism works. | | `{"Item": {…}}`, more than 7 days after | **Q60 fails.** TTL is enabled and not deleting. `/legal/privacy/` is then making a claim the database does not honour, and the page has to change or the mechanism does. | ### 10.3 Record it On the `{}` result, and only then: - Stamp §7's `Intake table` row: the behaviour is **observed**, with the date and the elapsed time, alongside the `ENABLED` reading it already carries. - Close **Q60** in §9, quoting the two timestamps. - Retire **R19** from §12 — its whole purpose was to keep this surfacing. - Tick the TTL item on `docs/06`'s checklist. It has two halves and this is the second: `ENABLED` proved the setting, this proves the behaviour. - Delete the probe row if it somehow survives, and delete `~/q60-ttl-probe-id.txt`. If it fails, that is a **published-disclosure defect**, not a backlog item: `src/pages/legal/privacy.astro` carries the matching `TODO(pouya)` and the page asserts the mechanism in as many words.