Files
adr-sml/docs/09-cutover-runbook.md
T
Pouya LajevardiandClaude Opus 5 a07193d561
Build and deploy / build-and-deploy (push) Failing after 4s
fix: pre-flight the CloudFront payload limits the dry run is the only guard for
The second `--apply` of 2026-09-04 created adr-sml-pdf-noindex and then failed
at create-origin-request-policy: InvalidArgument, "The parameter Comment is too
big" — 182 characters against a 128 cap. update-distribution never ran, so the
distribution is unchanged, but the account was left holding an orphaned policy.

Nothing local could have caught it, and that is now measured rather than
assumed: botocore/validate.py checks neither `max` nor `pattern` (range_check
reads only `min`; the word `pattern` does not appear in the file), and the 128
is not modelled as a constraint at all — `Comment` is a bare `string` and the
cap lives in the shape's documentation prose. So the dry run really is the only
pre-flight, and it now enforces PAYLOAD_LIMITS: 13 entries across both policy
payloads and the function ARN, each with the source it came from.

The entries that matter guard CLONED values rather than literals this file
authors — a literal is reviewed when it is written, while a value copied out of
the default behaviour's policy changes with no diff here. The API declares
TooLongCSPInResponseHeadersPolicy for exactly that case and docs/05 already
specifies a CSP that would land there. RemoveHeadersConfig is a recorded gap:
its cap is real but unpublished, and inventing a number would be worse.

Both comments are now 76 and 74 characters. `--function-arn` is validated
before any AWS call, and an unrecognised `--flag` is a usage error — the `=`
form was invisible to the parser and to the presence check, for a clean exit 0
with no router attached. A skipped section now exits 3, because docs/09 uses
exit 0 as its own success stamp and a partial run read as a complete one.

Confirmed by measurement, as asked: the next run REUSES the orphan by name,
matches every reconciled field, and stages it — create line gone, 4 changes
down to 3, no duplicate and no collision.

Reviewed twice. Round 2 found that the §7 record broke the table it lives in,
and that two comments asserted behaviour the code did not have. 17 findings
across both rounds, all fixed.

Nothing was applied to the distribution and nothing was deployed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Md3GndFqWPzK78xAoebsg5
2026-09-04 14:24:33 -04:00

1448 lines
74 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 09 — Cutover runbook: the exact command sequence
Authority: `AGENTS.md` §3 D11 (build everything, one clean cutover) and §7 for
every operational fact. `docs/06-deployment.md` owns the cutover **checklist**
what must be true. This file owns the **commands** — how to make it true. The
checklist cites this file; this file does not restate the checklist.
**Pouya runs everything here.** Parts 17 need administrative credentials that no
agent on this project holds, and Part 8 is a deploy that must not run as
`user/pouya` (§10). Every command is followed by the command that verifies it and
the output to expect. Where a verification can come back two ways, both are named
and the sequence branches — a branch driven by a measurement rather than by a
guess is the point.
> **Read `docs/06`'s cutover checklist alongside this.** The parts below discharge
> its Technical group. Its Content and compliance group — the `claims-auditor`
> pass and Pouya's own page-by-page read — is not commands and is not here.
---
## Why the order is this order, and what is broken while it runs
The bucket currently holds the **old** single-file site: `index.html` (2,206,032
bytes) plus two logo PNGs `[verified 2026-09-01 — list-objects-v2]`. So the live
site today is one page, and one page is all that works: measured on the live
distribution the same day, `/` returns **200** while `/about/` and
`/definitely-not-a-page/` both return **403 with an 111-byte `application/xml`
body** — S3's `AccessDenied`, served raw to the reader.
That is why the infrastructure goes first and the site goes last:
1. **Parts 14 (distribution) change nothing a visitor can see.** The router
rewrites `/` to `/index.html`, which is what the default root object already
did, and the paths it newly handles do not exist on the old site either way.
2. **Parts 57 (intake) are inert until `/contact/` exists.**
3. **Part 8 is the cutover.** The moment `index.html` syncs, the new site is
live — and by then the router, the 404 mapping and `/api/*` are already in
place, so there is no window in which 22 of 23 pages are broken.
Reversing this — site first, infrastructure second — would put every page except
`/` behind a 403 for as long as the CloudFront deployment took.
---
## Part 0 — Preflight
### 0.1 The substitution block
⚠️ **THIS IS THE ONLY PLACE IN THIS FILE THAT NAMES A RESOURCE, AND THAT IS A
DELIBERATE, BOUNDED EXCEPTION TO THE §7 RULE.** `AGENTS.md` §7 is the single
source of truth for operational facts and specs cite it rather than copying it —
but a runbook that cannot be pasted into a terminal is not a runbook. So the
values appear exactly once, and **0.2 is the mechanism that keeps the copy
honest**: it proves every one of them resolves to a real resource before anything
is changed. A duplicated fact needs a mechanism, not a promise.
Confirm each against §7 before pasting. `./scripts/aws-discover.sh` regenerates
them from AWS if §7 itself needs re-verifying.
```bash
cd ~/Dev/Websites/adr-sml
export AWS_REGION=ca-central-1
export AWS_DEFAULT_REGION=ca-central-1
export ACCT=327082975128
export FN=adr-intake-handler
export TABLE=adr-intake-submissions
export API_ID=4tl0m5igkj
export API_DOMAIN=4tl0m5igkj.execute-api.ca-central-1.amazonaws.com
export DIST_ID=E1OK7G98KNKUTA
export BUCKET=adr-smlcompany-site
export SITE=https://adr.smlcompany.ca
export NOTIFY_TO=info@smlcompany.ca
export MAIL_FROM=intake@smlcompany.ca
```
`NOTIFY_TO` is where an inquiry lands and `MAIL_FROM` is the From on both
messages. Both are choices rather than records: `info@` is the address `/contact/`
publishes, so a reply threads where the inquirer expects; `intake@` distinguishes
form mail from correspondence. Both must be at `smlcompany.ca`, which is the
verified sending identity — 0.2 checks that.
### 0.2 Prove every one of them exists
```bash
aws sts get-caller-identity
for v in AWS_REGION ACCT FN TABLE API_ID API_DOMAIN DIST_ID BUCKET SITE NOTIFY_TO MAIL_FROM; do
eval "val=\$$v"
[ -n "$val" ] || { echo "EMPTY: $v"; break; }
printf '%-12s %s\n' "$v" "$val"
done
aws lambda get-function-configuration --function-name "$FN" --query 'FunctionName' --output text
aws dynamodb describe-table --table-name "$TABLE" --query 'Table.TableStatus' --output text
aws apigatewayv2 get-api --api-id "$API_ID" --query 'Name' --output text
aws cloudfront get-distribution --id "$DIST_ID" --query 'Distribution.Status' --output text
aws s3api head-bucket --bucket "$BUCKET" && echo "bucket ok"
aws sesv2 get-email-identity --email-identity "${MAIL_FROM#*@}" \
--query 'VerifiedForSendingStatus' --output text
```
**Expect:** an identity with administrative rights; eleven non-empty values;
`adr-intake-handler`; `ACTIVE`; `adr-intake-api`; `Deployed`; `bucket ok`; `True`.
A `None`, an empty value, or any non-zero exit stops the run here. That is the
whole purpose of 0.2 — a stale identifier fails loudly at the top instead of
halfway through Part 6.
### 0.3 State this run starts from
Recorded so a later reader can tell what this sequence changed from what it found.
All `[verified 2026-09-01]`, read-only:
| | |
|---|---|
| Lambda | `nodejs24.x`, arm64, handler `index.handler`, timeout 10 s, memory 128 MB, **no environment variables**, no DLQ, code 1,527 bytes (the hand-built inline function), last modified 2026-05-26 |
| Lambda role | `adr-intake-lambda-role``AWSLambdaBasicExecutionRole` plus an inline policy granting `dynamodb:PutItem` on the table and `ses:SendEmail`/`ses:SendRawEmail`. **Sufficient for the new handler; no change needed** |
| Lambda invoke permission | one statement, `apigateway.amazonaws.com`, **`SourceArn` scoped to `…/4tl0m5igkj/*/*/submissions`** — the old route's path only |
| HTTP API | one route, `POST /submissions` → integration `0ftgjgv` (`AWS_PROXY`, payload format **2.0**, the correct one for this handler). Stage `$default`, auto-deploy on, **no throttling**, no access log. CORS allows `POST` from the site origin |
| DynamoDB | `ACTIVE`, on-demand, **partition key `submissionId` (S), no sort key**, 4 existing items, TTL **ENABLED** on attribute `ttl`, PITR **ENABLED** (35-day window), encryption at rest with the **AWS-owned** key (no CMK) |
| CloudFront | one S3 origin with OAC `E13GAFUL6UQP6R`; default behaviour `Managed-CachingOptimized` + `Managed-SecurityHeadersPolicy`, methods HEAD/GET; **no cache behaviours**, **no custom error responses**, **no function associations** |
| S3 | versioning `Enabled`; all four public-access blocks `true`; direct object GET on the S3 hostname returns 403, so OAC is in force |
| SES | domain verified for sending, DKIM `SUCCESS` and signing enabled, no custom MAIL FROM |
| SNS `ses-alerts` | email subscription to `info@smlcompany.ca` is **CONFIRMED** — it has a real subscription ARN, not `PendingConfirmation`. §7 recorded it as pending for six days after it stopped being pending; §12 R9 closes on this |
---
## Part 1 — Make S3 answer 404 instead of 403
**Why this is first.** Part 3 maps **404** to `/404.html` and deliberately does
**not** map 403. With OAC and no `s3:ListBucket`, S3 answers a request for a
missing key with `403 AccessDenied`, so a 404-only mapping would never fire and
every bad URL would keep returning raw XML. Granting the CloudFront service
principal `s3:ListBucket` changes that answer to `404 NoSuchKey`.
Mapping 403 as well would have been one command shorter and is the wrong trade:
custom error responses are distribution-wide and cannot be scoped to one
behaviour, so it would also convert a broken bucket policy or a broken OAC — a
real outage on every URL at once — into a tidy "page not found", and it would
turn the intake handler's `Origin` refusal into a 404 page. Keeping 403 meaning
403 is worth one statement.
```bash
aws s3api get-bucket-policy --bucket "$BUCKET" --query Policy --output text > /tmp/bucket-policy.before.json
python3 -m json.tool /tmp/bucket-policy.before.json
```
**Expect:** one statement, `AllowCloudFrontServicePrincipal`, `s3:GetObject` on
`arn:aws:s3:::$BUCKET/*`, conditioned on the distribution ARN. Keep that file —
Part 9 restores from it.
```bash
python3 - "$BUCKET" "$ACCT" "$DIST_ID" <<'PY' > /tmp/bucket-policy.after.json
import json, sys
bucket, acct, dist = sys.argv[1], sys.argv[2], sys.argv[3]
p = json.load(open('/tmp/bucket-policy.before.json'))
arn = f'arn:aws:cloudfront::{acct}:distribution/{dist}'
sid = 'AllowCloudFrontListForHonest404s'
p['Statement'] = [s for s in p['Statement'] if s.get('Sid') != sid]
p['Statement'].append({
'Sid': sid,
'Effect': 'Allow',
'Principal': {'Service': 'cloudfront.amazonaws.com'},
'Action': 's3:ListBucket',
'Resource': f'arn:aws:s3:::{bucket}',
'Condition': {'StringEquals': {'AWS:SourceArn': arn}},
})
json.dump(p, sys.stdout, indent=2)
PY
python3 -m json.tool /tmp/bucket-policy.after.json
aws s3api put-bucket-policy --bucket "$BUCKET" --policy "file:///tmp/bucket-policy.after.json"
```
**Verify — three things, and the third is the one worth doing:**
```bash
aws s3api get-bucket-policy --bucket "$BUCKET" --query Policy --output text | python3 -m json.tool
aws s3api get-public-access-block --bucket "$BUCKET" --query 'PublicAccessBlockConfiguration'
curl -s -o /dev/null -w 'direct S3 object: %{http_code}\n' \
"https://${BUCKET}.s3.${AWS_REGION}.amazonaws.com/index.html"
aws cloudfront get-distribution-config --id "$DIST_ID" \
--query 'DistributionConfig.DefaultCacheBehavior.{Cache:CachePolicyId,OriginRequest:OriginRequestPolicyId}'
aws cloudfront get-cache-policy --id 658327ea-f89d-4fab-a63d-7e88639e58f6 \
--query 'CachePolicy.CachePolicyConfig.ParametersInCacheKeyAndForwardedToOrigin.QueryStringsConfig'
```
**Expect:** two statements; all four public-access blocks still `true`; direct S3
`403`; `Cache` = `658327ea-…` (`Managed-CachingOptimized`) with
**`OriginRequest: null`**; and `QueryStringsConfig` = **`{"QueryStringBehavior":
"none"}"`**.
⚠️ **THOSE LAST TWO ARE THE CHECK, AND THE OBVIOUS ONE IS WORTHLESS.** The
tempting verification is `curl "${SITE}/?list-type=2&max-keys=5"` and confirming
it returns HTML rather than an XML `ListBucketResult` — **it cannot return
anything else, so it is evidence of nothing.** The distribution has
`DefaultRootObject: index.html` and, after Part 2, the router rewrites `/` to
`/index.html`; the origin request is therefore a `GetObject` on a key, never a
request for the bucket root, so `list-type=2` could not be read as a list
operation whether or not query strings reached S3. An operator running it sees
HTML and ticks a control that never ran, which is the Q22 shape exactly. Found by
`adversarial-reviewer`, 2026-09-01.
**What actually makes the grant safe, and the two commands above assert both
halves:** no viewer path resolves to the bucket root, and the default behaviour
forwards **no query strings** to the origin — with no origin request policy
attached to override that. If either assertion fails, revert this part (Part 9.1)
and map 403 instead.
---
## Part 2 — The trailing-slash router function
Source and test are in the repo: `infra/cloudfront/router.js` and
`router.test.mjs`. Read the header comment before publishing it — the reason it
must not be associated with `/api/*` is in there, and it is the difference
between a working form and a POST silently converted to a GET.
```bash
node infra/cloudfront/router.test.mjs
```
**Expect:** exit **0**. The case count is deliberately not quoted here — it has
already gone 15 → 22 → 30 in one day, and an `Expect` line that never matches
teaches the operator to stop reading `Expect` lines. The script asserts its own
count internally (`case count != CASES.length`), so exit 0 is the whole check.
```bash
aws cloudfront create-function \
--name adr-sml-router \
--function-config '{"Comment":"trailing-slash + index.html for the Astro directory build; docs/09","Runtime":"cloudfront-js-2.0"}' \
--function-code "fileb://infra/cloudfront/router.js" \
--query '{Name:FunctionSummary.FunctionMetadata.FunctionARN,Stage:FunctionSummary.FunctionMetadata.Stage,Status:FunctionSummary.Status}'
```
**Expect:** an ARN, `Stage: DEVELOPMENT`, `Status: UNPUBLISHED`.
**Test it in the real runtime before publishing.** The local test proves the
branches; this proves the runtime accepts the code, which the local test cannot.
```bash
ETAG=$(aws cloudfront describe-function --name adr-sml-router --query ETag --output text)
for URI in / /about/ /about /robots.txt /_astro/x.css /api/intake \
//evil.example.com/x '/\evil.example.com/x' //about/ //robots.txt; do
printf '{"version":"1.0","request":{"method":"GET","uri":"%s","querystring":{},"headers":{},"cookies":{}}}' "$URI" \
> /tmp/cf-event.json
printf '%-16s ' "$URI"
aws cloudfront test-function --name adr-sml-router --if-match "$ETAG" --stage DEVELOPMENT \
--event-object fileb:///tmp/cf-event.json \
--query 'TestResult.{Out:FunctionOutput,Err:FunctionErrorMessage}' --output text
done
```
`fileb://` rather than a base64 argument: it hands the CLI raw bytes and sidesteps
the question of whether the local `base64` wraps its output, which would make the
argument invalid in a way the error message would not explain. `test-function`
does not change the function, so one `ETAG` read serves the whole loop.
Then the query-string path, which none of the URIs above exercises:
```bash
cat > /tmp/cf-event-qs.json <<'JSON'
{"version":"1.0","request":{"method":"GET","uri":"/fees","headers":{},"cookies":{},
"querystring":{"utm":{"value":"a%20b"},"q":{"value":"x|y"},"bad":{"value":"z\"<>#&k=v"}}}}
JSON
aws cloudfront test-function --name adr-sml-router --if-match "$ETAG" --stage DEVELOPMENT \
--event-object fileb:///tmp/cf-event-qs.json \
--query 'TestResult.{Out:FunctionOutput,Err:FunctionErrorMessage}' --output text
```
**Expect,** in order for the ten URIs: `uri /index.html`; `uri
/about/index.html`; a **301 to `/about/`**; `uri /robots.txt` unchanged; `uri
/_astro/x.css` unchanged; `/api/intake` a **301 to `/api/intake/`**; and then
**four 301s that all stay on this origin with a single leading slash**
`/evil.example.com/x/`, `/evil.example.com/x/`, `/about/`, `/robots.txt`.
No `FunctionErrorMessage` on any of the ten.
From the query-string call: a **301 to
`/fees/?utm=a%20b&q=x|y&bad=z&k=v`**. Three things are being checked there and
each has been wrong once: `%20` survives (the values arrive percent-encoded, so
re-encoding would produce `%2520`); `|` survives (it is not in WHATWG's query
percent-encode set, so browsers send it raw, and one revision of the function
stripped it and silently corrupted campaign links); and `"`, `<`, `>` and `#` are
gone (`#` is the one that changes the *structure* of the header — left in, `&k=v`
lands in a fragment and the parameter is lost).
⚠️ **THE FIRST VERSION OF THIS LOOP PASSED `"querystring":{}` ON EVERY CASE AND
TRIED NEITHER REDIRECT PATH.** So in the only environment that runs the real
`cloudfront-js-2.0` runtime — the environment this step exists to reach — none of
the query-string handling and none of the normalisation was executed, while the
prose called it "the authoritative check". Found by `adversarial-reviewer`
round 2.
⚠️ **That last row is the reason this function must not be associated with the
`/api/*` behaviour, and it is in the list so you see it rather than read about
it.** A 301 turns a POST into a GET, so an intake submission routed through this
function would arrive at the handler as a GET with no body — and the visitor would
be told nothing was wrong. Part 3 associates the function with the **default**
behaviour only; `configure.mjs` sets `FunctionAssociations: { Quantity: 0 }` on
the `/api/*` behaviour explicitly rather than by omission.
```bash
aws cloudfront publish-function --name adr-sml-router --if-match "$ETAG"
export ROUTER_ARN=$(aws cloudfront describe-function --name adr-sml-router --stage LIVE \
--query 'FunctionSummary.FunctionMetadata.FunctionARN' --output text)
echo "ROUTER_ARN=$ROUTER_ARN"
```
**Expect:** a non-empty ARN. It is empty if `publish-function` failed — read the
status, not the absence of an error.
---
## Part 3 — Apply the five distribution changes
One script, `infra/cloudfront/configure.mjs`, because the alternative is
hand-editing a 300-line JSON document and posting it back with an `IfMatch` ETag.
It is **dry-run by default**, **idempotent**, and it resolves both managed policy
ids **by name from the account** rather than carrying them as literals.
```bash
node infra/cloudfront/configure.mjs --dist "$DIST_ID" --api-domain "$API_DOMAIN" \
--function-arn "$ROUTER_ARN"
```
⚠️ **THE BLOCK BELOW IS THE `+` CHANGE LINES AND THE TWO RESOLVED-POLICY LINES.
IT IS NOT THE WHOLE OUTPUT, AND IT SAID "exactly" UNTIL 2026-09-04.** Against
the **live** distribution the dry run is **59 lines**, and all 59 account for:
2 resolved-policy lines, **10** `·` lines, **3** `+` lines, a **40-line JSON
dump** of the `*.pdf` behaviour it would add, 2 blank lines, the
`N change(s) to distribution …` header and the `DRY RUN — nothing was sent.`
footer `[measured 2026-09-04, exit 0, nothing written]`.
🛑 **READ THE EXIT STATUS, AND IT HAS THREE VALUES.** `0` — everything this
script manages was applied or is already present. `2` — a usage error, before any
AWS call. **`3` — sections that could run did, and something was SKIPPED: read
the `⚠ … SKIPPED, not changed` block.** Anything else is a throw. `3` exists
because a skip used to exit `0`, and this document uses `exit 0` as its own
success stamp throughout — so a partial run read as a complete one. A skip is
never nothing: it means the `*.pdf` behaviour has no `X-Robots-Tag`, or `/api/*`
is not forwarding `CloudFront-Viewer-Address`, and the skip line says which.
⚠️ **THE SHAPE CHANGES WITH THE STATE, SO READ THE `+` LINES AND NOT THE
TOTAL.** Each item Parts 13 have already applied prints `·` when it is found
and `+` when it is staged, so in the Part 0.3 state four lines cross from one
column to the other and the totals move with them. **The `+` lines are the
check.**
**Expect** — this is the dry run, and on a distribution in the state Part 0.3
records the change lines are:
```
resolved Managed-CachingDisabled = 4135ea2d-6df8-44a3-9df3-4b5a84be39ad
resolved Managed-AllViewerExceptHostHeader = b689b0a8-53d0-40ab-baf2-68738e2966ac
8 change(s) to distribution E1OK7G98KNKUTA (ETag …):
+ DefaultCacheBehavior.FunctionAssociations viewer-request -> arn:…:function/adr-sml-router
+ CustomErrorResponses += 404 -> /404.html with status 404
+ Origins += intake-api -> …execute-api… (https-only, TLSv1.2)
+ CacheBehaviors += /api/* -> intake-api, CachingDisabled, AllViewerExceptHostHeader, POST allowed
+ create response-headers policy adr-sml-pdf-noindex (SecurityHeadersConfig cloned from … + X-Robots-Tag: noindex)
+ CacheBehaviors += *.pdf -> <s3-origin>, default cache policy, adr-sml-pdf-noindex (policy id created in the same --apply pass)
+ create origin request policy adr-sml-api-viewer-address (whitelist: CloudFront-Viewer-Address, Content-Type, Origin, Referer, User-Agent; cookies all; query strings all)
+ /api/* OriginRequestPolicyId b689b0a8-… -> adr-sml-api-viewer-address
DRY RUN — nothing was sent. Re-run with --apply to write it.
```
⚠️ **SECTION 4 CANNOT SHOW THE POLICY ID IN A DRY RUN, AND SAYS SO — IT IS
STILL ONE `--apply`.** The `*.pdf` behaviour has to carry the response-headers
policy's id, and on a first run that policy does not exist yet, so the dry run
prints the behaviour it *would* add with `(policy id created in the same --apply
pass)` where the id goes. **A single `--apply` creates the policy and adds the
behaviour in one call — do not run it twice.** The dry run reports both changes
either way; one that listed only the policy would hide the half that touches a
distribution serving 23 pages.
Fewer than eight changes means part of this is already done — read which lines
are prefixed `·` — but READ THE WORDS, not the bullet: `configure.mjs` uses `·`
for *already present* **and** for *would CREATE / would SET / would ADD*, so the
prefix alone does not say whether a line is done or still pending. **On the live distribution as at
2026-09-04, after the second `--apply` attempt, the dry run returns exactly
three `+` lines — **one** for section 4 (the behaviour; its policy already
exists, see the incident below) and the two for section 5** — `[measured 2026-09-04, dry run against `E1OK7G98KNKUTA`, ETag
`E2EUQ1WTGCTBG2`, exit 0, nothing written]`. More than eight, or a different set,
means the distribution is not in the state 0.3 recorded: stop and re-read it.
🛑 **INCIDENT — TWO `--apply` ATTEMPTS FAILED ON 2026-09-04, FOR TWO DIFFERENT
REASONS, AND THE SECOND ONE LEFT A POLICY BEHIND.** The distribution is
unchanged after both. Read both before the next attempt.
**ATTEMPT 1 — nothing reached the distribution and nothing was created.**
Section 4's clone was sent to `create-response-headers-policy` verbatim and the
AWS CLI rejected it **client-side**, before the call left the machine:
```
An error occurred (ParamValidation): Parameter validation failed:
Missing required parameter in ResponseHeadersPolicyConfig.SecurityHeadersConfig.ContentSecurityPolicy: "Override"
Missing required parameter in ResponseHeadersPolicyConfig.SecurityHeadersConfig.ContentSecurityPolicy: "ContentSecurityPolicy"
```
**The cause, and it generalises past this script: a config AWS hands back is not
necessarily a config AWS will accept.** `get-response-headers-policy` on
`Managed-SecurityHeadersPolicy` returns `"ContentSecurityPolicy": {}` — an empty
object standing for a member the policy does not define — and
`ResponseHeadersPolicySecurityHeadersConfig` has **no required members** while
**every one of its six members requires at least `Override`**. So an empty member
is always "undefined here" and is **never** a legal input.
⚠️ **AND THE SAME IS TRUE ONE LEVEL UP, WHICH THE FIRST FIX MISSED.** Every
sibling member of `ResponseHeadersPolicyConfig` also declares required fields —
`CorsConfig` five of them, `RemoveHeadersConfig` and `CustomHeadersConfig` a
`Quantity`, `ServerTimingHeadersConfig` an `Enabled` — while the container
itself requires only `Name`. So `{}` is a placeholder at **both** levels, and a
fix covering only the inner one turns the outer placeholder into a hard abort
instead of an omission. All of that is read out of the CLI's own service model,
not inferred from the symptom.
**State after the failure** ``[verified 2026-09-04 — `get-distribution-config`, `list-response-headers-policies --type custom`, `list-origin-request-policies --type custom`]``**:** `/api/*` still on
`b689b0a8-53d0-40ab-baf2-68738e2966ac`, **no** `*.pdf` behaviour, **zero** custom
response-headers policies, **zero** custom origin request policies.
⚠️ **THE "ONE REVIEWABLE TRANSACTION" PROPERTY IS ABOUT THE DISTRIBUTION, NOT
THE ACCOUNT — AND THIS RECORD ASSERTED THE WIDER VERSION FOR ONE ROUND.**
`update-distribution` is the script's last call, so a throw above it does leave
the **distribution** untouched. But sections 4 and 5 each make their own write
first — `create-response-headers-policy` and `create-origin-request-policy` —
and the script's own comment on section 5's drift throw documents a reachable
path where section 4 has **already created `adr-sml-pdf-noindex`** when section
5 aborts. **So after any failed `--apply`, check for an orphaned policy as well
as for a changed distribution**, with both of these:
```bash
aws cloudfront list-response-headers-policies --type custom --output json \
--query 'ResponseHeadersPolicyList.Quantity'
aws cloudfront list-origin-request-policies --type custom --output json \
--query 'OriginRequestPolicyList.Quantity'
```
An orphan is harmless and self-healing — the next run finds it by name, matches
it and attaches it — **so do not delete it by hand.**
⚠️ **THE EXPECTED VALUES DIFFER BY WHICH FAILURE YOU ARE RECOVERING FROM.** After
**attempt 1** both returned `0` and nothing needed doing. After **attempt 2**
they return **`1` and `0`** — the response-headers policy is the orphan recorded
below, and `1` is the correct reading, not a second problem
`[verified 2026-09-04]`.
**Two changes came out of it.** The clone now **omits** any empty member at
either of those two levels — a `ResponseHeadersPolicyConfig` member, or a
`SecurityHeadersConfig` member — and the dry run **asserts** that the generated
config carries no empty object at any *other* level, naming the dotted path if
it does. The asymmetry is deliberate: those two levels are where AWS is known to
synthesise a placeholder, and anywhere else is unaccounted for and stops the
run rather than being discarded quietly. The assertion runs before every branch, so **the dry run now catches this
class** rather than an `--apply` discovering it — and if it ever does fire it
**skips section 4** rather than throwing, so `router.js` can still be
re-applied.
**The proof is a command rather than a session**, which is the point of
`infra/cloudfront/policy-shapes.mjs` existing as its own module — `configure.mjs`
reads argv and calls AWS at import time, so the two functions could not
otherwise be reached:
```bash
node infra/cloudfront/policy-shapes.test.mjs
```
**Expect** `policy-shapes: 53 of 53 cases pass`, exit 0. Its first case is this
incident verbatim — the live `SecurityHeadersConfig`, empty
`ContentSecurityPolicy` and all. The same suite covers attempt 2's limit checks,
so this is the only command in this Part that proves both.
---
**ATTEMPT 2 — section 4 SUCCEEDED, section 5 FAILED, and the run left an orphan.**
With the clone fixed, `create-response-headers-policy` created
**`adr-sml-pdf-noindex` = `51c4e79b-d9c6-4c6f-907c-dbb0e73dd374`**. Section 5
then failed:
```
An error occurred (InvalidArgument) when calling the CreateOriginRequestPolicy
operation: The parameter Comment is too big
```
Its `Comment` was **182 characters** against a **128** cap. `update-distribution`
never ran, so the distribution is untouched — **but the account now holds a
response-headers policy that no behaviour references.**
🛑 **DO NOT DELETE THAT POLICY BY HAND.** This is the orphan case Part 3 and §7
predicted before it happened, and the recovery is **measured, not asserted**
`[measured 2026-09-04, dry run, exit 0, nothing written]`: the next run finds it
**by name**, matches it on every reconciled field, and stages it for attachment —
```
· response-headers policy adr-sml-pdf-noindex exists and matches the default behaviour
+ CacheBehaviors += *.pdf -> …, adr-sml-pdf-noindex (51c4e79b-d9c6-4c6f-907c-dbb0e73dd374)
```
— with the **create line gone** and the change count down from 4 to 3. **No
duplicate and no name collision.** (A collision would not be silent either: a
duplicate name returns `ResponseHeadersPolicyAlreadyExists`, which is a
different error from the `InvalidArgument` above. The script never reaches it,
because it looks the policy up by name first.)
⚠️ **`Comment` IS NOT RECONCILED**, so `51c4e79b` keeps its original
118-character text while the script now carries a shorter one. Deliberate:
adding `Comment` to the drift check would throw on this very policy and block
the run that attaches it.
**WHY NOTHING CAUGHT IT LOCALLY, AND THIS IS THE GENERAL LESSON.** Measured
2026-09-04: **`botocore/validate.py` checks neither `max` nor `pattern`** — its
`range_check()` reads only `min`, and the word `pattern` does not appear in the
file. And the 128 cap is not modelled as a constraint at all: on both policy
configs `Comment` is a bare `string`, and the number lives in the shape's
**`documentation` prose**. So **no client-side validation of any kind stood
between that 182-character string and the CloudFront API**, which is exactly why
the pre-flight below had to be built rather than relied upon.
**BOTH POLICY COMMENTS ARE NOW UNDER 80 CHARACTERS**, and the dry run enforces
the limits it knows about — `infra/cloudfront/policy-shapes.mjs`,
`PAYLOAD_LIMITS`, one entry per limit with the source it came from. **Every
entry with a cited AWS source is enforced by the service and by nothing local**
— see below. Two entries are stamped `[assumed]` and are not: no AWS source
states a policy **name** length, and those two constrain nothing this script
sends (our names are 19 and 26 characters). A breach **skips its section** rather than throwing, and — the part that
matters — **skips the whole section**, so a policy that is not created is never
staged for attachment. The suite named earlier in this Part
(`node infra/cloudfront/policy-shapes.test.mjs`) covers both the 182-character
`Comment` that failed here and the 118-character one that did not.
---
**One of the `·` lines carries a number worth reading**, and it is not a
warning:
```
· cloning 5 defined security header(s); omitting 1 the source does not define (ContentSecurityPolicy)
```
**Five is the number to read.** It is the count of security headers the PDF
policy will carry, and the verification block at the end of this Part counts the
same five arriving on the live PDF. A drop in this number is a partial clone
announcing itself one step earlier than that `curl` would.
⚠️ **RUN IT WITHOUT `--function-arn` ONLY IF THE ROUTER IS ALREADY ATTACHED.**
Omitting the flag prints `· no --function-arn given, leaving FunctionAssociations
alone` and skips change 1 — which is right on a re-run and wrong on a first one,
and the two look identical in a count.
🛑 **SECTION 5 IS THE ONLY ONE THAT REPLACES SOMETHING, AND WHAT IT REPLACES IS
ON THE INTAKE FORM'S PATH.** Sections 14 add. Section 5 swaps the origin request
policy on `/api/*` from `Managed-AllViewerExceptHostHeader` to a **whitelist** of
five headers, because AWS has no behaviour meaning "all viewer headers except
Host, plus a CloudFront header" — `allExcept` can only subtract, and
`allViewerAndWhitelistCloudFront` drags `Host` along and 403s at API Gateway.
Whitelisting is therefore forced, and the cost is that **a header missing from
that list is a header the handler never sees.** The list is the handler's four
`headerOf` reads plus `CloudFront-Viewer-Address`. The check prints the names,
so they can be compared to the whitelist rather than counted:
```bash
grep -o "headerOf(event, '[a-z-]*'" backend/intake/handler.mjs \
| sed "s/.*'\(.*\)'/\1/" | sort
```
**Expect** exactly `content-type`, `origin`, `referer`, `user-agent`.
⚠️ **`grep -n "headerOf(event"` WAS PRESCRIBED HERE AND RETURNS FIVE** — it
matches `function headerOf(event, name)`, the definition itself — so an operator
comparing it against a documented "four" concludes the handler grew a read.
**The failure mode is not an error.** Every submission would validate short and
redirect to `/contact/could-not-send/` — a real inquirer would read it as their
own browser misbehaving, and nothing would appear in a log as a fault. So the
block below is **not optional after an `--apply` that includes change 8**, and there are
**three** of them. The first is Part 7.1's probe with its output read differently
— **not "unchanged", which this said for one round**: §7.1 pipes into `head -12`
and reads the status by eye, while these read curl's own exit status and count
the `location` separately.
**Run all three, in this order, and each answers a different question:**
| # | probe | what only it can tell you |
|---|---|---|
| 1 | `Origin` + body | `Origin` is still forwarded — a **403** means it is not |
| 2 | `Referer`, no `Origin` | the Firefox fallback still works — nothing else tests it |
| 3 | honeypot value | the **body parsed** — probes 1 and 2 return the same 303 whether it did or not |
**PROBE 1 — is `Origin` still forwarded?**
```bash
curl -si -X POST "$SITE/api/intake" \
-H 'Origin: https://adr.smlcompany.ca' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data 'probe=1' -o /tmp/api.h
echo "curl_exit=$?" # curl's OWN status, on its own line
head -1 /tmp/api.h
grep -ic '^location: .*could-not-send' /tmp/api.h
```
**Expect** `curl_exit=0`, `HTTP/2 303`, and `1`. A **403** here means the
`Origin` header is no longer reaching the handler — i.e. the whitelist dropped
it — and the form is broken for everyone.
**PROBE 2 — the `Referer` fallback, which nothing else tests.** The handler
accepts `Referer` when `Origin` is absent (Firefox omits `Origin` on some
same-origin form navigations), so a whitelist that forwarded `Origin` and dropped
`Referer` passes probe 1 and fails for exactly those users:
```bash
curl -si -X POST "$SITE/api/intake" \
-H 'Referer: https://adr.smlcompany.ca/contact/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data 'company_website=probe' -o /tmp/api3.h
echo "curl_exit=$?"
head -1 /tmp/api3.h
grep -ic '^location: .*contact/received' /tmp/api3.h
```
**Expect** `curl_exit=0`, `303` and `1` `[verified against production 2026-09-04
— it returns 303 today, on the managed policy]`. A **403** means `Referer` is not
being forwarded.
🛑 **PROBE 3, AND NEITHER OF THE FIRST TWO CAN REPLACE IT: THEY CANNOT FAIL IN THE
INTERESTING DIRECTION.** `303 →
could-not-send` is what the handler returns **both** when it parsed the body and
found an empty submission **and** when `parseBody` threw because
`Content-Type` never arrived. Two opposite outcomes, one status, one location —
so a dropped `Content-Type` reads as a pass. This probe separates them, and
**writes nothing and sends nothing**:
```bash
curl -si -X POST "$SITE/api/intake" \
-H 'Origin: https://adr.smlcompany.ca' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data 'company_website=probe' -o /tmp/api2.h
echo "curl_exit=$?"
head -1 /tmp/api2.h
grep -ic '^location: .*contact/received' /tmp/api2.h
```
**Expect** `curl_exit=0`, `HTTP/2 303`, and `1` — location
`/contact/received/`, **not** `could-not-send`. That is the honeypot branch: it
is reached **only if the body parsed**, and it returns before validation, before
any DynamoDB write and before any SES send, so it leaves no record and sends no
email. `could-not-send` here means the body did not parse — `Content-Type` is
missing from the whitelist. **Roll back.**
⚠️ **IT DEPENDS ON THE HONEYPOT'S NAME** (`company_website`, `fields.mjs`). If
that is ever renamed, this probe degrades to the `could-not-send` branch — which
reads as a failure and starts an investigation, not as a pass. That direction is
the safe one; keep it that way if you change the probe.
**ROLLBACK, and it is one field.** Do not debug a broken intake form in place:
```bash
# ⚠️ THIS RETURNS THE ID THE BEHAVIOUR HAS NOW — which, if change 8 applied, is
# the whitelist you are rolling back FROM, not the value to restore. The value to
# restore is the managed id on the line below. Run this to confirm which state
# you are in, then PUT the managed id back with
# update-distribution --if-match. ⚠️ NOT by re-running configure.mjs: section 5
# converges FORWARD and cannot tell a deliberate revert from a first run — the
# two are byte-identical in the config — so --apply would re-attach the
# whitelist and put the form back in the state you are rolling back from.
aws cloudfront get-distribution-config --id "$DIST_ID" \
--query 'DistributionConfig.CacheBehaviors.Items[?PathPattern==`/api/*`].OriginRequestPolicyId'
# Managed-AllViewerExceptHostHeader = b689b0a8-53d0-40ab-baf2-68738e2966ac
```
Set that behaviour's `OriginRequestPolicyId` back to
`b689b0a8-53d0-40ab-baf2-68738e2966ac` and `update-distribution` with the current
ETag. `configure.mjs` prints the same id on the line it changes, prefixed ``, at
the moment it changes it.
⚠️ **THE HANDLER STILL STORES THE EDGE ADDRESS AFTER THIS.** Forwarding the
header does not change what is recorded, and it must not be made to as a
follow-up edit: what the record holds is published field by field on
`/legal/privacy/`, so storing `CloudFront-Viewer-Address` is a **disclosure**
change governed by §7.2's decision table, not a code tidy. Pouya's ruling of
2026-09-04 is *measured, not yet acted on*.
⚠️ **AND `adr-sml-pdf-noindex` IS RECONCILED ON EVERY RUN, NOT ONLY CREATED.** A
response-headers policy **replaces** rather than merges, so the PDF policy has to
carry everything the default behaviour's policy carries. If they have diverged —
someone adds the `Content-Security-Policy` or `Permissions-Policy` that
`docs/05` specifies to one and not the other — the script **throws and names the
diff** rather than passing. That is deliberate: the failure it guards is the PDF
being served different headers from the pages, which is silent.
```bash
node infra/cloudfront/configure.mjs --dist "$DIST_ID" --api-domain "$API_DOMAIN" \
--function-arn "$ROUTER_ARN" --apply
aws cloudfront wait distribution-deployed --id "$DIST_ID"
echo "deployed: $?"
```
**Verify the config landed:**
```bash
aws cloudfront get-distribution-config --id "$DIST_ID" \
--query 'DistributionConfig.{Fn:DefaultCacheBehavior.FunctionAssociations.Items[].EventType,Err:CustomErrorResponses.Items[].{Code:ErrorCode,Page:ResponsePagePath,Status:ResponseCode},Beh:CacheBehaviors.Items[].{P:PathPattern,O:TargetOriginId,RHP:ResponseHeadersPolicyId,Fn2:FunctionAssociations.Items[].EventType,Methods:AllowedMethods.Items},Origins:Origins.Items[].Id}'
```
**Expect:** `Fn: ["viewer-request"]`; one error response `404 → /404.html → 404`;
**two** behaviours — `/api/*` → `intake-api`, POST in its method list, **no
`RHP` and `Fn2: null`** (the association is withheld there deliberately: a 301
would turn the form's POST into a GET and drop the body), and `*.pdf` → the S3
origin **with an `RHP` id and `Fn2: ["viewer-request"]`**; two origins.
⚠️ **THAT QUERY DOES NOT PROJECT `OriginRequestPolicyId`, SO IT CANNOT SEE
CHANGE 8.** Read it separately rather than concluding anything from its absence:
```bash
aws cloudfront get-distribution-config --id "$DIST_ID" \
--query 'DistributionConfig.CacheBehaviors.Items[].{P:PathPattern,ORP:OriginRequestPolicyId}'
```
**Expect** `/api/*` carrying the **`adr-sml-api-viewer-address`** id — *not*
`b689b0a8-53d0-40ab-baf2-68738e2966ac`, which is the managed policy it replaced
and is what a rollback restores.
**Then verify the header actually arrives, because the config landing is not the
same fact:**
```bash
curl -D /tmp/pdf.h -o /dev/null "$SITE/pouya-lajevardi-bio.pdf"
echo "curl_exit=$?" # curl's OWN status, on its own line
for h in x-robots-tag strict-transport-security x-content-type-options \
x-frame-options x-xss-protection referrer-policy; do
printf '%-28s %s\n' "$h" "$(grep -ic "^$h:" /tmp/pdf.h)"
done
```
**Expect** `curl_exit=0` and **`1` against every one of the six** — the five
security headers *and* `x-robots-tag`.
⚠️ **THE SHAPE OF THIS BLOCK IS THE POINT, and its first version got all three
wrong.** It piped `curl -sI` into one `grep -E` with six alternatives and read
`$?`. That reports **grep's** status, not curl's, so a DNS failure, a TLS failure
and a 5xx all read as `exit=1` — indistinguishable from "the headers are
missing", with `-s` deleting the message that would have told them apart. And an
alternation exits **0 if ANY ONE** matches, so `exit=0` would not have meant the
five arrived, which is the only regression the block exists to catch. Counting
each header separately is what makes a partial clone visible. (`CLAUDE.md`: never
suppress stderr, never read a pipeline's status as its first command's, and a
uniform pass is the result that ends a check rather than starting one.)
⚠️ **If any of the five security headers reads `0`, the policy did not clone them
and the PDF has LOST headers it had before this change.**
---
## Part 4 — Verify the distribution before the site exists
Run this now, against the **old** bucket contents. It is the check that the
plumbing works independently of the deploy.
```bash
for p in / /index.html /about/ /nope-not-a-page/; do
printf '%-22s ' "$p"
curl -s -o /dev/null -w 'status=%{http_code} ct=%{content_type}\n' "${SITE}${p}"
done
printf 'no-slash redirect: '; curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' "${SITE}/about"
```
**Expect:** `/` and `/index.html` → **200 text/html** (the old page still
serves — the router did not break it); `/about/` and `/nope-not-a-page/` → **404**
rather than the 403 they returned before, because Part 1 changed S3's answer and
Part 3 mapped it; `/about` → **301 → `https://adr.smlcompany.ca/about/`**.
The body of those 404s is not the 404 page yet — `404.html` is not in the bucket
until Part 8. **The status is what Part 4 proves; the page is what Part 8.4
proves.** Do not tick `docs/06`'s "404 returns a 404 status" item here; tick it
after 8.4, when both halves are true at once.
---
## Part 5 — Deploy the handler
### 5.1 Package
🛑 **THREE FILES SINCE 2026-09-04, AND THE ZIP FOLLOWS NO IMPORT.**
`handler.mjs` imports both `./fields.mjs` and `./spam-score.mjs`; a zip missing
either fails at cold start with `Runtime.ImportModuleError` and every submission
then 500s. **The list is now derived from the directory** — `ls *.mjs` minus the
tests — in this step and in 5.5, so a new module is packaged without editing
anything. It was typed out in both until 2026-09-04, and this banner still said
so, fifteen lines above the paragraph that says otherwise.
```bash
rm -f /tmp/intake.zip
(cd backend/intake \
&& echo "packaging: $(ls *.mjs | grep -v '\.test\.' | tr '\n' ' ')" \
&& zip -q -X /tmp/intake.zip $(ls *.mjs | grep -v '\.test\.'))
unzip -l /tmp/intake.zip
```
⚠️ **THE LIST IS SUBSTITUTED DIRECTLY, NOT HELD IN A VARIABLE, AND THAT IS NOT
STYLE.** A first version read `MODULES=$(ls …)` then `zip … $MODULES`. **In zsh
that packages ONE file whose name is all three joined by newlines** — zsh does
not word-split parameter expansions, only command substitutions — so it fails on
the shell this project is actually operated from while working in bash.
`CLAUDE.md` names this trap; it was reintroduced here and caught by running the
block in both shells rather than by reading it.
**Expect:** exactly three entries — `handler.mjs`, `fields.mjs`,
`spam-score.mjs` — **42,604 bytes uncompressed and 18,462 zipped**
`[measured 2026-09-04]`. All three at the zip root: the imports are `./`-relative,
so a nested directory breaks them at cold start. *(This read "two entries, ≈ 25.7
KB / ≈ 10.8 KB" `[measured 2026-09-01]`, before the scorer existed.)*
⚠️ **`spam-score.test.mjs` IS NOT IN THE ZIP AND MUST NOT BE.** Run it at a
keyboard — `node backend/intake/spam-score.test.mjs`, **39 of 39** — before
packaging. It is the only check on the scorer, whose failure mode is labelling
real inquiries rather than throwing.
### 5.2 Configuration first, code second
The handler calls `requireEnv()` at **module scope** and throws at cold start on
any of six missing variables — deliberately, so a misconfigured function cannot
accept a submission it will not store. Set the configuration before the code, so
there is never a moment when the new code runs against no environment.
```bash
ENV_JSON=$(node scripts/intake-env.mjs --table "$TABLE" --notify "$NOTIFY_TO" --from "$MAIL_FROM")
echo "$ENV_JSON" | python3 -m json.tool
```
**Expect** six variables. ⚠️ **`RESPONSE_TIME` and `NO_RETAINER_NOTICE` are read
out of `src/data/site.ts` and must never be typed here.** Both are published
commitments — the two-business-day response (§4, Q27) and the no-retainer notice
`docs/01` §`/contact/` requires, **including its fourth clause about not itself
creating a conflict check**, which a hand-typed copy inside the handler had
already dropped once. The notice also contains an en dash in "mediatorparty",
which is exactly the character a retype loses. `scripts/intake-env.mjs` asserts
both and exits non-zero rather than emitting a softened commitment.
```bash
aws lambda update-function-configuration --function-name "$FN" \
--handler handler.handler --timeout 15 --memory-size 512 --environment "$ENV_JSON"
aws lambda wait function-updated --function-name "$FN"
aws lambda get-function-configuration --function-name "$FN" \
--query '{Handler:Handler,Timeout:Timeout,Memory:MemorySize,Vars:sort(keys(Environment.Variables)),State:State,Update:LastUpdateStatus}'
```
**Expect:** `Handler: handler.handler` — the file is `handler.mjs` and the export
is `handler`, where the hand-built function was `index.handler`; `Timeout: 15`;
`Memory: 512`; the six names sorted; `State: Active`; `Update: Successful`.
Timeout 10 → 15 s and memory 128 → 512 MB are changes, not restatements. The
handler does a DynamoDB write and two SES sends per invocation, and 128 MB is
where an AWS SDK cold start is slowest — more memory shortens the billed duration
enough that it is usually the cheaper setting, not the dearer one.
### 5.3 Code
```bash
aws lambda update-function-code --function-name "$FN" --zip-file fileb:///tmp/intake.zip
aws lambda wait function-updated --function-name "$FN"
aws lambda get-function-configuration --function-name "$FN" \
--query '{CodeSize:CodeSize,Runtime:Runtime,Update:LastUpdateStatus,Modified:LastModified}'
```
**Expect** `Update: Successful`, and a `CodeSize` that says **which path you
took** — it is the ZIP, not the source:
| path | expected `CodeSize` |
|---|---|
| 5.1's plain three-file zip | **≈ 18,462** |
| 5.5's bundled variant | **low single-digit MB** — it was **3,307,021** on 2026-09-02 `[measured 2026-09-04 — get-function-configuration]` |
🛑 **5.5 IS THE PATH THAT WAS ACTUALLY TAKEN AT CUTOVER.** The live function
carries the bundled zip, so **a redeploy that runs 5.1 and stops would replace it
with an unbundled one and reintroduce the `Runtime.ImportModuleError` 5.5 exists
to fix.** Run 5.4 after 5.3, every time, and follow it to 5.5 if it fires.
*(This line said "≈ 10,800", and before that "around 23,000" — 5.1's uncompressed
figure applied to a different quantity. Both were written against the unbundled
path, which is not the one in production.)*
### 5.4 Prove it loads, without writing anything
```bash
cat > /tmp/probe-no-origin.json <<'JSON'
{"version":"2.0","routeKey":"POST /api/intake","rawPath":"/api/intake",
"headers":{"content-type":"application/x-www-form-urlencoded"},
"requestContext":{"http":{"method":"POST","path":"/api/intake","sourceIp":"127.0.0.1"}},
"body":"probe=1","isBase64Encoded":false}
JSON
aws lambda invoke --function-name "$FN" --cli-binary-format raw-in-base64-out \
--payload file:///tmp/probe-no-origin.json /tmp/probe-out.json
cat /tmp/probe-out.json; echo
```
**Expect:** the invoke prints `"StatusCode": 200` with **no `FunctionError`**, and
`/tmp/probe-out.json` is
```json
{"statusCode":403,"headers":{"Cache-Control":"no-store"},"body":""}
```
That 403 is the handler's `Origin` check refusing a request with neither `Origin`
nor `Referer`. It happens **before** any DynamoDB write and before any email,
which is what makes this probe safe to run against production.
**Two other outcomes, and each has one cause:**
- `"FunctionError": "Unhandled"` with `Runtime.ImportModuleError` in the body —
the `nodejs24.x` runtime does not provide the AWS SDK v3 clients this handler
imports. Run 5.5, then repeat 5.4.
- A body naming `intake handler: <NAME> is not set` — that variable did not
reach the function. Re-run 5.2 and read the `Vars` list.
### 5.5 Bundled variant — only if 5.4 said `Runtime.ImportModuleError`
Versions are resolved from the registry at run time rather than pinned in this
file: `CLAUDE.md`'s rule is that a version is checked against the registry and
never recalled, and a literal here would be stale the week after it was written.
⚠️ **THIS PATH WAS TAKEN — 2026-09-02, and the live function is the bundled
zip.** ⚠️ **THE TWO RESOLVED VERSIONS AND THE COMMIT THE DEPLOYED SOURCES MATCH
ARE IN `AGENTS.md` §7 AND ARE DELIBERATELY NOT REPEATED HERE.** They were
repeated here until 2026-09-04, and the paragraph directly above is the argument
against it — **the literals went stale in two days rather than a week**: the
2026-09-04 redeploy moved both packages one patch and added a third source file,
and this copy still named the old versions and a superseded commit while reading
as a measurement. Read §7's two Lambda rows; this step's duty is to **update**
them, not to mirror them.
✅ **THE `cp` AND `zip` LINES BELOW DERIVE THE FILE LIST THE SAME WAY 5.1 DOES.**
They were a second hand-typed copy until 2026-09-04, not derived from 5.1's and
with nothing checking that the two agreed — so a module added to one and not the
other would ship from whichever path the operator happened to take. Both now read
the directory.
```bash
rm -rf /tmp/intake-bundle && mkdir -p /tmp/intake-bundle
echo "bundling: $(cd backend/intake && ls *.mjs | grep -v '\.test\.' | tr '\n' ' ')"
(cd backend/intake && cp $(ls *.mjs | grep -v '\.test\.') /tmp/intake-bundle/)
# ⚠️ ASSERT THE COPY LANDED. A glob that matches nothing makes `cp` fail, `zip`
# succeed on an empty set, and `update-function-code` upload a bundle with no
# handler — a silent failure that only shows up as 5xx on the live form.
test -f /tmp/intake-bundle/handler.mjs || { echo "FATAL: handler.mjs not copied"; exit 1; }
echo "copied: $(ls /tmp/intake-bundle/*.mjs | wc -l | tr -d ' ') module(s)"
( cd /tmp/intake-bundle \
&& npm init -y > /dev/null \
&& npm install --omit=dev --no-audit --no-fund \
"@aws-sdk/client-dynamodb@$(npm view @aws-sdk/client-dynamodb version)" \
"@aws-sdk/client-sesv2@$(npm view @aws-sdk/client-sesv2 version)" )
rm -f /tmp/intake.zip
( cd /tmp/intake-bundle && zip -qr -X /tmp/intake.zip $(ls *.mjs) node_modules package.json )
unzip -l /tmp/intake.zip | tail -1
aws lambda update-function-code --function-name "$FN" --zip-file fileb:///tmp/intake.zip
aws lambda wait function-updated --function-name "$FN"
```
**Expect** a zip in the low single-digit megabytes, well under the 50 MB direct-
upload limit. **If this path is taken, say so in the `AGENTS.md` Change Log and
add both packages to §7** — they become pins this project maintains, and `R11`
covers them from that moment.
---
## Part 6 — Route the API at `/api/intake`
### 6.1 The invoke permission, before the route
⚠️ **THE EXISTING PERMISSION DOES NOT COVER THE NEW ROUTE, AND THIS IS THE STEP
WHOSE OMISSION IS HARDEST TO DIAGNOSE.** Part 0.3 records the one statement on the
function: `SourceArn` `…/4tl0m5igkj/*/*/submissions`. Add a route at
`/api/intake` without adding a permission for it and API Gateway is refused
permission to invoke — the caller gets a **500**, the Lambda logs nothing at all
because it was never entered, and the only trace is an API Gateway metric.
```bash
aws lambda add-permission --function-name "$FN" \
--statement-id apigw-post-api-intake \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:${AWS_REGION}:${ACCT}:${API_ID}/*/POST/api/intake"
aws lambda get-policy --function-name "$FN" --query Policy --output text | python3 -m json.tool
```
**Expect:** two statements — the old one scoped to `…/submissions` and the new one
to `…/POST/api/intake`.
### 6.2 The route
```bash
export INTEG_ID=$(aws apigatewayv2 get-integrations --api-id "$API_ID" \
--query "Items[?IntegrationUri=='arn:aws:lambda:${AWS_REGION}:${ACCT}:function:${FN}'].IntegrationId | [0]" \
--output text)
echo "INTEG_ID=$INTEG_ID"
[ -n "$INTEG_ID" ] && [ "$INTEG_ID" != "None" ] || echo "STOP — no integration points at $FN"
aws apigatewayv2 create-route --api-id "$API_ID" \
--route-key 'POST /api/intake' --target "integrations/${INTEG_ID}" \
--query '{RouteId:RouteId,RouteKey:RouteKey,Target:Target}'
aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].{Key:RouteKey,Target:Target}'
```
**Expect:** `INTEG_ID` non-empty and not `None`; then two routes,
`POST /submissions` and `POST /api/intake`, both on the same integration. The
stage is `$default` with auto-deploy on, so the route is live immediately — there
is no deployment to create.
### 6.3 Throttling
⚠️ **THIS IS NOT THE PER-IP LIMIT `docs/05` ASKS FOR, AND IT CANNOT BE.** That
spec says "Rate limit by source IP at API Gateway: 5 requests / 5 minutes".
**API Gateway throttling is aggregate — per route and per stage — not per source
IP.** Per-IP rate limiting needs AWS WAF with a rate-based rule on the
distribution, which is a paid service and therefore Pouya's decision, recorded on
`docs/06`'s checklist as explicitly **not** a launch blocker. What is set here is
a real control over total volume; describing it as per-IP would be the Q22 shape
again — a documented control that does not exist.
```bash
aws apigatewayv2 update-stage --api-id "$API_ID" --stage-name '$default' \
--route-settings '{"POST /api/intake":{"ThrottlingRateLimit":1,"ThrottlingBurstLimit":5,"DetailedMetricsEnabled":true}}'
aws apigatewayv2 get-stage --api-id "$API_ID" --stage-name '$default' \
--query 'RouteSettings'
```
**Expect:** the `POST /api/intake` entry with rate 1/s, burst 5, detailed metrics
on. One per second sustained with a burst of five is far above any human filling
this form and far below anything worth paying for.
### 6.4 Retire the old route
Nothing posts to `/submissions` any more — the form posts to `/api/intake` and
the handler that answered the old contract is gone. An unused public route on the
same function is surface with no purpose.
```bash
export OLD_ROUTE=$(aws apigatewayv2 get-routes --api-id "$API_ID" \
--query "Items[?RouteKey=='POST /submissions'].RouteId | [0]" --output text)
echo "OLD_ROUTE=$OLD_ROUTE"
aws apigatewayv2 delete-route --api-id "$API_ID" --route-id "$OLD_ROUTE"
aws lambda remove-permission --function-name "$FN" \
--statement-id 70aeb597-e4d8-5141-8fb1-6b6190f7b2ab
aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].RouteKey'
aws lambda get-policy --function-name "$FN" --query Policy --output text | python3 -m json.tool
```
**Expect:** one route, `POST /api/intake`; one permission statement, scoped to it.
Confirm the statement id in the `remove-permission` call against the policy you
printed in 6.1 before running it — it is an id, not a name, and it will differ if
the permission was ever rebuilt.
> **The `execute-api` hostname stays reachable**, so `/api/intake` can be reached
> without passing through CloudFront. That is not a new weakness and not a
> loose end: `DisableExecuteApiEndpoint` cannot be turned on, because the
> distribution's own origin **is** that hostname. The `Origin` check is a CSRF
> control rather than authentication either way, and 6.3's throttle applies at
> the stage, so it covers both paths.
---
## Part 7 — End to end, with a real submission
### 7.1 The route answers
```bash
curl -si -X POST "${SITE}/api/intake" \
-H 'Origin: https://adr.smlcompany.ca' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data 'deploy-route-probe=1' | head -12
```
**Expect:** `HTTP/2 303` and `location: https://adr.smlcompany.ca/contact/could-not-send/`.
The handler accepted the request, validated it, found an empty submission and
redirected to the failure page — **before** any write and any email. This is the
same probe `scripts/deploy-local.sh` runs at the end of every deploy.
⚠️ **404 IS AMBIGUOUS AND THE FIRST DRAFT OF THIS LINE WAS NOT.** It means
**either** the `/api/*` behaviour is missing (Part 3) **or** the `POST
/api/intake` route is missing or misspelled (Part 6.2) — two different repairs.
And because Part 3's custom error response is **distribution-wide**, a 404 from
API Gateway is served as the styled `/404.html` body, so the one string that would
have named the cause — API Gateway's `{"message":"Not Found"}` — is replaced before
you see it. **Check the route first; it is one command:**
`aws apigatewayv2 get-routes --api-id "$API_ID" --query 'Items[].RouteKey'`.
**403** means the `Origin` header did not arrive, and **as of 2026-09-04 there
are two policies it could be** — read which one the behaviour carries before
repairing:
- **`adr-sml-api-viewer-address`** (Part 3, change 8) — a **whitelist**. If
`Origin` is missing from its Headers list, or the list drifted, every real
submission 403s. Roll back by PUTting the managed id below with
`update-distribution --if-match` — **not** by re-running `configure.mjs`,
which converges forward and would re-attach the whitelist.
- **`Managed-AllViewerExceptHostHeader`** (`b689b0a8-53d0-40ab-baf2-68738e2966ac`)
— what it replaced, and what a rollback restores.
Either way, a policy that drops `Origin` turns every real submission into a 403.
**500** means Part 6.1 was skipped.
### 7.2 A real submission, from the real form
Do this in a browser at `${SITE}/contact/` **after Part 8**, because the form
does not exist until the site is deployed. Fill it as an inquirer would, with
`Your name: Cutover test <date>` so the record is identifiable, and a real
address you can read.
**Expect:** the browser lands on `/contact/received/`, and two emails arrive — the
notification at `$NOTIFY_TO`, replyable to the address you entered, and the
confirmation at that address. Read the confirmation and check three things: the
response-time sentence, the **four**-clause no-retainer notice, and that the
field summary uses readable labels ("Your name", "Subject matter") rather than
field names.
Then find the record. The notification email prints `submissionId <uuid>` —
that value **is** the partition key, so it can be used directly:
```bash
export SUB_ID='<the uuid from the notification email>'
aws dynamodb get-item --table-name "$TABLE" \
--key "{\"submissionId\":{\"S\":\"${SUB_ID}\"}}" \
--query 'Item.{id:submissionId.S,at:submittedAt.S,ttl:ttl.N,name:name.S,area:practiceArea.S,sourceIp:sourceIp.S}'
curl -s https://checkip.amazonaws.com
```
⚠️ **READ `sourceIp` AGAINST YOUR OWN ADDRESS — THIS IS THE ONLY PLACE THE PROXY
CHAIN GETS MEASURED, AND IT HAS ALREADY BEEN REASONED ABOUT WRONGLY TWICE.** The
handler stores `requestContext.http.sourceIp`, which is the TCP peer. Behind the
`/api/*` behaviour that peer is a **CloudFront edge**, so the expectation is an
AWS address, not yours. Three outcomes and each has a different consequence:
| what `sourceIp` holds | what it means |
|---|---|
| an **AWS** address (not the `checkip` value) | As designed. The field records the CDN, so it **cannot serve abuse investigation**, and `/legal/privacy/`'s *"your IP address"* is inaccurate — fold it into the Q62 edit on the same page rather than leaving two wrong sentences there |
| **your** address, matching `checkip` | Better than expected, and worth knowing before anyone relies on it. Do not conclude it is trustworthy: verify it is not simply echoing a header by resubmitting with `-H 'X-Forwarded-For: 8.8.8.8'` and confirming `8.8.8.8` is **not** what lands. ⚠️ **AND CORRECT `/legal/privacy/` §What is collected**, whose network-address paragraph says the address is *"normally the network's own rather than your connection's"* — wrong in this branch, and it understates what is held about the reader. This row carried **no instruction at all** until 2026-09-02, so two of these three outcomes had nothing reconciling the page with the measurement (`adversarial-reviewer`, round 2) |
| `8.8.8.8` after that resubmission | **Stop.** The field is client-controlled and a record can be made to name an uninvolved third party. Revert to storing nothing rather than storing that |
⚠️ **AN EARLIER REVISION OF THE HANDLER READ `x-forwarded-for` HERE, AND THAT WAS
THE THIRD OUTCOME.** CloudFront **appends** the viewer address to a
client-supplied XFF rather than replacing it, so the leftmost entry is whatever
the client sent. The fix, if a usable value is wanted, is a **custom** origin
request policy on `/api/*` forwarding `CloudFront-Viewer-Address`, which
CloudFront generates and overwrites — not the managed
`AllViewerAndCloudFrontHeaders`, which forwards `Host` and would 403 every request
at API Gateway. ⚠️ **THAT CHANGE IS NOW IN THIS RUNBOOK — Part 3, change 8, written
2026-09-04 on Pouya's ruling and NOT YET APPLIED.** This paragraph said it was
*"deliberately not in this runbook: measure first"*, which was true until the
ruling and false afterwards. **Forwarding the header does not change what is
stored:** `viewerIp()` still records `requestContext.http.sourceIp`, and the
decision table above is still the procedure for changing that, because what the
record holds is published field by field on `/legal/privacy/`. Measure first
still governs the STORING, not the forwarding.
**Expect** the item, with `ttl` a 10-digit epoch-seconds value. Check it is 24
months out — read it, do not assume it:
```bash
python3 - <<PY
import datetime, os
ttl = int("${SUB_ID}" and os.popen(
'aws dynamodb get-item --table-name ${TABLE} '
'--key \'{"submissionId":{"S":"${SUB_ID}"}}\' --query Item.ttl.N --output text'
).read().strip())
d = datetime.datetime.fromtimestamp(ttl, datetime.timezone.utc)
now = datetime.datetime.now(datetime.timezone.utc)
months = (d.year - now.year) * 12 + (d.month - now.month)
print(f'ttl {ttl} -> {d.isoformat()} ({months} months from now)')
print('PASS' if months == 24 else 'FAIL — /legal/privacy/ promises 24 months')
PY
```
**Expect:** `24 months from now` and `PASS`. `/legal/privacy/` states the period
publicly, so a number that is not 24 is a false disclosure rather than a config
error.
⚠️ **DELETE THE TEST RECORD when you are done with it** — it is a real row in a
table whose contents are governed by a published privacy policy.
```bash
aws dynamodb delete-item --table-name "$TABLE" --key "{\"submissionId\":{\"S\":\"${SUB_ID}\"}}"
```
### 7.3 The four pre-existing items
The table held **4 items** before any of this `[verified 2026-09-01 —
describe-table ItemCount]`. They were written by the hand-built handler, whose
code is in `docs/reference/AWS-Hosting-Guide.md` Part 8.3 and **writes no `ttl`
attribute** — so DynamoDB will never expire them and they are retained
indefinitely, against a policy that says otherwise.
**This is Pouya's call and not a command in this runbook**, because it depends on
something no agent can determine: whether those four are the guide's own smoke
test (`Test User`) or real inquiries that arrived through the old site. Read them
in the console, then either delete the test rows or write a `ttl` onto the real
ones. Tracked on `docs/06`'s checklist.
---
## Part 8 — The first production deploy
⚠️ **DO NOT RUN 8.3 UNTIL POUYA HAS READ EVERY PAGE.** `docs/06`'s Content and
compliance group — the `claims-auditor` pass and his own page-by-page read — is
the gate on this part, and it is not a technical check.
### 8.1 Archive what is there
S3 versioning is `Enabled`, so pass 3's `--delete` is recoverable — but
`docs/06`'s post-cutover list says to archive the old build rather than rely on
that, and a local copy costs one command.
```bash
mkdir -p _archive/pre-cutover-$(date +%F)
aws s3 sync "s3://${BUCKET}" "_archive/pre-cutover-$(date +%F)/" --no-progress
ls -la "_archive/pre-cutover-$(date +%F)/"
find "_archive/pre-cutover-$(date +%F)" -type f | wc -l
```
**Expect:** 3 files — `index.html` at 2,206,032 bytes and two PNGs under
`assets/`. Do not commit them; `docs/06` wants them kept, not versioned.
### 8.2 Dry run — the three passes, in order, writing nothing
```bash
npm run check && npm run build && npm run check:claims && npm run og:proof && npm run check:intake
echo "gates exit=$?"
npm run lighthouse; echo "lighthouse exit=$?"
```
**Expect:** every one exit 0. `lighthouse` is local-only and cannot run in CI; it
is a keyboard gate and this is the keyboard.
```bash
aws s3 sync ./dist "s3://${BUCKET}" --dryrun \
--exclude "*" --include "_astro/*" --include "fonts/*" \
--cache-control "public, max-age=31536000, immutable" --no-progress | tail -5
aws s3 sync ./dist "s3://${BUCKET}" --dryrun \
--exclude "*" --include "*.avif" --include "*.webp" --include "*.jpg" \
--include "*.png" --include "*.svg" \
--cache-control "public, max-age=604800" --no-progress | tail -5
aws s3 sync ./dist "s3://${BUCKET}" --dryrun \
--exclude "_astro/*" --exclude "fonts/*" \
--cache-control "public, max-age=0, must-revalidate" --delete --no-progress | tail -8
```
**Expect,** on the bucket state Part 0.3 records — measured 2026-09-01, so these
are counts rather than shapes: **51** `(dryrun) upload:` lines in pass 1, **56**
in pass 2, and **52** lines in pass 3 of which **two are
`(dryrun) delete:`** — `assets/sml-logo-full.png` and `assets/sml-logo-mark.png`,
the old site's two logos. `404.html` appears in pass 3. `index.html` is not
deleted; it is overwritten.
⚠️ **PASS 2's DRY RUN IS A SUPERSET OF WHAT PASS 2 ACTUALLY UPLOADS, AND THE REAL
RUN WILL PRINT FEWER LINES.** 34 of those 56 are files under `_astro/`, which
pass 1 uploads first. Each `--dryrun` above is compared against the bucket **as it
is now**, so it cannot see the uploads the preceding pass would have made; in the
real sequential run `aws s3 sync` skips them as already in sync. That skipping is
exactly what preserves pass 1's `immutable` header on those files rather than
overwriting it with pass 2's week-long one — `docs/06` §Cache policy says the same
thing about pass 3, and this is why the pass order is load-bearing. **Do not
reorder the passes to make this output tidier, and do not read the smaller real
counts as a failed upload.**
⚠️ **Check the largest file in `./dist` while you are here.** The deploy user has
no `s3:AbortMultipartUpload`, which is safe only while nothing approaches
`aws s3 sync`'s 8 MB multipart threshold (`docs/06`).
```bash
find ./dist -type f -exec stat -f '%z %N' {} + | sort -rn | head -3
```
**Expect** the largest well under 8,388,608 bytes — it was **171,876** on
2026-09-01. GNU `find -printf` is not available here; this is the BSD/macOS form,
and writing it as `find -printf … || find -exec …` would not have fallen back,
because the exit status of that pipeline belongs to `head`.
### 8.3 The deploy — on Pouya's word only
⚠️ **NOT AS `user/pouya`.** The script refuses that identity and §10 is why: the
account is shared with unrelated production systems. Use the `adr-sml-deploy`
key, which §7 records as provisioned.
```bash
AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… \
AWS_REGION="$AWS_REGION" S3_BUCKET="$BUCKET" CLOUDFRONT_DISTRIBUTION_ID="$DIST_ID" \
npm run deploy
```
The script runs `npm run check`, the build, `check:claims`, the three passes, the
`/*` invalidation, and then the intake route check. **Read its last line.** It
warns rather than fails on a bad intake route, because by then the site is
already published and failing the job would not un-publish it.
```bash
git tag "v$(date +%Y).1" && git push --tags
```
### 8.4 Verify the site, not the deploy
```bash
for p in / /about/ /mediation/ /arbitration/ /med-arb/ /practice/ /practice/construction/ \
/process/ /for-parties/ /fees/ /insights/ /contact/ /bio/ /legal/privacy/ \
/legal/terms/ /robots.txt /sitemap-index.xml /pouya-lajevardi-bio.pdf; do
printf '%-32s ' "$p"
curl -s -o /dev/null -w 'status=%{http_code} bytes=%{size_download}\n' "${SITE}${p}"
done
printf '%-32s ' "404 body"; curl -s "${SITE}/nope-not-a-page/" -o /tmp/404.html -w 'status=%{http_code}\n'
grep -c 'That page is not here' /tmp/404.html
printf '%-32s ' "cache header on HTML"; curl -sI "${SITE}/about/" | grep -i '^cache-control'
printf '%-32s ' "cache header on asset"
curl -s "${SITE}/about/" | grep -o '/_astro/[^"]*\.css' | head -1 \
| xargs -I{} curl -sI "${SITE}{}" | grep -i '^cache-control'
```
**Expect:** every page 200 with real bytes; the 404 path **404** *and* the phrase
`That page is not here` found once; `max-age=0, must-revalidate` on HTML;
`max-age=31536000, immutable` on the hashed CSS. Only now is `docs/06`'s "404
returns a 404 status" item true on both halves.
---
## Part 9 — Rollback, per part
Each of these is independent. None of them needs the others undone first.
**9.1 Part 1** — `aws s3api put-bucket-policy --bucket "$BUCKET" --policy "file:///tmp/bucket-policy.before.json"`.
Missing keys go back to 403 and the 404 mapping stops firing; nothing else changes.
**9.2 Parts 23** — re-run `configure.mjs` is *not* a rollback; it is idempotent
forward-only, **and that now matters most for change 8**: section 5 re-attaches
the `/api/*` whitelist on the next `--apply`, because a deliberately reverted
behaviour and a never-configured one are byte-identical in the config and no
detector can separate them. To undo, `get-distribution-config`, remove the
`FunctionAssociations` entry / the `404` custom error response / the `/api/*`
behaviour and the `intake-api` origin, and `update-distribution --if-match`.
**Sections 4 and 5 were added after this paragraph and undo the same way:**
put `/api/*`'s `OriginRequestPolicyId` back to
`b689b0a8-53d0-40ab-baf2-68738e2966ac` (the managed policy) and/or remove the
`*.pdf` behaviour, with `update-distribution --if-match`. The two custom policies
`adr-sml-api-viewer-address` and `adr-sml-pdf-noindex` can then be deleted with
`delete-origin-request-policy` / `delete-response-headers-policy`, each of which
**fails while still attached** — the same ordering feature as the function below.
⚠️ **Deleting the policies does not prevent re-attachment either** — the next
`--apply` simply creates them again by name and attaches them. Nothing in this
script can be made to remember a deliberate revert, because a reverted behaviour
and a never-configured one are byte-identical in the config. **The rollback holds
only until someone runs `configure.mjs --apply` again**; that is a property of a
forward-converging script, and the fix if it ever matters is a flag, not a
deletion. Then
`aws cloudfront delete-function --name adr-sml-router --if-match <etag>`, which
fails while the function is still associated — that ordering is a feature.
**9.3 Part 5** — the previous code is not recoverable from Lambda: the hand-built
function was edited in the console and its 1,527 bytes exist only in
`docs/reference/AWS-Hosting-Guide.md` Part 8.3. That is the rollback source, and
it answers a different contract (JSON in, `{"ok":true}` out) that the current
`/contact/` form does not speak. **Forward is the only real fix here** — which is
worth knowing before 5.3, not after.
**9.4 Part 6** — `delete-route` the new route, `remove-permission` the new
statement, and re-add the `/submissions` route and its permission if anything
still needs it. Nothing does.
**9.5 Part 8** — S3 versioning is `Enabled`. Restore the previous version of
`index.html`, `aws s3 rm` what the new build added, invalidate `/*`. Or
`git revert` and re-deploy, which is cleaner and is what the tag in 8.3 is for.
---
## Part 10 — Q60: prove TTL deletes, not just that it is enabled
`AGENTS.md` §7 records `TimeToLiveStatus: ENABLED` on attribute `ttl`
`[re-verified 2026-09-01]`, and **that proves the setting and not the behaviour**.
`/legal/privacy/` tells the public that a record is *"deleted automatically by the
database rather than by someone remembering to do it"* — an assertion about the
mechanism. Q60 closes when a record has been watched to disappear, and not
before. §12 R19 keeps it surfacing until then.
### 10.1 Write one record with a near-future `ttl`
Not through the form: the handler writes a 24-month `ttl` and waiting two years
is not a test. Write it directly, with a shape that cannot be mistaken for an
inquiry.
```bash
export TTL_ID="q60-ttl-probe-$(date +%Y%m%dT%H%M%SZ)"
export TTL_AT=$(python3 -c 'import time; print(int(time.time()) + 300)')
python3 -c "import datetime,os; print('ttl', os.environ['TTL_AT'], '->', datetime.datetime.fromtimestamp(int(os.environ['TTL_AT']), datetime.timezone.utc).isoformat())"
aws dynamodb put-item --table-name "$TABLE" --item "$(cat <<JSON
{"submissionId":{"S":"${TTL_ID}"},
"submittedAt":{"S":"$(date -u +%Y-%m-%dT%H:%M:%SZ)"},
"ttl":{"N":"${TTL_AT}"},
"note":{"S":"AGENTS.md Q60 — TTL behaviour probe, no personal information, safe to delete"}}
JSON
)"
aws dynamodb get-item --table-name "$TABLE" --key "{\"submissionId\":{\"S\":\"${TTL_ID}\"}}" \
--query 'Item.{id:submissionId.S,ttl:ttl.N,note:note.S}'
echo "$TTL_ID" > ~/q60-ttl-probe-id.txt
echo "$TTL_AT" >> ~/q60-ttl-probe-id.txt
```
**Expect:** the item, with `ttl` five minutes in the future. The id is written to
a file because the check below happens on a different day and this is the only
thing that connects the two.
### 10.2 The check, and it is not five minutes later
⚠️ **DYNAMODB'S TTL SWEEPER IS NOT PROMPT AND AWS DOES NOT PROMISE THAT IT IS.**
Deletion typically happens within a couple of days of the timestamp passing, and
**up to 48 hours or more is normal and not a fault.** A check run at `ttl + 5
minutes` that finds the item present has measured nothing — it is the same
mistake as reading a value before the transition finishes. So:
- **Earliest useful check: 48 hours after `TTL_AT`.**
- **Do not conclude a failure before 7 days.**
```bash
export TTL_ID=$(sed -n 1p ~/q60-ttl-probe-id.txt)
export TTL_AT=$(sed -n 2p ~/q60-ttl-probe-id.txt)
python3 -c "import time,os; d=time.time()-int(os.environ['TTL_AT']); print(f'{d/3600:.1f} hours since the ttl passed')"
aws dynamodb get-item --table-name "$TABLE" \
--key "{\"submissionId\":{\"S\":\"${TTL_ID}\"}}" --output json
echo "get-item exit=$?"
```
**Read both the output and the exit status.** `get-item` on a missing key exits
**0** with an **empty response body** — it is not an error, and a script that only
checked the exit status would report the record present and absent identically.
| result | meaning |
|---|---|
| `{"Item": {…}}`, under 48 h since `TTL_AT` | inconclusive. Wait. |
| `{}` (empty), any time after `TTL_AT` | **Q60 closes.** The mechanism works. |
| `{"Item": {…}}`, more than 7 days after | **Q60 fails.** TTL is enabled and not deleting. `/legal/privacy/` is then making a claim the database does not honour, and the page has to change or the mechanism does. |
### 10.3 Record it
On the `{}` result, and only then:
- Stamp §7's `Intake table` row: the behaviour is **observed**, with the date and
the elapsed time, alongside the `ENABLED` reading it already carries.
- Close **Q60** in §9, quoting the two timestamps.
- Retire **R19** from §12 — its whole purpose was to keep this surfacing.
- Tick the TTL item on `docs/06`'s checklist. It has two halves and this is the
second: `ENABLED` proved the setting, this proves the behaviour.
- Delete the probe row if it somehow survives, and delete
`~/q60-ttl-probe-id.txt`.
If it fails, that is a **published-disclosure defect**, not a backlog item:
`src/pages/legal/privacy.astro` carries the matching `TODO(pouya)` and the page
asserts the mechanism in as many words.