adr-sml/docs/06-deployment.md

# 06 — Deployment and cutover

Authority: `AGENTS.md` §3 **D3 as amended 2026-08-26** (git + **Gitea Actions**
→ existing S3/CloudFront) and D11 (build everything, one clean cutover).
Existing infrastructure: **`AGENTS.md` §7 is authoritative.**
`docs/reference/AWS-Hosting-Guide.md` records how that infrastructure was
originally built — it is a historical record carrying a do-not-execute banner,
not a procedure, and §7 wins wherever the two disagree (Q24).

---

## Topology

```
Gitea push to main
  └─ Gitea Actions (act_runner)
       ├─ npm ci && npm run build        → ./dist
       ├─ static scoped IAM user key      (from Gitea secrets — NOT OIDC)
       ├─ aws s3 sync ./dist s3://<bucket>   (three passes, see Cache policy)
       └─ cloudfront create-invalidation
Namecheap DNS → CloudFront → S3 (OAC)
API Gateway → Lambda → DynamoDB / SES     (intake, unchanged path)
```

DNS is at **Namecheap, not Route 53** `[verified 2026-08-25]`. Nothing in the
pipeline touches DNS. Certificate renewal is ACM-automatic as long as the
validation CNAME stays in place at Namecheap — **do not delete it.**

## Today, deploys run locally

**`npm run deploy`** (`scripts/deploy-local.sh`) is the current path. It runs
the same guard, the same three sync passes in the same order with the same
cache headers, and the same invalidation as the workflow — at this scale the
pipeline changes only **how a deploy is triggered**, not what it does. Treat the
script and the workflow as one artefact in two places: change one, change both.

Two things block the workflow, and neither is a fact to look up:

- **`adr-sml-deploy` does not exist** — `aws iam get-user` returns
  `NoSuchEntity` (§7, Q22). Create it from *Create the user* below.
- **Actions are not enabled and no runner is registered** (Q23). The Gitea
  instance is jointly administered, so both need its second administrator.

The script **refuses to run as `user/pouya`** — the broadly-permissioned
personal user that has been authenticating to this account. See §10.

## CI runs on Gitea, not GitHub

`AGENTS.md` D3 as amended, 2026-08-26: self-hosted **Gitea**. The instance,
version, and repository are recorded in §7 — the version is comfortably above
the floor for the `vars` context, so the first-step guard is belt-and-braces
rather than load-bearing.

**The live pipeline is `.gitea/workflows/deploy.yml`.** Gitea Actions speaks
GitHub Actions syntax, so it is a near-direct port — the build steps, the
three-pass sync, and the cache headers are unchanged. The GitHub Actions original,
with its OIDC role assumption, stays in the repo as
`docs/reference/github-actions-oidc.yml.example` — deliberately outside
`.github/workflows/`, because Gitea falls back to that directory when
`.gitea/workflows` is absent.

### The one real difference: no OIDC

Gitea is not an AWS OIDC provider. There is no role to assume, so deploys
authenticate with a **scoped IAM user** whose access key lives only in the
repository's Gitea secrets.

This is a genuine step down in security from an OIDC setup — which was designed
here but never built — and it should be treated as one. The mitigations are the
policy scope and the rotation schedule.

**Create the user:**

1. IAM → Users → `adr-sml-deploy`. **Programmatic access only** — no console
   password, no MFA device, no group membership.
2. Attach this inline policy and nothing else. Substitute the real bucket name,
   account ID, and distribution ID from `scripts/aws-discover.sh`:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListSiteBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET_NAME"
    },
    {
      "Sid": "WriteSiteObjects",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    },
    {
      "Sid": "InvalidateOneDistribution",
      "Effect": "Allow",
      "Action": "cloudfront:CreateInvalidation",
      "Resource": "arn:aws:cloudfront::ACCOUNT_ID:distribution/DISTRIBUTION_ID"
    }
  ]
}
```

Four actions on one bucket and one distribution. No `Action: "*"`, no
`Resource: "*"` — the only wildcard is `BUCKET_NAME/*`, which scopes to the
objects of that one bucket. `s3:PutObjectAcl` was dropped on 2026-08-26:
`aws s3 sync` does not use it without `--acl`, and it is inert under Origin
Access Control with ACLs disabled. If a deploy step needs a permission this
policy lacks, the correct response is to question the step, not to widen the
policy.

3. Create an access key. **Copy it once** — AWS will not show the secret again.

**`s3:AbortMultipartUpload` is deliberately absent, and here is the actual
reason.** `aws s3 sync` switches to multipart above its 8 MB
`multipart_threshold`; an interrupted multipart upload then cannot clean up its
own parts, and orphaned parts accrue storage charges that do not appear in the
bucket listing. What makes that safe today is simply that **nothing here comes
close to 8 MB** — the largest file the pipeline
uploads is well under it. The biggest source asset is
`src/assets/pouya-lajevardi.jpg` at 357,627 bytes `[verified 2026-08-26 — stat]`,
Astro emits it smaller still after AVIF/WebP conversion, and the self-hosted font
files are smaller again. **Re-measure `./dist` after the first successful build**
— that, not the repository, is what gets synced. No lifecycle rule exists; do not describe one
as the mitigation, because it is not there.

**Revisit if any single asset approaches 8 MB** — a video, a large PDF, an
un-optimised photograph. At that point either add an S3 lifecycle rule aborting
incomplete multipart uploads after 7 days (preferred — it costs no IAM
permission), or grant `s3:AbortMultipartUpload` on `BUCKET_NAME/*`.

### Gitea configuration

**Repository → Settings → Actions → Secrets:**

| Name | Value |
|---|---|
| `AWS_ACCESS_KEY_ID` | from the IAM user |
| `AWS_SECRET_ACCESS_KEY` | from the IAM user |

**Repository → Settings → Actions → Variables** — not secrets. These are not
sensitive, and keeping them as variables means they appear in run logs where
they are useful for debugging.

| Variable | Value |
|---|---|
| `AWS_REGION` | `AGENTS.md` §7 — Region |
| `S3_BUCKET` | §7 — S3 bucket |
| `CLOUDFRONT_DISTRIBUTION_ID` | §7 — CloudFront |
| `INTAKE_ENDPOINT` | §7 — Intake API |
| `BOOKING_URL` | *(empty — parked, R6)* |

The same four values fill the IAM policy's `BUCKET_NAME`, `ACCOUNT_ID` and
`DISTRIBUTION_ID` placeholders. **They are deliberately not restated here** —
§7 is the single source of truth for operational facts, and the copy that goes
stale is always the one nobody re-reads. `scripts/aws-discover.sh` regenerates
them from AWS if §7 ever needs re-verifying.

> **Read this before creating the key.** The AWS account is **not** a
> single-project account: it is shared with several unrelated sites and with a
> bucket whose name indicates another business's production client-database
> backups. `AGENTS.md` §10 has the specifics and the account identifier; they
> are kept there rather than repeated here. A static deploy key for a marketing
> site lives in that same account, and the scoped policy is what keeps a
> compromised Gitea runner from reaching any of it. Do not widen it, and never
> put the `user/pouya` credentials in CI.

### A runner must exist

Gitea Actions needs `act_runner` registered to this repository or its
organisation, and Actions enabled both site-wide in `app.ini`
(`[actions] ENABLED = true`) and per-repository. Without a runner the workflow
queues silently and never runs — which looks exactly like a broken pipeline.

The workflow installs the AWS CLI if the runner image lacks it, and runs
`aws sts get-caller-identity` before touching anything. **That check is
narrower than it looks:** `sts:GetCallerIdentity` requires no IAM permission at
all, so it succeeds for any valid key regardless of policy. It catches a
missing, malformed, or revoked key; it does **not** catch an under-scoped
policy, which still fails halfway through a sync and leaves the bucket
partially updated. Read it as a key check, not a permissions check.

### The variable guard runs first

The workflow's first step — before checkout, before the build, before any AWS
call — fails the run if `AWS_REGION`, `S3_BUCKET`, or
`CLOUDFRONT_DISTRIBUTION_ID` is empty.

This exists because Gitea only added the `vars` context in 1.21. On an older
instance every `${{ vars.* }}` interpolates to an empty string with no warning,
the sync target becomes `s3://`, and the run dies halfway through with an error
that names nothing useful. The guard converts that into a clean failure that
says which variable is missing — **on every Gitea version**. A recorded version
number would have gone stale; the guard does not.

### Key rotation — an operational obligation

**Rotate `adr-sml-deploy` quarterly.** OIDC would have made this unnecessary;
with a static key it is a standing task:

1. Create a second access key on the same user.
2. Update the Gitea secrets.
3. Run the workflow and confirm it succeeds.
4. **Delete the old key.** Rotation that leaves the old key active is not
   rotation.

Set a calendar reminder. A key that is never rotated is the failure mode this
whole section exists to bound.

## Finding the AWS identifiers

`scripts/aws-discover.sh` re-collects the inventory — bucket, distribution
ID, regions, API endpoint, certificate, SES identities, and whether S3 versioning
is on. Read-only; no call creates or mutates anything.

```bash
chmod +x scripts/aws-discover.sh
./scripts/aws-discover.sh > aws-inventory.txt
```

The output contains resource names and IDs but no secrets.

## Why OIDC would have been better — and why it is unavailable

> **Do not execute this section.** It describes the design that was rejected
> because Gitea cannot support it. The live procedure is *Create the user* above.
> Nothing here should be created in AWS. Following it would add an unused GitHub
> federation trust to the shared AWS account (`AGENTS.md` §10).

A static `AWS_ACCESS_KEY_ID` never expires, is invisible once set, and grants its
permissions to anyone who can reach the repository. OIDC issues a short-lived
token per run, scoped to one repository and one branch — strictly better, and the
reason the rotation schedule above is not optional here.

It needs an identity provider AWS will federate with. GitHub and GitLab both
publish one; **Gitea and Forgejo do not**, so there is nothing for AWS to trust
and no role to assume. That is the whole of the constraint (D3 as amended).

If the project ever moves to GitHub, the workflow to adopt is
`docs/reference/github-actions-oidc.yml.example`, and the setup is: register
`token.actions.githubusercontent.com` as an IAM OIDC provider with audience
`sts.amazonaws.com`; create a role trusting it, conditioned on the `sub` claim
matching the repository and `refs/heads/main`; attach the same four-action policy
given above; then delete `adr-sml-deploy` and its key.

## Cache policy

The mistake to avoid is caching HTML aggressively — a stale index page is a site
that does not update.

| Pattern | `Cache-Control` |
|---|---|
| `*.html` | `public, max-age=0, must-revalidate` |
| `/_astro/*` (hashed) | `public, max-age=31536000, immutable` |
| Fonts | `public, max-age=31536000, immutable` |
| Images | `public, max-age=604800` |
| `robots.txt`, `sitemap*.xml` | `public, max-age=0, must-revalidate` |

Sync in **three** passes, in this order: hashed assets and fonts with the long
TTL, then images, then everything else. Uploading HTML last means a user never
fetches a new page whose assets have not landed yet.

Two ordering dependencies are load-bearing and easy to break:

- Pass 3 re-walks the whole tree; the image headers from pass 2 survive only
  because `aws s3 sync` skips objects it has just uploaded. Reordering the
  passes silently overwrites them with the HTML header.
- Pass 3's `--exclude "_astro/*" --exclude "fonts/*"` also excludes those
  prefixes from `--delete`, so hashed assets from previous deploys are kept
  deliberately — pages still in a browser cache need them. Do not "fix" it.

`robots.txt` and `sitemap*.xml` fall through to pass 3 and get the HTML header.
That is the intended behaviour: both should be re-fetched, and the table above
records what the pipeline actually does rather than an unimplemented ideal.

Invalidate `/*` on deploy. At this traffic volume the cost is nil, and partial
invalidation paths are a reliable source of confusing bugs.

## CloudFront configuration

- Origin: S3 with **Origin Access Control**, bucket not public. Verify the
  bucket policy grants access only to the CloudFront distribution's OAC
  principal and to nothing else, and that public access is still blocked.
- Redirect HTTP → HTTPS. TLS 1.2 minimum.
- Default root object `index.html`.
- **Custom error response:** 404 → `/404.html` with **response code 404**, not
  200. Returning 200 for a missing page tells crawlers every bad URL is real
  content, and it is the single most common misconfiguration in this stack.
- Compression on. Response-headers policy from `05-backend-spec.md`.
- A CloudFront Function for trailing-slash normalisation, so `/about` and
  `/about/` do not both resolve as separate indexable URLs.

## Branch model

`main` is production; a push to `main` is what triggers a deploy. Work on
short-lived branches, open a PR, merge.

**The CI pipeline has never run.** Not for want of a lockfile — `npm ci`,
`astro check` and `astro build` all work now — but because the deploy user does
not exist (Q22) and Actions are not enabled with a runner registered (Q23).
Treat "every push deploys" as the design; today the path is `npm run deploy`.

**Pull request checks — planned, not implemented:** `npm run build` ·
`astro check` · lint · Lighthouse CI against the budgets in `04-seo-spec.md` ·
link check. `.gitea/workflows/deploy.yml` has **no `pull_request` trigger**
(only `push` on `main` and `workflow_dispatch`), so nothing gates a merge today.
`npm run build`, `npm run check` and `npm run lint` all run clean locally.

**Lighthouse is not one of the checks that could be wired today.** `@lhci/cli`
was removed on 2026-08-26 and there is no `npm run lighthouse` script any more —
`AGENTS.md` §7 records why and what re-adding it at build step 7 requires. Wire
the other four; do not write a workflow step that calls a script that does not
exist.

Tag every production deploy `v<year>.<n>` so a rollback has something to name.

## Rollback

1. Re-run the workflow at the last good tag, or
2. `git revert` and push, or
3. Restore from S3 object versioning — **already Enabled** on the site bucket
   (`AGENTS.md` §7). It is the
   difference between a rollback and a rebuild; do not turn it off.

Then invalidate `/*`.

## Cutover checklist — D11 is a single shot, so run all of it

**Content and compliance**
- [ ] Every claim traced to `AGENTS.md` §4 Verified
- [ ] No `TODO(pouya)` remains in any shipped page
- [ ] No matter counts, rates, dollar figures, or testimonials anywhere
- [ ] Q.Arb described as **commenced August 2026** everywhere it appears — §4's
      wording, not the looser "in progress"
- [ ] `/fees/` carries the rates confirmed in D14 and `docs/07-fees.md`, or the page does not ship
- [ ] Privacy policy matches the backend as actually built

**Technical**
- [ ] **Re-add `@lhci/cli`** (removed 2026-08-26 — `AGENTS.md` §7) with a pin
      verified against the registry that day, and a `lighthouserc` carrying the
      budgets from `04-seo-spec.md`. This box gates the next one
- [ ] Lighthouse ≥ 95 mobile on `/`, `/about/`, a practice page, an article
- [ ] Every page renders fully with JavaScript disabled
- [ ] `curl` of each URL returns real content, not a shell
- [ ] All internal links resolve; no orphan pages
- [ ] Sitemap generated and correct; `robots.txt` served, not 403
- [ ] Rich Results Test passes; OG previews render in LinkedIn and Slack
- [ ] 404 returns a 404 status
- [ ] Security headers present (`securityheaders.com` A or better)
- [ ] **SES identities verified for sending** — confirmed 2026-08-26, re-check at cutover: `aws sesv2 get-email-identity --email-identity smlcompany.ca` and confirm `VerifiedForSendingStatus: true`
- [ ] **SES bounce/complaint alarms actually notify someone** — `AGENTS.md` §7 records the `ses-alerts` email subscription as **pending confirmation**, and an unconfirmed SNS subscription drops every message. Confirm it, then `aws sns list-subscriptions-by-topic` and check the ARN is not `PendingConfirmation`. *(SES production access itself is granted — Q19 closed.)*
- [ ] Intake form tested end to end: DynamoDB record written to the intake table (`AGENTS.md` §7), both emails delivered to a real inbox, TTL set
- [ ] Booking link works, including the no-JavaScript fallback — **conditional on R6**; booking is parked and `BOOKING_URL` is empty, so this passes vacuously until a tool is chosen
- [ ] Favicon set complete
- [ ] Tested on iOS Safari, Android Chrome, desktop Safari/Chrome/Firefox
- [ ] Tested at 320 px and at 200% zoom

**Infrastructure**
- [ ] S3 versioning enabled
- [ ] Bucket not publicly readable; OAC in force
- [ ] ACM certificate valid; Namecheap validation CNAME still present
- [ ] CloudWatch alarms: Lambda errors, DLQ depth, 5xx rate
- [ ] Billing budget/alarm still active — `aws budgets describe-budgets --account-id "$(aws sts get-caller-identity --query Account --output text)"`. `docs/reference/AWS-Hosting-Guide.md` set up an **AWS Budget**, which `cloudwatch describe-alarms` will never return. Whether one was actually created is not recorded anywhere: confirm, do not assume

**Post-cutover, same day**
- [ ] Sitemap submitted to Google Search Console and Bing Webmaster Tools
- [ ] Live site fetched as an anonymous crawler to confirm indexable content
- [ ] LinkedIn profile and ADRIC/ADRIO listings updated to point here
- [ ] Archive the old single-file build to `_archive/` — do not delete it
- [ ] `AGENTS.md` Change Log entry recording the cutover