From 649dd737c71782d5f09538017866d334c5c4e27a Mon Sep 17 00:00:00 2001 From: Lars Nolden Date: Mon, 29 Jun 2026 17:23:08 +0200 Subject: [PATCH] Readme update --- README.md | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/README.md b/README.md index 1ad5496..5885159 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,78 @@ host-to-host traffic: - `db-host` only accepts Postgres (port 5432) from `10.0.0.0/8` (firewall rule in `nix/hosts/db-host.nix`). +## MQTT ingest path + +How a sender device's telemetry reaches Postgres, and the reasoning behind each +choice. The device firmware is the fixed end of this contract, so the rest of +the stack is built to match it rather than the other way around. + +``` +device ──MQTTS:8883──▶ Caddy (layer4, TLS term) ──MQTT:1883──▶ HiveMQ CE ──▶ ingester ──▶ Postgres + acrios// loopback file-RBAC IMSI→tenant +``` + +### Broker auth: file-RBAC, not allow-all + +HiveMQ CE ships the `hivemq-allow-all` extension — any client can publish or +subscribe to anything. We replaced it with the vendored +[`hivemq-file-rbac-extension`](https://github.com/hivemq/hivemq-file-rbac-extension) +(`nix/modules/gebos-hivemq.nix`), which adds username/password auth and +topic-level authorization. `gebos-hivemq.preStart` reconciles the writable +extensions dir on every start: it installs the pinned extension and **removes +allow-all**, so a redeploy fixes already-provisioned boxes too. Credentials +(`credentials.xml`) are rendered by sops-nix (`services.gebos.secrets.hivemq`) +and symlinked from the `/run/secrets` tmpfs into the extension's `conf/` — they +never touch the Nix store or the state dir. `password-type` is `PLAIN` because +the file is already an encrypted secret at rest and `0400`/`gebos-hivemq` at +runtime; flip to `HASHED` for defence-in-depth. + +There are three broker users: `ingester` (subscribes the device tree), `admin` +(break-glass superuser), and `bender` (the shared device account, below). + +### Topic scheme: the device's `acrios//…` is the source of truth + +Senders publish to `acrios//` — `mqtt_topic_base` plus the SIM's +IMSI plus the metric name. We adopted that namespace verbatim rather than +reshaping it into the `t//d//…` form the schema originally +imagined, because the firmware can't emit our `tenant_id`/`device_id` **UUIDs** +— it only knows its IMSI. So the IMSI is the natural device key, and the +ingester (subscribed to `acrios/#`) will resolve `IMSI → (tenant_id, +device_id)` via a **device registry** before inserting into `public.telemetry`. +That registry table and the ingester's topic parsing are still TODO — the +ingester is currently a stub. + +### Device auth: shared user now (Option A), per-device later (Option B) + +The sender logs in with a single shared account (`bender`), authorized to +publish/subscribe under `acrios/#`. Tenant isolation is therefore enforced +**downstream** by the ingester's IMSI registry, not at the broker — any device +could publish under any IMSI. That's an accepted trade-off for a small trusted +fleet, and it gets data flowing without per-device provisioning. + +The production answer (**Option B**, a TODO in `nix/modules/gebos-secrets.nix`) +is one broker user per device with `username == IMSI`, scoped to +`acrios/${{username}}/#` so the broker itself prevents a device from spoofing +another's IMSI. It's not wired up because it needs a firmware change +(`mqtt_user = `) and a per-device password provisioning flow. Note the +firmware's client-id (`acrcv-`) carries a prefix the topic doesn't, so +per-device scoping must key on username, not `${{clientid}}`. + +### TLS: caddy-l4 terminates, HiveMQ stays loopback + +HiveMQ binds `127.0.0.1:1883` only; it never faces the network. Senders connect +MQTTS on `:8883`, and Caddy — rebuilt with the +[`caddy-l4`](https://github.com/mholt/caddy-l4) layer4 module — terminates TLS +and proxies cleartext to the broker (`nix/modules/gebos-caddy.nix`). We reuse +Caddy here (rather than a native HiveMQ TLS listener + keystore) so there's one +ACME story for the whole fleet. The wrinkle: caddy-l4's `tls` handler only +*terminates* — it never obtains certs — so each `tcpProxy` site also emits a +companion HTTPS block purely to make Caddy's automatic HTTPS provision the cert +that the layer4 handler then serves by SNI. That's why `mqtt-ingest` opens +**80/443** (ACME challenge + renewal) alongside **8883**. The custom Caddy build +is only used on hosts that actually have a `tcpProxy` site; app-host's +static/reverse-proxy sites stay on stock Caddy. + ## Repo layout ```