Readme update
deploy / deploy (push) Failing after 3m20s
check / flake-check (push) Successful in 17s

This commit is contained in:
Lars Nolden
2026-06-29 17:23:08 +02:00
parent bf8b25f374
commit 649dd737c7
+72
View File
@@ -31,6 +31,78 @@ host-to-host traffic:
- `db-host` only accepts Postgres (port 5432) from `10.0.0.0/8` (firewall rule - `db-host` only accepts Postgres (port 5432) from `10.0.0.0/8` (firewall rule
in `nix/hosts/db-host.nix`). in `nix/hosts/db-host.nix`).
## MQTT ingest path
How a sender device's telemetry reaches Postgres, and the reasoning behind each
choice. The device firmware is the fixed end of this contract, so the rest of
the stack is built to match it rather than the other way around.
```
device ──MQTTS:8883──▶ Caddy (layer4, TLS term) ──MQTT:1883──▶ HiveMQ CE ──▶ ingester ──▶ Postgres
acrios/<IMSI>/<metric> loopback file-RBAC IMSI→tenant
```
### Broker auth: file-RBAC, not allow-all
HiveMQ CE ships the `hivemq-allow-all` extension — any client can publish or
subscribe to anything. We replaced it with the vendored
[`hivemq-file-rbac-extension`](https://github.com/hivemq/hivemq-file-rbac-extension)
(`nix/modules/gebos-hivemq.nix`), which adds username/password auth and
topic-level authorization. `gebos-hivemq.preStart` reconciles the writable
extensions dir on every start: it installs the pinned extension and **removes
allow-all**, so a redeploy fixes already-provisioned boxes too. Credentials
(`credentials.xml`) are rendered by sops-nix (`services.gebos.secrets.hivemq`)
and symlinked from the `/run/secrets` tmpfs into the extension's `conf/` — they
never touch the Nix store or the state dir. `password-type` is `PLAIN` because
the file is already an encrypted secret at rest and `0400`/`gebos-hivemq` at
runtime; flip to `HASHED` for defence-in-depth.
There are three broker users: `ingester` (subscribes the device tree), `admin`
(break-glass superuser), and `bender` (the shared device account, below).
### Topic scheme: the device's `acrios/<IMSI>/…` is the source of truth
Senders publish to `acrios/<IMSI>/<metric>``mqtt_topic_base` plus the SIM's
IMSI plus the metric name. We adopted that namespace verbatim rather than
reshaping it into the `t/<tenant>/d/<device>/…` form the schema originally
imagined, because the firmware can't emit our `tenant_id`/`device_id` **UUIDs**
— it only knows its IMSI. So the IMSI is the natural device key, and the
ingester (subscribed to `acrios/#`) will resolve `IMSI → (tenant_id,
device_id)` via a **device registry** before inserting into `public.telemetry`.
That registry table and the ingester's topic parsing are still TODO — the
ingester is currently a stub.
### Device auth: shared user now (Option A), per-device later (Option B)
The sender logs in with a single shared account (`bender`), authorized to
publish/subscribe under `acrios/#`. Tenant isolation is therefore enforced
**downstream** by the ingester's IMSI registry, not at the broker — any device
could publish under any IMSI. That's an accepted trade-off for a small trusted
fleet, and it gets data flowing without per-device provisioning.
The production answer (**Option B**, a TODO in `nix/modules/gebos-secrets.nix`)
is one broker user per device with `username == IMSI`, scoped to
`acrios/${{username}}/#` so the broker itself prevents a device from spoofing
another's IMSI. It's not wired up because it needs a firmware change
(`mqtt_user = <IMSI>`) and a per-device password provisioning flow. Note the
firmware's client-id (`acrcv-<IMSI>`) carries a prefix the topic doesn't, so
per-device scoping must key on username, not `${{clientid}}`.
### TLS: caddy-l4 terminates, HiveMQ stays loopback
HiveMQ binds `127.0.0.1:1883` only; it never faces the network. Senders connect
MQTTS on `:8883`, and Caddy — rebuilt with the
[`caddy-l4`](https://github.com/mholt/caddy-l4) layer4 module — terminates TLS
and proxies cleartext to the broker (`nix/modules/gebos-caddy.nix`). We reuse
Caddy here (rather than a native HiveMQ TLS listener + keystore) so there's one
ACME story for the whole fleet. The wrinkle: caddy-l4's `tls` handler only
*terminates* — it never obtains certs — so each `tcpProxy` site also emits a
companion HTTPS block purely to make Caddy's automatic HTTPS provision the cert
that the layer4 handler then serves by SNI. That's why `mqtt-ingest` opens
**80/443** (ACME challenge + renewal) alongside **8883**. The custom Caddy build
is only used on hosts that actually have a `tcpProxy` site; app-host's
static/reverse-proxy sites stay on stock Caddy.
## Repo layout ## Repo layout
``` ```