Readme update
This commit is contained in:
@@ -31,6 +31,78 @@ host-to-host traffic:
|
||||
- `db-host` only accepts Postgres (port 5432) from `10.0.0.0/8` (firewall rule
|
||||
in `nix/hosts/db-host.nix`).
|
||||
|
||||
## MQTT ingest path
|
||||
|
||||
How a sender device's telemetry reaches Postgres, and the reasoning behind each
|
||||
choice. The device firmware is the fixed end of this contract, so the rest of
|
||||
the stack is built to match it rather than the other way around.
|
||||
|
||||
```
|
||||
device ──MQTTS:8883──▶ Caddy (layer4, TLS term) ──MQTT:1883──▶ HiveMQ CE ──▶ ingester ──▶ Postgres
|
||||
acrios/<IMSI>/<metric> loopback file-RBAC IMSI→tenant
|
||||
```
|
||||
|
||||
### Broker auth: file-RBAC, not allow-all
|
||||
|
||||
HiveMQ CE ships the `hivemq-allow-all` extension — any client can publish or
|
||||
subscribe to anything. We replaced it with the vendored
|
||||
[`hivemq-file-rbac-extension`](https://github.com/hivemq/hivemq-file-rbac-extension)
|
||||
(`nix/modules/gebos-hivemq.nix`), which adds username/password auth and
|
||||
topic-level authorization. `gebos-hivemq.preStart` reconciles the writable
|
||||
extensions dir on every start: it installs the pinned extension and **removes
|
||||
allow-all**, so a redeploy fixes already-provisioned boxes too. Credentials
|
||||
(`credentials.xml`) are rendered by sops-nix (`services.gebos.secrets.hivemq`)
|
||||
and symlinked from the `/run/secrets` tmpfs into the extension's `conf/` — they
|
||||
never touch the Nix store or the state dir. `password-type` is `PLAIN` because
|
||||
the file is already an encrypted secret at rest and `0400`/`gebos-hivemq` at
|
||||
runtime; flip to `HASHED` for defence-in-depth.
|
||||
|
||||
There are three broker users: `ingester` (subscribes the device tree), `admin`
|
||||
(break-glass superuser), and `bender` (the shared device account, below).
|
||||
|
||||
### Topic scheme: the device's `acrios/<IMSI>/…` is the source of truth
|
||||
|
||||
Senders publish to `acrios/<IMSI>/<metric>` — `mqtt_topic_base` plus the SIM's
|
||||
IMSI plus the metric name. We adopted that namespace verbatim rather than
|
||||
reshaping it into the `t/<tenant>/d/<device>/…` form the schema originally
|
||||
imagined, because the firmware can't emit our `tenant_id`/`device_id` **UUIDs**
|
||||
— it only knows its IMSI. So the IMSI is the natural device key, and the
|
||||
ingester (subscribed to `acrios/#`) will resolve `IMSI → (tenant_id,
|
||||
device_id)` via a **device registry** before inserting into `public.telemetry`.
|
||||
That registry table and the ingester's topic parsing are still TODO — the
|
||||
ingester is currently a stub.
|
||||
|
||||
### Device auth: shared user now (Option A), per-device later (Option B)
|
||||
|
||||
The sender logs in with a single shared account (`bender`), authorized to
|
||||
publish/subscribe under `acrios/#`. Tenant isolation is therefore enforced
|
||||
**downstream** by the ingester's IMSI registry, not at the broker — any device
|
||||
could publish under any IMSI. That's an accepted trade-off for a small trusted
|
||||
fleet, and it gets data flowing without per-device provisioning.
|
||||
|
||||
The production answer (**Option B**, a TODO in `nix/modules/gebos-secrets.nix`)
|
||||
is one broker user per device with `username == IMSI`, scoped to
|
||||
`acrios/${{username}}/#` so the broker itself prevents a device from spoofing
|
||||
another's IMSI. It's not wired up because it needs a firmware change
|
||||
(`mqtt_user = <IMSI>`) and a per-device password provisioning flow. Note the
|
||||
firmware's client-id (`acrcv-<IMSI>`) carries a prefix the topic doesn't, so
|
||||
per-device scoping must key on username, not `${{clientid}}`.
|
||||
|
||||
### TLS: caddy-l4 terminates, HiveMQ stays loopback
|
||||
|
||||
HiveMQ binds `127.0.0.1:1883` only; it never faces the network. Senders connect
|
||||
MQTTS on `:8883`, and Caddy — rebuilt with the
|
||||
[`caddy-l4`](https://github.com/mholt/caddy-l4) layer4 module — terminates TLS
|
||||
and proxies cleartext to the broker (`nix/modules/gebos-caddy.nix`). We reuse
|
||||
Caddy here (rather than a native HiveMQ TLS listener + keystore) so there's one
|
||||
ACME story for the whole fleet. The wrinkle: caddy-l4's `tls` handler only
|
||||
*terminates* — it never obtains certs — so each `tcpProxy` site also emits a
|
||||
companion HTTPS block purely to make Caddy's automatic HTTPS provision the cert
|
||||
that the layer4 handler then serves by SNI. That's why `mqtt-ingest` opens
|
||||
**80/443** (ACME challenge + renewal) alongside **8883**. The custom Caddy build
|
||||
is only used on hosts that actually have a `tcpProxy` site; app-host's
|
||||
static/reverse-proxy sites stay on stock Caddy.
|
||||
|
||||
## Repo layout
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user