Gebos/README.md

Gebos

IoT telemetry stack on bare-metal NixOS.

This repo is a monorepo containing all code, NixOS modules, host configurations, and CI/CD pipelines for the project. Everything is one flake.nix.

Architecture

Three hosts (see issue #21 for the full design discussion):

Host	Role	Public hostname	Private IP
mqtt-ingest	HiveMQ CE + Go MQTT→Postgres ingester	ingest.gebos.online	10.0.0.4
db-host	Postgres 17 + TimescaleDB (telemetry+auth)	db.gebos.online (SSH only)	10.0.0.2
app-host	Caddy + Kong + Supabase (compose) + SPA	app.gebos.online, api.gebos.online	10.0.0.3

Public REST surface is PostgREST + SQL /rpc/ functions, fronted by Kong, TLS-terminated by Caddy. Supabase Studio is bound to 127.0.0.1 on app-host — reach it with ssh -L 3000:127.0.0.1:3000 app-host.

Networking

There is no private DNS. Each host has a public hostname (used for deploy-rs SSH access from the Gitea runner, which is not on the private network, plus TLS ingress where applicable), and a static 10.0.0.0/8 IP used for all host-to-host traffic:

• db.gebos.online is for SSH/deploy only — Postgres is never exposed publicly.
• app-host and mqtt-ingest reach Postgres at 10.0.0.2:5432 over the private network.
• db-host only accepts Postgres (port 5432) from 10.0.0.0/8 (firewall rule in nix/hosts/db-host.nix).

MQTT ingest path

How a sender device's telemetry reaches Postgres, and the reasoning behind each choice. The device firmware is the fixed end of this contract, so the rest of the stack is built to match it rather than the other way around.

device ──MQTTS:8883──▶ Caddy (layer4, TLS term) ──MQTT:1883──▶ HiveMQ CE ──▶ ingester ──▶ Postgres
         acrios/<IMSI>/<metric>          loopback                file-RBAC      IMSI→tenant

Broker auth: file-RBAC, not allow-all

HiveMQ CE ships the hivemq-allow-all extension — any client can publish or subscribe to anything. We replaced it with the vendored hivemq-file-rbac-extension (nix/modules/gebos-hivemq.nix), which adds username/password auth and topic-level authorization. gebos-hivemq.preStart reconciles the writable extensions dir on every start: it installs the pinned extension and removes allow-all, so a redeploy fixes already-provisioned boxes too. Credentials (credentials.xml) are rendered by sops-nix (services.gebos.secrets.hivemq) and symlinked from the /run/secrets tmpfs into the extension's conf/ — they never touch the Nix store or the state dir. password-type is PLAIN because the file is already an encrypted secret at rest and 0400/gebos-hivemq at runtime; flip to HASHED for defence-in-depth.

There are three broker users: ingester (subscribes the device tree), admin (break-glass superuser), and bender (the shared device account, below).

Topic scheme: the device's acrios/<IMSI>/… is the source of truth

Senders publish to acrios/<IMSI>/<metric> — mqtt_topic_base plus the SIM's IMSI plus the metric name. We adopted that namespace verbatim rather than reshaping it into the t/<tenant>/d/<device>/… form the schema originally imagined, because the firmware can't emit our tenant_id/device_id UUIDs — it only knows its IMSI. So the IMSI is the natural device key, and the ingester (subscribed to acrios/#) will resolve IMSI → (tenant_id, device_id) via a device registry before inserting into public.telemetry. That registry table and the ingester's topic parsing are still TODO — the ingester is currently a stub.

Device auth: shared user now (Option A), per-device later (Option B)

The sender logs in with a single shared account (bender), authorized to publish/subscribe under acrios/#. Tenant isolation is therefore enforced downstream by the ingester's IMSI registry, not at the broker — any device could publish under any IMSI. That's an accepted trade-off for a small trusted fleet, and it gets data flowing without per-device provisioning.

The production answer (Option B, a TODO in nix/modules/gebos-secrets.nix) is one broker user per device with username == IMSI, scoped to acrios/${{username}}/# so the broker itself prevents a device from spoofing another's IMSI. It's not wired up because it needs a firmware change (mqtt_user = <IMSI>) and a per-device password provisioning flow. Note the firmware's client-id (acrcv-<IMSI>) carries a prefix the topic doesn't, so per-device scoping must key on username, not ${{clientid}}.

TLS: caddy-l4 terminates, HiveMQ stays loopback

HiveMQ binds 127.0.0.1:1883 only; it never faces the network. Senders connect MQTTS on :8883, and Caddy — rebuilt with the caddy-l4 layer4 module — terminates TLS and proxies cleartext to the broker (nix/modules/gebos-caddy.nix). We reuse Caddy here (rather than a native HiveMQ TLS listener + keystore) so there's one ACME story for the whole fleet. The wrinkle: caddy-l4's tls handler only terminates — it never obtains certs — so each tcpProxy site also emits a companion HTTPS block purely to make Caddy's automatic HTTPS provision the cert that the layer4 handler then serves by SNI. That's why mqtt-ingest opens 80/443 (ACME challenge + renewal) alongside 8883. The custom Caddy build is only used on hosts that actually have a tcpProxy site; app-host's static/reverse-proxy sites stay on stock Caddy.

Repo layout

flake.nix
frontend/                # Vite + React SPA
ingester/                # Go MQTT → Postgres
nix/
  modules/               # NixOS modules (one per service)
  hosts/                 # nixosConfigurations: mqtt-ingest, db-host, app-host
  supabase/              # vendored Supabase docker-compose, db init SQL
  dev/                   # process-compose for local development
  deploy.nix             # deploy-rs node map
.gitea/workflows/        # CI + CD

Local development

nix run .#dev

Brings the full stack up on one machine via process-compose-flake (Postgres, Supabase compose, Kong, Caddy, ingester, Vite dev server). NixOS required.

Deployment

There are two distinct phases. Initial provisioning turns a blank box into a NixOS host (nixos-anywhere, run once per machine). Updates push new closures to a host that already runs NixOS (deploy-rs, run on every change).

SSH key setup (do this first)

Both phases authenticate over SSH with ~/.ssh/larsnolden, which is passphrase-protected. Load it into an ssh-agent once so the deploy tools can reuse it without prompting:

eval (ssh-agent -c)        # bash/zsh: eval "$(ssh-agent -s)"
ssh-add ~/.ssh/larsnolden  # enter the passphrase once
ssh-add -l                 # confirm the key is loaded

This is required for deploy-rs, not just a convenience: with magicRollback = true (see nix/deploy.nix) activation opens two concurrent SSH connections — the activation command and a rollback waiter. Without an agent, both race to read the passphrase from the terminal, one loses, and the deploy fails with Permission denied (publickey,keyboard-interactive) even though manual SSH and the copy step work. The agent serves the key to every connection, so no prompt is needed.

Initial provisioning (nixos-anywhere)

deploy-rs only updates a machine that already runs NixOS — it copies a prebuilt closure and activates it. A fresh box (e.g. a stock Debian image with only a root user) has no Nix store and no NixOS generation to switch to, so deploy-rs fails with nix-store: command not found. Use nixos-anywhere to install NixOS over SSH first; after that, deploy-rs takes over for all subsequent deploys.

nixos-anywhere SSHes in as root, kexecs into an in-memory NixOS installer, partitions and formats the disk per the host's disko config, installs nixosConfigurations.<host>, and reboots into NixOS. This wipes the target disk.

Prerequisites, per host, before running it:

1. A real disk layout. Hosts currently import the fictional nix/hosts/placeholder-hardware.nix (it only exists so nix flake check evaluates). Replace that import with a disko config describing the actual disk device (/dev/sda vs /dev/vda/nvme) and firmware (UEFI vs legacy BIOS). disko replaces the hand-generated hardware-configuration.nix.
2. Root SSH access to the box. The deploy user and its authorized keys are created by nix/hosts/common.nix during the install, so deploy-rs access works automatically once NixOS is up.
3. Host secrets key present so sops-nix can decrypt at first boot — see nix/secrets/README.md. Otherwise services that read /run/secrets/* (e.g. the ingester) fail to start after reboot.

Then, from the repo root:

# installs NixOS onto the target, wiping its disk
nix run github:nix-community/nixos-anywhere -- \
  --flake .#mqtt-ingest root@ingest.gebos.online

Repeat with .#db-host root@db.gebos.online and .#app-host root@app.gebos.online. Provision db-host first if you intend to deploy updates immediately afterward (see the ordering note below). Once a host has rebooted into NixOS, never run nixos-anywhere against it again — use deploy-rs.

Updates (deploy-rs)

deploy-rs from a Gitea Actions runner on push to main. Closures are built once, copied to each host, activated with auto-rollback. Order: db-host → app-host → mqtt-ingest.

Running it by hand needs the key loaded into an ssh-agent first — see SSH key setup above.

nix run github:serokell/deploy-rs -- .#db-host       # one host
nix run github:serokell/deploy-rs -- .                # all hosts

Secrets

sops-nix + age. Single encrypted file at nix/secrets/secrets.yaml; each host decrypts only the keys it needs at activation, rendered into a tmpfs env file consumed by systemd EnvironmentFile=. Plaintext never enters the Nix store. See nix/secrets/README.md for bootstrap and rotation.

Local dev needs no secrets bootstrap — nix run .#dev, go run ./ingester, and npm --prefix frontend run dev all default to the local dev stack values defined in nix/dev/process-compose.nix.