Gebos/README.md

# Gebos

IoT telemetry stack on bare-metal NixOS.

This repo is a monorepo containing all code, NixOS modules, host configurations,
and CI/CD pipelines for the project. Everything is one `flake.nix`.

## Architecture

Three hosts (see issue #21 for the full design discussion):

| Host          | Role                                      | Public hostname        | Private IP |
| ------------- | ----------------------------------------- | ---------------------- | ---------- |
| `mqtt-ingest` | HiveMQ CE + Go MQTT→Postgres ingester     | `ingest.gebos.online`     | `10.0.0.4` |
| `db-host`     | Postgres 17 + TimescaleDB (telemetry+auth)| `db.gebos.online` (SSH only) | `10.0.0.2` |
| `app-host`    | Caddy + Kong + Supabase (compose) + SPA   | `app.gebos.online`, `api.gebos.online` | `10.0.0.3` |

Public REST surface is PostgREST + SQL `/rpc/` functions, fronted by Kong, TLS-terminated by Caddy.
Supabase Studio is bound to `127.0.0.1` on `app-host` — reach it with `ssh -L 3000:127.0.0.1:3000 app-host`.

### Networking

There is no private DNS. Each host has a public hostname (used for `deploy-rs`
SSH access from the Gitea runner, which is **not** on the private network, plus
TLS ingress where applicable), and a static `10.0.0.0/8` IP used for all
host-to-host traffic:

- `db.gebos.online` is for SSH/deploy only — Postgres is never exposed publicly.
- `app-host` and `mqtt-ingest` reach Postgres at `10.0.0.2:5432` over the private
  network.
- `db-host` only accepts Postgres (port 5432) from `10.0.0.0/8` (firewall rule
  in `nix/hosts/db-host.nix`).

## MQTT ingest path

How a sender device's telemetry reaches Postgres, and the reasoning behind each
choice. The device firmware is the fixed end of this contract, so the rest of
the stack is built to match it rather than the other way around.

```
device ──MQTTS:8883──▶ Caddy (layer4, TLS term) ──MQTT:1883──▶ HiveMQ CE ──▶ ingester ──▶ Postgres
         acrios/<IMSI>/<metric>          loopback                file-RBAC      IMSI→tenant
```

### Broker auth: file-RBAC, not allow-all

HiveMQ CE ships the `hivemq-allow-all` extension — any client can publish or
subscribe to anything. We replaced it with the vendored
[`hivemq-file-rbac-extension`](https://github.com/hivemq/hivemq-file-rbac-extension)
(`nix/modules/gebos-hivemq.nix`), which adds username/password auth and
topic-level authorization. `gebos-hivemq.preStart` reconciles the writable
extensions dir on every start: it installs the pinned extension and **removes
allow-all**, so a redeploy fixes already-provisioned boxes too. Credentials
(`credentials.xml`) are rendered by sops-nix (`services.gebos.secrets.hivemq`)
and symlinked from the `/run/secrets` tmpfs into the extension's `conf/` — they
never touch the Nix store or the state dir. `password-type` is `PLAIN` because
the file is already an encrypted secret at rest and `0400`/`gebos-hivemq` at
runtime; flip to `HASHED` for defence-in-depth.

There are three broker users: `ingester` (subscribes the device tree), `admin`
(break-glass superuser), and `bender` (the shared device account, below).

### Topic scheme: the device's `acrios/<IMSI>/…` is the source of truth

Senders publish to `acrios/<IMSI>/<metric>` — `mqtt_topic_base` plus the SIM's
IMSI plus the metric name. We adopted that namespace verbatim rather than
reshaping it into the `t/<tenant>/d/<device>/…` form the schema originally
imagined, because the firmware can't emit our `tenant_id`/`device_id` **UUIDs**
— it only knows its IMSI. So the IMSI is the natural device key, and the
ingester (subscribed to `acrios/#`) will resolve `IMSI → (tenant_id,
device_id)` via a **device registry** before inserting into `public.telemetry`.
That registry table and the ingester's topic parsing are still TODO — the
ingester is currently a stub.

### Device auth: shared user now (Option A), per-device later (Option B)

The sender logs in with a single shared account (`bender`), authorized to
publish/subscribe under `acrios/#`. Tenant isolation is therefore enforced
**downstream** by the ingester's IMSI registry, not at the broker — any device
could publish under any IMSI. That's an accepted trade-off for a small trusted
fleet, and it gets data flowing without per-device provisioning.

The production answer (**Option B**, a TODO in `nix/modules/gebos-secrets.nix`)
is one broker user per device with `username == IMSI`, scoped to
`acrios/${{username}}/#` so the broker itself prevents a device from spoofing
another's IMSI. It's not wired up because it needs a firmware change
(`mqtt_user = <IMSI>`) and a per-device password provisioning flow. Note the
firmware's client-id (`acrcv-<IMSI>`) carries a prefix the topic doesn't, so
per-device scoping must key on username, not `${{clientid}}`.

### TLS: caddy-l4 terminates, HiveMQ stays loopback

HiveMQ binds `127.0.0.1:1883` only; it never faces the network. Senders connect
MQTTS on `:8883`, and Caddy — rebuilt with the
[`caddy-l4`](https://github.com/mholt/caddy-l4) layer4 module — terminates TLS
and proxies cleartext to the broker (`nix/modules/gebos-caddy.nix`). We reuse
Caddy here (rather than a native HiveMQ TLS listener + keystore) so there's one
ACME story for the whole fleet. The wrinkle: caddy-l4's `tls` handler only
*terminates* — it never obtains certs — so each `tcpProxy` site also emits a
companion HTTPS block purely to make Caddy's automatic HTTPS provision the cert
that the layer4 handler then serves by SNI. That's why `mqtt-ingest` opens
**80/443** (ACME challenge + renewal) alongside **8883**. The custom Caddy build
is only used on hosts that actually have a `tcpProxy` site; app-host's
static/reverse-proxy sites stay on stock Caddy.

## Repo layout

```
flake.nix
frontend/                # Vite + React SPA
ingester/                # Go MQTT → Postgres
nix/
  modules/               # NixOS modules (one per service)
  hosts/                 # nixosConfigurations: mqtt-ingest, db-host, app-host
  supabase/              # vendored Supabase docker-compose, db init SQL
  dev/                   # process-compose for local development
  deploy.nix             # deploy-rs node map
.gitea/workflows/        # CI + CD
```

## Local development

```
nix run .#dev
```

Brings the full stack up on one machine via process-compose-flake (Postgres,
Supabase compose, Kong, Caddy, ingester, Vite dev server). NixOS required.

## Deployment

There are two distinct phases. **Initial provisioning** turns a blank box into a
NixOS host (`nixos-anywhere`, run once per machine). **Updates** push new
closures to a host that already runs NixOS (`deploy-rs`, run on every change).

### SSH key setup (do this first)

Both phases authenticate over SSH with `~/.ssh/larsnolden`, which is
passphrase-protected. Load it into an `ssh-agent` once so the deploy tools can
reuse it without prompting:

```fish
eval (ssh-agent -c)        # bash/zsh: eval "$(ssh-agent -s)"
ssh-add ~/.ssh/larsnolden  # enter the passphrase once
ssh-add -l                 # confirm the key is loaded
```

This is **required** for `deploy-rs`, not just a convenience: with
`magicRollback = true` (see `nix/deploy.nix`) activation opens two concurrent SSH
connections — the activation command and a rollback waiter. Without an agent,
both race to read the passphrase from the terminal, one loses, and the deploy
fails with `Permission denied (publickey,keyboard-interactive)` even though
manual SSH and the copy step work. The agent serves the key to every connection,
so no prompt is needed.

### Initial provisioning (`nixos-anywhere`)

`deploy-rs` only *updates* a machine that already runs NixOS — it copies a
prebuilt closure and activates it. A fresh box (e.g. a stock Debian image with
only a `root` user) has no Nix store and no NixOS generation to switch to, so
`deploy-rs` fails with `nix-store: command not found`. Use
[`nixos-anywhere`](https://github.com/nix-community/nixos-anywhere) to install
NixOS over SSH first; after that, `deploy-rs` takes over for all subsequent
deploys.

`nixos-anywhere` SSHes in as `root`, kexecs into an in-memory NixOS installer,
partitions and formats the disk per the host's [`disko`](https://github.com/nix-community/disko)
config, installs `nixosConfigurations.<host>`, and reboots into NixOS. **This
wipes the target disk.**

Prerequisites, per host, before running it:

1. **A real disk layout.** Hosts currently import the fictional
   `nix/hosts/placeholder-hardware.nix` (it only exists so `nix flake check`
   evaluates). Replace that import with a `disko` config describing the actual
   disk device (`/dev/sda` vs `/dev/vda`/nvme) and firmware (UEFI vs legacy
   BIOS). `disko` replaces the hand-generated `hardware-configuration.nix`.
2. **Root SSH access** to the box. The `deploy` user and its authorized keys are
   created by `nix/hosts/common.nix` during the install, so deploy-rs access
   works automatically once NixOS is up.
3. **Host secrets key** present so sops-nix can decrypt at first boot — see
   [`nix/secrets/README.md`](nix/secrets/README.md). Otherwise services that
   read `/run/secrets/*` (e.g. the ingester) fail to start after reboot.

Then, from the repo root:

```
# installs NixOS onto the target, wiping its disk
nix run github:nix-community/nixos-anywhere -- \
  --flake .#mqtt-ingest root@ingest.gebos.online
```

Repeat with `.#db-host root@db.gebos.online` and `.#app-host root@app.gebos.online`.
Provision `db-host` first if you intend to deploy updates immediately afterward
(see the ordering note below). Once a host has rebooted into NixOS, never run
`nixos-anywhere` against it again — use `deploy-rs`.

### Updates (`deploy-rs`)

`deploy-rs` from a Gitea Actions runner on push to `main`. Closures are built
once, copied to each host, activated with auto-rollback. Order: `db-host` →
`app-host` → `mqtt-ingest`.

Running it by hand needs the key loaded into an `ssh-agent` first — see
[SSH key setup](#ssh-key-setup-do-this-first) above.

```
nix run github:serokell/deploy-rs -- .#db-host       # one host
nix run github:serokell/deploy-rs -- .                # all hosts
```

## Secrets

[sops-nix](https://github.com/Mic92/sops-nix) + [age](https://github.com/FiloSottile/age).
Single encrypted file at `nix/secrets/secrets.yaml`; each host decrypts only the
keys it needs at activation, rendered into a tmpfs env file consumed by systemd
`EnvironmentFile=`. Plaintext never enters the Nix store. See
[`nix/secrets/README.md`](nix/secrets/README.md) for bootstrap and rotation.

Local dev needs **no** secrets bootstrap — `nix run .#dev`, `go run ./ingester`,
and `npm --prefix frontend run dev` all default to the local dev stack values
defined in `nix/dev/process-compose.nix`.