Compare commits
8 Commits
b10d22bd57
...
649dd737c7
| Author | SHA1 | Date | |
|---|---|---|---|
 Lars Nolden
|
649dd737c7 | ||
|
Lars Nolden
|
bf8b25f374 | ||
|
Lars Nolden
|
70c2ada986 | ||
|
Lars Nolden
|
4f9c6086bd | ||
|
Lars Nolden
|
c526cf671f | ||
|
Lars Nolden
|
30b325393a | ||
|
Lars Nolden
|
7579d4dcf5 | ||
|
Lars Nolden
|
9a6a901dfe |
@@ -13,7 +13,7 @@ jobs:
|
||||
run: |
|
||||
mkdir -p ~/.ssh
|
||||
install -m 600 /dev/stdin ~/.ssh/id_ed25519 <<< "${{ secrets.DEPLOY_SSH_KEY }}"
|
||||
ssh-keyscan -H db-host.gebos.internal app-host.gebos.internal ingest.ge-bos.de >> ~/.ssh/known_hosts 2>/dev/null || true
|
||||
ssh-keyscan -H db.gebos.online app.gebos.online ingest.gebos.online >> ~/.ssh/known_hosts 2>/dev/null || true
|
||||
|
||||
- name: Build all closures
|
||||
run: nix flake check --no-build
|
||||
|
||||
+5
-5
@@ -10,17 +10,17 @@
|
||||
|
||||
keys:
|
||||
# ---- developers (laptops, hardware keys) ----
|
||||
- &dev_lars age1TODO_lars_personal_age_pubkey_replace_me_before_first_real_secret
|
||||
- &dev_lars age1cx8ul285kjkzmnhw6skdstnzrxnnme4xkflknzn7yhv52fgxqevqkd66cn
|
||||
|
||||
# ---- hosts (derived from each host's ssh_host_ed25519_key.pub via ssh-to-age) ----
|
||||
- &host_db age1TODO_db_host_age_pubkey_replace_me
|
||||
- &host_app age1TODO_app_host_age_pubkey_replace_me
|
||||
- &host_ingest age1TODO_mqtt_ingest_host_age_pubkey_replace_me
|
||||
- &host_db age1wpl3vz60tlt880p5fmfedr8c59dm4kf0czsacv0w5my706twuf5q9p0x43
|
||||
- &host_app age1g57pznep69nlyv3sltz6k0sml47v68m03gh68jkqwcq4jdx68vtsvnefq0
|
||||
- &host_ingest age190gu75rf3ra89mhk27xe3tv87tad087altqhugjlhkerqwe2jfqsnu738d
|
||||
|
||||
creation_rules:
|
||||
# Supabase compose stack → app-host only.
|
||||
- path_regex: nix/secrets/secrets\.yaml$
|
||||
encrypted_regex: ^(supabase_|ingester_|postgres_admin_)
|
||||
encrypted_regex: ^(supabase_|ingester_|postgres_admin_|hivemq_)
|
||||
key_groups:
|
||||
- age:
|
||||
- *dev_lars
|
||||
|
||||
@@ -9,15 +9,100 @@ and CI/CD pipelines for the project. Everything is one `flake.nix`.
|
||||
|
||||
Three hosts (see issue #21 for the full design discussion):
|
||||
|
||||
| Host | Role | Public hostname |
|
||||
| ------------- | ----------------------------------------- | ---------------------- |
|
||||
| `mqtt-ingest` | HiveMQ CE + Go MQTT→Postgres ingester | `ingest.ge-bos.de` |
|
||||
| `db-host` | Postgres 17 + TimescaleDB (telemetry+auth)| internal only |
|
||||
| `app-host` | Caddy + Kong + Supabase (compose) + SPA | `app.ge-bos.de`, `api.ge-bos.de` |
|
||||
| Host | Role | Public hostname | Private IP |
|
||||
| ------------- | ----------------------------------------- | ---------------------- | ---------- |
|
||||
| `mqtt-ingest` | HiveMQ CE + Go MQTT→Postgres ingester | `ingest.gebos.online` | `10.0.0.4` |
|
||||
| `db-host` | Postgres 17 + TimescaleDB (telemetry+auth)| `db.gebos.online` (SSH only) | `10.0.0.2` |
|
||||
| `app-host` | Caddy + Kong + Supabase (compose) + SPA | `app.gebos.online`, `api.gebos.online` | `10.0.0.3` |
|
||||
|
||||
Public REST surface is PostgREST + SQL `/rpc/` functions, fronted by Kong, TLS-terminated by Caddy.
|
||||
Supabase Studio is bound to `127.0.0.1` on `app-host` — reach it with `ssh -L 3000:127.0.0.1:3000 app-host`.
|
||||
|
||||
### Networking
|
||||
|
||||
There is no private DNS. Each host has a public hostname (used for `deploy-rs`
|
||||
SSH access from the Gitea runner, which is **not** on the private network, plus
|
||||
TLS ingress where applicable), and a static `10.0.0.0/8` IP used for all
|
||||
host-to-host traffic:
|
||||
|
||||
- `db.gebos.online` is for SSH/deploy only — Postgres is never exposed publicly.
|
||||
- `app-host` and `mqtt-ingest` reach Postgres at `10.0.0.2:5432` over the private
|
||||
network.
|
||||
- `db-host` only accepts Postgres (port 5432) from `10.0.0.0/8` (firewall rule
|
||||
in `nix/hosts/db-host.nix`).
|
||||
|
||||
## MQTT ingest path
|
||||
|
||||
How a sender device's telemetry reaches Postgres, and the reasoning behind each
|
||||
choice. The device firmware is the fixed end of this contract, so the rest of
|
||||
the stack is built to match it rather than the other way around.
|
||||
|
||||
```
|
||||
device ──MQTTS:8883──▶ Caddy (layer4, TLS term) ──MQTT:1883──▶ HiveMQ CE ──▶ ingester ──▶ Postgres
|
||||
acrios/<IMSI>/<metric> loopback file-RBAC IMSI→tenant
|
||||
```
|
||||
|
||||
### Broker auth: file-RBAC, not allow-all
|
||||
|
||||
HiveMQ CE ships the `hivemq-allow-all` extension — any client can publish or
|
||||
subscribe to anything. We replaced it with the vendored
|
||||
[`hivemq-file-rbac-extension`](https://github.com/hivemq/hivemq-file-rbac-extension)
|
||||
(`nix/modules/gebos-hivemq.nix`), which adds username/password auth and
|
||||
topic-level authorization. `gebos-hivemq.preStart` reconciles the writable
|
||||
extensions dir on every start: it installs the pinned extension and **removes
|
||||
allow-all**, so a redeploy fixes already-provisioned boxes too. Credentials
|
||||
(`credentials.xml`) are rendered by sops-nix (`services.gebos.secrets.hivemq`)
|
||||
and symlinked from the `/run/secrets` tmpfs into the extension's `conf/` — they
|
||||
never touch the Nix store or the state dir. `password-type` is `PLAIN` because
|
||||
the file is already an encrypted secret at rest and `0400`/`gebos-hivemq` at
|
||||
runtime; flip to `HASHED` for defence-in-depth.
|
||||
|
||||
There are three broker users: `ingester` (subscribes the device tree), `admin`
|
||||
(break-glass superuser), and `bender` (the shared device account, below).
|
||||
|
||||
### Topic scheme: the device's `acrios/<IMSI>/…` is the source of truth
|
||||
|
||||
Senders publish to `acrios/<IMSI>/<metric>` — `mqtt_topic_base` plus the SIM's
|
||||
IMSI plus the metric name. We adopted that namespace verbatim rather than
|
||||
reshaping it into the `t/<tenant>/d/<device>/…` form the schema originally
|
||||
imagined, because the firmware can't emit our `tenant_id`/`device_id` **UUIDs**
|
||||
— it only knows its IMSI. So the IMSI is the natural device key, and the
|
||||
ingester (subscribed to `acrios/#`) will resolve `IMSI → (tenant_id,
|
||||
device_id)` via a **device registry** before inserting into `public.telemetry`.
|
||||
That registry table and the ingester's topic parsing are still TODO — the
|
||||
ingester is currently a stub.
|
||||
|
||||
### Device auth: shared user now (Option A), per-device later (Option B)
|
||||
|
||||
The sender logs in with a single shared account (`bender`), authorized to
|
||||
publish/subscribe under `acrios/#`. Tenant isolation is therefore enforced
|
||||
**downstream** by the ingester's IMSI registry, not at the broker — any device
|
||||
could publish under any IMSI. That's an accepted trade-off for a small trusted
|
||||
fleet, and it gets data flowing without per-device provisioning.
|
||||
|
||||
The production answer (**Option B**, a TODO in `nix/modules/gebos-secrets.nix`)
|
||||
is one broker user per device with `username == IMSI`, scoped to
|
||||
`acrios/${{username}}/#` so the broker itself prevents a device from spoofing
|
||||
another's IMSI. It's not wired up because it needs a firmware change
|
||||
(`mqtt_user = <IMSI>`) and a per-device password provisioning flow. Note the
|
||||
firmware's client-id (`acrcv-<IMSI>`) carries a prefix the topic doesn't, so
|
||||
per-device scoping must key on username, not `${{clientid}}`.
|
||||
|
||||
### TLS: caddy-l4 terminates, HiveMQ stays loopback
|
||||
|
||||
HiveMQ binds `127.0.0.1:1883` only; it never faces the network. Senders connect
|
||||
MQTTS on `:8883`, and Caddy — rebuilt with the
|
||||
[`caddy-l4`](https://github.com/mholt/caddy-l4) layer4 module — terminates TLS
|
||||
and proxies cleartext to the broker (`nix/modules/gebos-caddy.nix`). We reuse
|
||||
Caddy here (rather than a native HiveMQ TLS listener + keystore) so there's one
|
||||
ACME story for the whole fleet. The wrinkle: caddy-l4's `tls` handler only
|
||||
*terminates* — it never obtains certs — so each `tcpProxy` site also emits a
|
||||
companion HTTPS block purely to make Caddy's automatic HTTPS provision the cert
|
||||
that the layer4 handler then serves by SNI. That's why `mqtt-ingest` opens
|
||||
**80/443** (ACME challenge + renewal) alongside **8883**. The custom Caddy build
|
||||
is only used on hosts that actually have a `tcpProxy` site; app-host's
|
||||
static/reverse-proxy sites stay on stock Caddy.
|
||||
|
||||
## Repo layout
|
||||
|
||||
```
|
||||
@@ -44,10 +129,81 @@ Supabase compose, Kong, Caddy, ingester, Vite dev server). NixOS required.
|
||||
|
||||
## Deployment
|
||||
|
||||
There are two distinct phases. **Initial provisioning** turns a blank box into a
|
||||
NixOS host (`nixos-anywhere`, run once per machine). **Updates** push new
|
||||
closures to a host that already runs NixOS (`deploy-rs`, run on every change).
|
||||
|
||||
### SSH key setup (do this first)
|
||||
|
||||
Both phases authenticate over SSH with `~/.ssh/larsnolden`, which is
|
||||
passphrase-protected. Load it into an `ssh-agent` once so the deploy tools can
|
||||
reuse it without prompting:
|
||||
|
||||
```fish
|
||||
eval (ssh-agent -c) # bash/zsh: eval "$(ssh-agent -s)"
|
||||
ssh-add ~/.ssh/larsnolden # enter the passphrase once
|
||||
ssh-add -l # confirm the key is loaded
|
||||
```
|
||||
|
||||
This is **required** for `deploy-rs`, not just a convenience: with
|
||||
`magicRollback = true` (see `nix/deploy.nix`) activation opens two concurrent SSH
|
||||
connections — the activation command and a rollback waiter. Without an agent,
|
||||
both race to read the passphrase from the terminal, one loses, and the deploy
|
||||
fails with `Permission denied (publickey,keyboard-interactive)` even though
|
||||
manual SSH and the copy step work. The agent serves the key to every connection,
|
||||
so no prompt is needed.
|
||||
|
||||
### Initial provisioning (`nixos-anywhere`)
|
||||
|
||||
`deploy-rs` only *updates* a machine that already runs NixOS — it copies a
|
||||
prebuilt closure and activates it. A fresh box (e.g. a stock Debian image with
|
||||
only a `root` user) has no Nix store and no NixOS generation to switch to, so
|
||||
`deploy-rs` fails with `nix-store: command not found`. Use
|
||||
[`nixos-anywhere`](https://github.com/nix-community/nixos-anywhere) to install
|
||||
NixOS over SSH first; after that, `deploy-rs` takes over for all subsequent
|
||||
deploys.
|
||||
|
||||
`nixos-anywhere` SSHes in as `root`, kexecs into an in-memory NixOS installer,
|
||||
partitions and formats the disk per the host's [`disko`](https://github.com/nix-community/disko)
|
||||
config, installs `nixosConfigurations.<host>`, and reboots into NixOS. **This
|
||||
wipes the target disk.**
|
||||
|
||||
Prerequisites, per host, before running it:
|
||||
|
||||
1. **A real disk layout.** Hosts currently import the fictional
|
||||
`nix/hosts/placeholder-hardware.nix` (it only exists so `nix flake check`
|
||||
evaluates). Replace that import with a `disko` config describing the actual
|
||||
disk device (`/dev/sda` vs `/dev/vda`/nvme) and firmware (UEFI vs legacy
|
||||
BIOS). `disko` replaces the hand-generated `hardware-configuration.nix`.
|
||||
2. **Root SSH access** to the box. The `deploy` user and its authorized keys are
|
||||
created by `nix/hosts/common.nix` during the install, so deploy-rs access
|
||||
works automatically once NixOS is up.
|
||||
3. **Host secrets key** present so sops-nix can decrypt at first boot — see
|
||||
[`nix/secrets/README.md`](nix/secrets/README.md). Otherwise services that
|
||||
read `/run/secrets/*` (e.g. the ingester) fail to start after reboot.
|
||||
|
||||
Then, from the repo root:
|
||||
|
||||
```
|
||||
# installs NixOS onto the target, wiping its disk
|
||||
nix run github:nix-community/nixos-anywhere -- \
|
||||
--flake .#mqtt-ingest root@ingest.gebos.online
|
||||
```
|
||||
|
||||
Repeat with `.#db-host root@db.gebos.online` and `.#app-host root@app.gebos.online`.
|
||||
Provision `db-host` first if you intend to deploy updates immediately afterward
|
||||
(see the ordering note below). Once a host has rebooted into NixOS, never run
|
||||
`nixos-anywhere` against it again — use `deploy-rs`.
|
||||
|
||||
### Updates (`deploy-rs`)
|
||||
|
||||
`deploy-rs` from a Gitea Actions runner on push to `main`. Closures are built
|
||||
once, copied to each host, activated with auto-rollback. Order: `db-host` →
|
||||
`app-host` → `mqtt-ingest`.
|
||||
|
||||
Running it by hand needs the key loaded into an `ssh-agent` first — see
|
||||
[SSH key setup](#ssh-key-setup-do-this-first) above.
|
||||
|
||||
```
|
||||
nix run github:serokell/deploy-rs -- .#db-host # one host
|
||||
nix run github:serokell/deploy-rs -- . # all hosts
|
||||
|
||||
Generated
+47
-5
@@ -20,6 +20,26 @@
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"disko": {
|
||||
"inputs": {
|
||||
"nixpkgs": [
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1781152676,
|
||||
"narHash": "sha256-RxWs5ND31KzTG7wvMM+PMfUjyNpmIEr999lqNARaM5o=",
|
||||
"owner": "nix-community",
|
||||
"repo": "disko",
|
||||
"rev": "ff8702b4de27f72b4c78573dfb89ec74e36abdf1",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-community",
|
||||
"repo": "disko",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"flake-compat": {
|
||||
"flake": false,
|
||||
"locked": {
|
||||
@@ -87,16 +107,16 @@
|
||||
},
|
||||
"nixpkgs_2": {
|
||||
"locked": {
|
||||
"lastModified": 1779560665,
|
||||
"narHash": "sha256-tpyBcxPpcQb8ukyNF7DoCwfSY3VPsxHoYwj00Cayv5o=",
|
||||
"lastModified": 1782233679,
|
||||
"narHash": "sha256-QyuGP5+QOtmXpy4i2X4DhBVBaySBdDKQEhqKcphcp34=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "64c08a7ca051951c8eae34e3e3cb1e202fe36786",
|
||||
"rev": "667d5cf1c59585031d743c78b394b0a647537c35",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixos-unstable",
|
||||
"ref": "nixos-26.05",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
@@ -119,10 +139,12 @@
|
||||
"root": {
|
||||
"inputs": {
|
||||
"deploy-rs": "deploy-rs",
|
||||
"disko": "disko",
|
||||
"flake-parts": "flake-parts",
|
||||
"nixpkgs": "nixpkgs_2",
|
||||
"process-compose-flake": "process-compose-flake",
|
||||
"services-flake": "services-flake"
|
||||
"services-flake": "services-flake",
|
||||
"sops-nix": "sops-nix"
|
||||
}
|
||||
},
|
||||
"services-flake": {
|
||||
@@ -140,6 +162,26 @@
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"sops-nix": {
|
||||
"inputs": {
|
||||
"nixpkgs": [
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1782165805,
|
||||
"narHash": "sha256-478kKQBvK6SYTOdN2h9jhKJv94nbXRbFMfuL1WshErg=",
|
||||
"owner": "Mic92",
|
||||
"repo": "sops-nix",
|
||||
"rev": "56b24064fdcaedca53553b1a6d607fd23b613a24",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "Mic92",
|
||||
"repo": "sops-nix",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"systems": {
|
||||
"locked": {
|
||||
"lastModified": 1681028828,
|
||||
|
||||
@@ -2,11 +2,15 @@
|
||||
description = "Gebos — IoT telemetry stack on bare-metal NixOS";
|
||||
|
||||
inputs = {
|
||||
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
|
||||
nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
|
||||
flake-parts.url = "github:hercules-ci/flake-parts";
|
||||
deploy-rs.url = "github:serokell/deploy-rs";
|
||||
process-compose-flake.url = "github:Platonic-Systems/process-compose-flake";
|
||||
services-flake.url = "github:juspay/services-flake";
|
||||
disko = {
|
||||
url = "github:nix-community/disko";
|
||||
inputs.nixpkgs.follows = "nixpkgs";
|
||||
};
|
||||
sops-nix = {
|
||||
url = "github:Mic92/sops-nix";
|
||||
inputs.nixpkgs.follows = "nixpkgs";
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
{ buildNpmPackage, lib }:
|
||||
{ buildNpmPackage }:
|
||||
|
||||
buildNpmPackage {
|
||||
pname = "gebos-frontend";
|
||||
version = "0.0.0";
|
||||
src = ./.;
|
||||
|
||||
# Filled in once package-lock.json exists.
|
||||
npmDepsHash = lib.fakeHash;
|
||||
npmDepsHash = "sha256-GeXgNbf94QRLNaZP8ma2vx19Vzm8jDWPeOTvE9exwm0=";
|
||||
|
||||
installPhase = ''
|
||||
runHook preInstall
|
||||
|
||||
@@ -11,7 +11,7 @@ const DEV_SUPABASE_ANON_KEY =
|
||||
|
||||
const SUPABASE_URL =
|
||||
import.meta.env.VITE_SUPABASE_URL ??
|
||||
(import.meta.env.DEV ? DEV_SUPABASE_URL : "https://api.ge-bos.de");
|
||||
(import.meta.env.DEV ? DEV_SUPABASE_URL : "https://api.gebos.online");
|
||||
|
||||
const SUPABASE_ANON_KEY =
|
||||
import.meta.env.VITE_SUPABASE_ANON_KEY ??
|
||||
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
/// <reference types="vite/client" />
|
||||
@@ -8,6 +8,10 @@ buildGoModule {
|
||||
# Set once `go mod tidy` has produced a real go.sum.
|
||||
vendorHash = null;
|
||||
|
||||
# The main package lives in cmd/gebos-ingester/, so `go install` names the
|
||||
# binary after that directory — matching mainProgram and the systemd unit.
|
||||
subPackages = [ "cmd/gebos-ingester" ];
|
||||
|
||||
meta = {
|
||||
description = "MQTT → Postgres telemetry ingester for Gebos";
|
||||
mainProgram = "gebos-ingester";
|
||||
|
||||
+6
-3
@@ -18,8 +18,11 @@ in
|
||||
|
||||
nodes = {
|
||||
# Order matters: the Gitea Actions workflow invokes them in this sequence.
|
||||
db-host = mkNode "db-host" "db-host.gebos.internal"; # TODO: real address
|
||||
app-host = mkNode "app-host" "app-host.gebos.internal"; # TODO: real address
|
||||
mqtt-ingest = mkNode "mqtt-ingest" "ingest.ge-bos.de";
|
||||
# SSH/deploy targets use public hostnames because the Gitea runner is not on
|
||||
# the private network. Host-to-host traffic still uses the 10.0.0.0/8 IPs.
|
||||
# See the host address table in README.md.
|
||||
db-host = mkNode "db-host" "db.gebos.online";
|
||||
app-host = mkNode "app-host" "app.gebos.online";
|
||||
mqtt-ingest = mkNode "mqtt-ingest" "ingest.gebos.online";
|
||||
};
|
||||
}
|
||||
|
||||
@@ -38,6 +38,13 @@ let
|
||||
VITE_SUPABASE_URL = dev.supabaseUrl;
|
||||
VITE_SUPABASE_ANON_KEY = dev.anonKey;
|
||||
};
|
||||
|
||||
# services-flake ships no MQTT broker, so run mosquitto as a plain
|
||||
# process-compose process. Anonymous access on the dev listener only.
|
||||
mosquittoConf = pkgs.writeText "mosquitto-dev.conf" ''
|
||||
listener ${toString dev.mqttPort} ${dev.mqttHost}
|
||||
allow_anonymous true
|
||||
'';
|
||||
in
|
||||
{
|
||||
imports = [ services-flake.processComposeModules.default ];
|
||||
@@ -48,16 +55,20 @@ in
|
||||
port = dev.pgPort;
|
||||
initialDatabases = [{ name = dev.pgDB; }];
|
||||
};
|
||||
|
||||
mosquitto."broker" = {
|
||||
enable = true;
|
||||
port = dev.mqttPort;
|
||||
};
|
||||
};
|
||||
|
||||
settings.processes = {
|
||||
broker = {
|
||||
command = "${pkgs.mosquitto}/bin/mosquitto -c ${mosquittoConf}";
|
||||
readiness_probe = {
|
||||
exec.command = "${pkgs.mosquitto}/bin/mosquitto_sub -h ${dev.mqttHost} -p ${toString dev.mqttPort} -t '$$SYS/#' -C 1 -W 2";
|
||||
initial_delay_seconds = 1;
|
||||
period_seconds = 2;
|
||||
};
|
||||
};
|
||||
|
||||
ingester = {
|
||||
command = "${pkgs.go}/bin/go run ./ingester";
|
||||
command = "${pkgs.go}/bin/go run ./ingester/cmd/gebos-ingester";
|
||||
environment = ingesterEnv;
|
||||
depends_on = {
|
||||
"db".condition = "process_healthy";
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
services.gebos.supabase = {
|
||||
enable = true;
|
||||
# Compose stack points at external db-host instead of the bundled `db` service.
|
||||
postgresHost = "db-host.gebos.internal"; # TODO: real internal hostname
|
||||
postgresHost = "10.0.0.2"; # db-host on the private network
|
||||
postgresPort = 5432;
|
||||
# Studio bound to loopback only — reach via `ssh -L 3000:127.0.0.1:3000 app-host`.
|
||||
studioBindAddress = "127.0.0.1";
|
||||
@@ -18,8 +18,8 @@
|
||||
enable = true;
|
||||
# Caddy terminates TLS, proxies to Kong on 127.0.0.1:8000.
|
||||
sites = {
|
||||
"api.ge-bos.de" = { upstream = "127.0.0.1:8000"; }; # → Kong → PostgREST/GoTrue
|
||||
"app.ge-bos.de" = { staticRoot = "${config.services.gebos.frontend.package}/share/frontend"; };
|
||||
"api.gebos.online" = { upstream = "127.0.0.1:8000"; }; # → Kong → PostgREST/GoTrue
|
||||
"app.gebos.online" = { staticRoot = "${config.services.gebos.frontend.package}/share/frontend"; };
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
+15
-1
@@ -1,7 +1,17 @@
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
{
|
||||
system.stateVersion = "25.05";
|
||||
system.stateVersion = "26.05";
|
||||
|
||||
# Bootloader. The disko layout (hardware-configurations/disk-config.nix) lays
|
||||
# down both a BIOS-boot (EF02) partition and an ESP mounted at /boot, so grub
|
||||
# works whether the VM firmware is legacy BIOS or UEFI. disko fills in
|
||||
# boot.loader.grub.devices from the EF02 partition; we just enable grub here.
|
||||
boot.loader.grub = {
|
||||
enable = true;
|
||||
efiSupport = true;
|
||||
efiInstallAsRemovable = true;
|
||||
};
|
||||
|
||||
nix.settings = {
|
||||
experimental-features = [ "nix-command" "flakes" ];
|
||||
@@ -23,6 +33,10 @@
|
||||
enable = true;
|
||||
settings.PasswordAuthentication = false;
|
||||
};
|
||||
# TODO: restrict inbound SSH (port 22) to the Gitea deployment runner's IP
|
||||
# only, instead of leaving it open to the world. Likely via
|
||||
# services.openssh.openFirewall = false + a firewall.extraInputRules rule
|
||||
# accepting tcp dport 22 from the runner's source address.
|
||||
|
||||
networking.firewall.enable = true;
|
||||
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
{
|
||||
# Postgres data lives on the dedicated second disk (/dev/sdb), not the OS disk.
|
||||
# The shared hardware-configuration.nix + sda disk layout come from ./default.nix.
|
||||
imports = [ ./hardware-configurations/db-data-disk.nix ];
|
||||
|
||||
networking.hostName = "db-host";
|
||||
|
||||
services.gebos.secrets.postgresAdmin = true;
|
||||
|
||||
# Skeleton: real hardware-configuration.nix + boot/fs lives next to this file
|
||||
# once we have the actual box. Marked here so the structure is visible.
|
||||
# imports = [ ./db-host.hardware.nix ];
|
||||
|
||||
services.gebos.postgres = {
|
||||
enable = true;
|
||||
listenAddresses = [ "0.0.0.0" ];
|
||||
|
||||
@@ -10,6 +10,14 @@ let
|
||||
modules = [
|
||||
./common.nix
|
||||
inputs.sops-nix.nixosModules.sops
|
||||
inputs.disko.nixosModules.disko
|
||||
# All three boxes are identical qemu guests, so they share one
|
||||
# hardware-configuration.nix and one disko disk layout (single
|
||||
# /dev/sda → GPT → LVM → ext4 root). disko derives fileSystems +
|
||||
# bootloader devices from disk-config.nix. db-host additionally
|
||||
# imports the /dev/sdb data disk from ./db-host.nix.
|
||||
./hardware-configurations/hardware-configuration.nix
|
||||
./hardware-configurations/disk-config.nix
|
||||
modules.gebos-secrets
|
||||
modules.gebos-postgres
|
||||
modules.gebos-supabase
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
{ lib, ... }:
|
||||
|
||||
# Second disk for db-host: the Postgres data directory lives on /dev/sdb, kept
|
||||
# separate from the OS disk (/dev/sda). The whole disk is one GPT partition,
|
||||
# ext4, mounted at /var/lib/postgresql — so the cluster data dir
|
||||
# (/var/lib/postgresql/17, the upstream NixOS default) sits on this drive and
|
||||
# survives an OS-disk rebuild.
|
||||
{
|
||||
disko.devices.disk.disk2 = {
|
||||
device = lib.mkDefault "/dev/sdb";
|
||||
type = "disk";
|
||||
content = {
|
||||
type = "gpt";
|
||||
partitions = {
|
||||
data = {
|
||||
name = "data";
|
||||
size = "100%";
|
||||
content = {
|
||||
type = "filesystem";
|
||||
format = "ext4";
|
||||
mountpoint = "/var/lib/postgresql";
|
||||
mountOptions = [ "defaults" ];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
{ lib, ... }:
|
||||
{
|
||||
disko.devices = {
|
||||
disk.disk1 = {
|
||||
device = lib.mkDefault "/dev/sda";
|
||||
type = "disk";
|
||||
content = {
|
||||
type = "gpt";
|
||||
partitions = {
|
||||
boot = {
|
||||
name = "boot";
|
||||
size = "1M";
|
||||
type = "EF02";
|
||||
};
|
||||
esp = {
|
||||
name = "ESP";
|
||||
size = "500M";
|
||||
type = "EF00";
|
||||
content = {
|
||||
type = "filesystem";
|
||||
format = "vfat";
|
||||
mountpoint = "/boot";
|
||||
};
|
||||
};
|
||||
root = {
|
||||
name = "root";
|
||||
size = "100%";
|
||||
content = {
|
||||
type = "lvm_pv";
|
||||
vg = "pool";
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
lvm_vg = {
|
||||
pool = {
|
||||
type = "lvm_vg";
|
||||
lvs = {
|
||||
root = {
|
||||
size = "100%FREE";
|
||||
content = {
|
||||
type = "filesystem";
|
||||
format = "ext4";
|
||||
mountpoint = "/";
|
||||
mountOptions = [
|
||||
"defaults"
|
||||
];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{ config, lib, pkgs, modulesPath, ... }:
|
||||
|
||||
{
|
||||
imports =
|
||||
[ (modulesPath + "/profiles/qemu-guest.nix")
|
||||
];
|
||||
|
||||
boot.initrd.availableKernelModules = [ "ahci" "xhci_pci" "virtio_pci" "virtio_scsi" "sd_mod" "sr_mod" ];
|
||||
boot.initrd.kernelModules = [ "dm-snapshot" ];
|
||||
boot.kernelModules = [ ];
|
||||
boot.extraModulePackages = [ ];
|
||||
|
||||
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
|
||||
}
|
||||
@@ -4,6 +4,8 @@
|
||||
networking.hostName = "mqtt-ingest";
|
||||
|
||||
services.gebos.secrets.ingester = true;
|
||||
# Broker authn/authz credentials (file-RBAC: ingester + admin MQTT users).
|
||||
services.gebos.secrets.hivemq = true;
|
||||
|
||||
services.gebos.hivemq = {
|
||||
enable = true;
|
||||
@@ -15,17 +17,20 @@
|
||||
services.gebos.ingester = {
|
||||
enable = true;
|
||||
# mqttBroker defaults to tcp://127.0.0.1:1883 (the local HiveMQ).
|
||||
postgresUrl = "postgres://gebos_ingest@db-host.gebos.internal:5432/postgres?sslmode=require";
|
||||
postgresUrl = "postgres://gebos_ingest@10.0.0.2:5432/postgres?sslmode=require";
|
||||
# GEBOS_POSTGRES_PASSWORD sourced from /run/secrets/gebos-env (sops-nix).
|
||||
};
|
||||
|
||||
# Caddy fronts TLS for `ingest.ge-bos.de` and forwards to HiveMQ.
|
||||
# Caddy (layer4) terminates TLS on :8883 and forwards cleartext MQTT to
|
||||
# HiveMQ's loopback listener on :1883. tcpProxy is the upstream backend.
|
||||
services.gebos.caddy = {
|
||||
enable = true;
|
||||
sites = {
|
||||
"ingest.ge-bos.de" = { tcpProxy = "127.0.0.1:8883"; };
|
||||
"ingest.gebos.online" = { tcpProxy = "127.0.0.1:1883"; };
|
||||
};
|
||||
};
|
||||
|
||||
networking.firewall.allowedTCPPorts = [ 8883 ];
|
||||
# 8883: MQTTS for senders. 80/443: ACME challenge so Caddy can obtain/renew
|
||||
# the ingest.gebos.online cert that the layer4 TLS handler serves.
|
||||
networking.firewall.allowedTCPPorts = [ 80 443 8883 ];
|
||||
}
|
||||
|
||||
+56
-11
@@ -3,7 +3,24 @@
|
||||
let
|
||||
cfg = config.services.gebos.caddy;
|
||||
|
||||
siteBlock = host: site:
|
||||
tcpProxySites = lib.filterAttrs (_: s: s ? tcpProxy) cfg.sites;
|
||||
httpSites = lib.filterAttrs (_: s: !(s ? tcpProxy)) cfg.sites;
|
||||
needsLayer4 = tcpProxySites != { };
|
||||
|
||||
# External MQTTS port the senders connect to (mqtt_port = 8883, SSL).
|
||||
tlsListenPort = 8883;
|
||||
|
||||
# Caddy built with the layer4 module (caddy-l4) — needed to TLS-terminate raw
|
||||
# MQTT and proxy it to HiveMQ, which Caddy's HTTP-only core can't do. Only
|
||||
# built when a host actually has a tcpProxy site; app-host's static/upstream
|
||||
# sites run on stock Caddy so they don't pay for the custom build.
|
||||
caddyWithL4 = pkgs.caddy.withPlugins {
|
||||
plugins = [ "github.com/mholt/caddy-l4@v0.1.1" ];
|
||||
hash = "sha256-O6GuC2q1mA/Fa0utb2Yg7ZE73iq13oVYhJI1IVyOvog=";
|
||||
};
|
||||
|
||||
# HTTP-style sites: static file server or reverse proxy.
|
||||
httpSiteBlock = host: site:
|
||||
if site ? staticRoot then ''
|
||||
${host} {
|
||||
root * ${site.staticRoot}
|
||||
@@ -18,14 +35,38 @@ let
|
||||
encode zstd gzip
|
||||
}
|
||||
''
|
||||
else if site ? tcpProxy then ''
|
||||
${host}:8883 {
|
||||
# TLS termination for MQTT — Caddy's `layer4` app would be ideal here;
|
||||
# for now, document that the upstream HiveMQ port is ${site.tcpProxy}.
|
||||
# TODO: switch to the caddy-l4 module or terminate TLS in HiveMQ directly.
|
||||
}
|
||||
''
|
||||
else throw "site `${host}` needs one of: staticRoot, upstream, tcpProxy";
|
||||
|
||||
# A tcpProxy site needs TWO Caddyfile pieces:
|
||||
# 1. a layer4 listener (in the global options block) that terminates TLS on
|
||||
# :8883 and proxies cleartext to the upstream (HiveMQ on 127.0.0.1:1883);
|
||||
# 2. a normal HTTPS site block for the host, purely so Caddy's automatic
|
||||
# HTTPS OBTAINS the ACME cert that the layer4 `tls` handler then serves by
|
||||
# SNI — caddy-l4's `tls` handler only terminates, it never provisions
|
||||
# certs. This is why ports 80/443 must stay open for the ACME challenge.
|
||||
layer4Listener = _host: site: ''
|
||||
:${toString tlsListenPort} {
|
||||
route {
|
||||
tls
|
||||
proxy ${site.tcpProxy}
|
||||
}
|
||||
}
|
||||
'';
|
||||
|
||||
certSiteBlock = host: _site: ''
|
||||
${host} {
|
||||
respond "gebos MQTT ingest — connect MQTTS on :${toString tlsListenPort}" 200
|
||||
}
|
||||
'';
|
||||
|
||||
# Caddyfile global options block must come first and exist once; we fold every
|
||||
# layer4 listener into a single `layer4 { ... }` inside it.
|
||||
globalOptions = lib.optionalString needsLayer4 ''
|
||||
{
|
||||
layer4 {
|
||||
${lib.concatStrings (lib.mapAttrsToList layer4Listener tcpProxySites)} }
|
||||
}
|
||||
'';
|
||||
in
|
||||
{
|
||||
options.services.gebos.caddy = {
|
||||
@@ -39,9 +80,13 @@ in
|
||||
config = lib.mkIf cfg.enable {
|
||||
services.caddy = {
|
||||
enable = true;
|
||||
email = "ops@ge-bos.de"; # TODO: confirm ACME contact
|
||||
extraConfig = lib.concatStringsSep "\n"
|
||||
(lib.mapAttrsToList siteBlock cfg.sites);
|
||||
package = if needsLayer4 then caddyWithL4 else pkgs.caddy;
|
||||
email = "ops@gebos.online"; # TODO: confirm ACME contact
|
||||
extraConfig = lib.concatStringsSep "\n" (
|
||||
[ globalOptions ]
|
||||
++ lib.mapAttrsToList httpSiteBlock httpSites
|
||||
++ lib.mapAttrsToList certSiteBlock tcpProxySites
|
||||
);
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
@@ -2,31 +2,274 @@
|
||||
|
||||
let
|
||||
cfg = config.services.gebos.hivemq;
|
||||
|
||||
# HiveMQ CE — fetched from the upstream GitHub release and unpacked into the
|
||||
# Nix store. The store tree is read-only; the runtime data/log/extension
|
||||
# folders are redirected to the writable state dir (see ExecStart below).
|
||||
hivemq-ce = pkgs.stdenvNoCC.mkDerivation rec {
|
||||
pname = "hivemq-ce";
|
||||
version = "2026.5";
|
||||
|
||||
src = pkgs.fetchurl {
|
||||
url = "https://github.com/hivemq/hivemq-community-edition/releases/download/${version}/hivemq-ce-${version}.zip";
|
||||
hash = "sha256-/QBeU17KC6/7CF4RsVEohdBs4AeG+WFwHHZEmBb33sU=";
|
||||
};
|
||||
|
||||
nativeBuildInputs = [ pkgs.unzip ];
|
||||
dontConfigure = true;
|
||||
dontBuild = true;
|
||||
|
||||
installPhase = ''
|
||||
runHook preInstall
|
||||
mkdir -p "$out"
|
||||
cp -r . "$out/"
|
||||
runHook postInstall
|
||||
'';
|
||||
|
||||
meta = with lib; {
|
||||
description = "HiveMQ Community Edition — open-source MQTT 3.x / 5 broker";
|
||||
homepage = "https://github.com/hivemq/hivemq-community-edition";
|
||||
license = licenses.asl20;
|
||||
platforms = platforms.linux;
|
||||
};
|
||||
};
|
||||
|
||||
# HiveMQ file-RBAC extension — adds username/password authentication and
|
||||
# topic-level authorization, replacing the insecure hivemq-allow-all that
|
||||
# ships in the broker package. The zip unpacks to a single top-level
|
||||
# `hivemq-file-rbac-extension/` folder which we keep verbatim in $out.
|
||||
hivemq-file-rbac = pkgs.stdenvNoCC.mkDerivation rec {
|
||||
pname = "hivemq-file-rbac-extension";
|
||||
version = "4.6.15";
|
||||
|
||||
src = pkgs.fetchurl {
|
||||
url = "https://github.com/hivemq/hivemq-file-rbac-extension/releases/download/${version}/hivemq-file-rbac-extension-${version}.zip";
|
||||
hash = "sha256-R6PM+qAvEKMHdX3FTR0CfR8yarJrc2WeFAEu4eXH3Ps=";
|
||||
};
|
||||
|
||||
nativeBuildInputs = [ pkgs.unzip ];
|
||||
dontConfigure = true;
|
||||
dontBuild = true;
|
||||
|
||||
# stdenv unpacks the single top-level archive dir and cds into it, so $PWD
|
||||
# already holds the extension's files (jar, conf/, hivemq-extension.xml).
|
||||
installPhase = ''
|
||||
runHook preInstall
|
||||
mkdir -p "$out/hivemq-file-rbac-extension"
|
||||
cp -r . "$out/hivemq-file-rbac-extension/"
|
||||
runHook postInstall
|
||||
'';
|
||||
|
||||
meta = with lib; {
|
||||
description = "HiveMQ file-based role-based access control extension";
|
||||
homepage = "https://github.com/hivemq/hivemq-file-rbac-extension";
|
||||
license = licenses.asl20;
|
||||
platforms = platforms.linux;
|
||||
};
|
||||
};
|
||||
|
||||
stateDir = "/var/lib/gebos-hivemq";
|
||||
|
||||
# Non-secret RBAC extension config (reload interval + password format). The
|
||||
# users/roles live in the secret credentials.xml, symlinked in at runtime.
|
||||
rbacConfigXml = pkgs.writeText "hivemq-file-rbac-config.xml" ''
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<extension-configuration>
|
||||
<credentials-reload-interval>${toString cfg.rbac.reloadInterval}</credentials-reload-interval>
|
||||
<password-type>${cfg.rbac.passwordType}</password-type>
|
||||
</extension-configuration>
|
||||
'';
|
||||
|
||||
# config.xml rendered from the module options. The local TCP listener is what
|
||||
# the ingester (and, eventually, the Caddy TLS frontend) connects to.
|
||||
configXml = pkgs.writeText "hivemq-config.xml" ''
|
||||
<?xml version="1.0"?>
|
||||
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:noNamespaceSchemaLocation="config.xsd">
|
||||
<listeners>
|
||||
<tcp-listener>
|
||||
<port>${toString cfg.port}</port>
|
||||
<bind-address>${cfg.bindAddress}</bind-address>
|
||||
</tcp-listener>
|
||||
</listeners>
|
||||
<persistence>
|
||||
<mode>${if cfg.persistentSessions then "file" else "in-memory"}</mode>
|
||||
</persistence>
|
||||
<anonymous-usage-statistics>
|
||||
<enabled>false</enabled>
|
||||
</anonymous-usage-statistics>
|
||||
</hivemq>
|
||||
'';
|
||||
|
||||
# Read-only conf folder: our generated config.xml plus the upstream
|
||||
# logback.xml / config.xsd so HiveMQ logs and validates as shipped.
|
||||
confDir = pkgs.runCommand "hivemq-conf" { } ''
|
||||
mkdir -p "$out"
|
||||
cp ${configXml} "$out/config.xml"
|
||||
cp ${hivemq-ce}/conf/logback.xml "$out/logback.xml"
|
||||
cp ${hivemq-ce}/conf/config.xsd "$out/config.xsd"
|
||||
'';
|
||||
|
||||
# run.sh's JVM flags, reproduced so we can invoke java directly and point
|
||||
# HiveMQ at the writable state dir via -Dhivemq.*.folder.
|
||||
javaOpts = lib.concatStringsSep " " [
|
||||
"-Djava.net.preferIPv4Stack=true"
|
||||
"--add-opens java.base/java.lang=ALL-UNNAMED"
|
||||
"--add-opens java.base/java.nio=ALL-UNNAMED"
|
||||
"--add-opens java.base/sun.nio.ch=ALL-UNNAMED"
|
||||
"--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED"
|
||||
"--add-exports java.base/jdk.internal.misc=ALL-UNNAMED"
|
||||
"-Djava.security.egd=file:/dev/./urandom"
|
||||
"-Duser.language=en -Duser.region=US"
|
||||
"-XX:+CrashOnOutOfMemoryError"
|
||||
"-XX:+HeapDumpOnOutOfMemoryError"
|
||||
"-XX:HeapDumpPath=${stateDir}/log/heap-dump.hprof"
|
||||
"-XX:ErrorFile=${stateDir}/log/hs_err_pid%p.log"
|
||||
];
|
||||
in
|
||||
{
|
||||
options.services.gebos.hivemq = {
|
||||
enable = lib.mkEnableOption "HiveMQ CE single-node broker";
|
||||
|
||||
package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
default = hivemq-ce;
|
||||
defaultText = lib.literalExpression "pkgs.hivemq-ce (vendored)";
|
||||
description = "HiveMQ CE package to run.";
|
||||
};
|
||||
|
||||
port = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 1883;
|
||||
description = "Plain MQTT TCP listener port (what the ingester connects to).";
|
||||
};
|
||||
|
||||
bindAddress = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "127.0.0.1";
|
||||
description = ''
|
||||
Address the MQTT listener binds to. Defaults to loopback: external
|
||||
clients reach the broker through the Caddy TLS frontend, not directly.
|
||||
'';
|
||||
};
|
||||
|
||||
persistentSessions = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = true;
|
||||
description = ''
|
||||
Whether client sessions and queued messages survive a restart
|
||||
(file persistence) or are kept only in memory.
|
||||
'';
|
||||
};
|
||||
|
||||
tlsPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 8883;
|
||||
description = ''
|
||||
Externally advertised MQTTS port. TLS is currently terminated by the
|
||||
Caddy frontend (see gebos-caddy), not by HiveMQ directly, so this value
|
||||
is informational until a keystore is wired into the broker.
|
||||
'';
|
||||
};
|
||||
|
||||
rbac = {
|
||||
package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
default = hivemq-file-rbac;
|
||||
defaultText = lib.literalExpression "pkgs.hivemq-file-rbac (vendored)";
|
||||
description = "HiveMQ file-RBAC authn/authz extension package.";
|
||||
};
|
||||
|
||||
passwordType = lib.mkOption {
|
||||
type = lib.types.enum [ "HASHED" "PLAIN" ];
|
||||
default = "PLAIN";
|
||||
description = ''
|
||||
Whether passwords in the credentials file are PBKDF2 HASHED or PLAIN
|
||||
text. PLAIN is acceptable here because the credentials file is a
|
||||
sops-encrypted secret at rest and rendered 0400/gebos-hivemq into a
|
||||
tmpfs at runtime; switch to HASHED for defence-in-depth.
|
||||
'';
|
||||
};
|
||||
|
||||
reloadInterval = lib.mkOption {
|
||||
type = lib.types.ints.positive;
|
||||
default = 60;
|
||||
description = "Seconds between credentials.xml reloads by the extension.";
|
||||
};
|
||||
|
||||
credentialsFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
default = config.services.gebos.secrets.hivemqCredentialsFile;
|
||||
defaultText = lib.literalExpression "config.services.gebos.secrets.hivemqCredentialsFile";
|
||||
description = ''
|
||||
Path to the file-RBAC credentials.xml (users, roles, topic
|
||||
permissions). Defaults to the sops-rendered secret; symlinked into
|
||||
the extension's conf/ dir so it never lands in the writable state dir.
|
||||
'';
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
# TODO: package HiveMQ CE (download tarball + JRE wrapper) and wire as a
|
||||
# systemd unit. Until then this module only declares options so dependent
|
||||
# hosts can be evaluated.
|
||||
users.users.gebos-hivemq = {
|
||||
isSystemUser = true;
|
||||
group = "gebos-hivemq";
|
||||
description = "Gebos HiveMQ CE service user";
|
||||
};
|
||||
users.groups.gebos-hivemq = { };
|
||||
|
||||
systemd.services.gebos-hivemq = {
|
||||
description = "HiveMQ CE";
|
||||
after = [ "network.target" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
|
||||
# HiveMQ wants the extensions folder writable (it manages enable/disable
|
||||
# markers there), so the extension lives under the state dir. We reconcile
|
||||
# it on every start: (re)install the vendored file-RBAC extension when its
|
||||
# store path changes, refresh its non-secret config, link the secret
|
||||
# credentials from the sops tmpfs path, and remove the insecure allow-all
|
||||
# extension that earlier deploys seeded from the broker package.
|
||||
preStart = ''
|
||||
set -eu
|
||||
mkdir -p ${stateDir}/{data,log,extensions}
|
||||
|
||||
extDir=${stateDir}/extensions/hivemq-file-rbac-extension
|
||||
marker=${stateDir}/extensions/.rbac-version
|
||||
|
||||
if [ "$(cat "$marker" 2>/dev/null || true)" != "${cfg.rbac.package}" ]; then
|
||||
rm -rf "$extDir"
|
||||
cp -r --no-preserve=mode ${cfg.rbac.package}/hivemq-file-rbac-extension "$extDir"
|
||||
echo "${cfg.rbac.package}" > "$marker"
|
||||
fi
|
||||
|
||||
cp --no-preserve=mode ${rbacConfigXml} "$extDir/conf/config.xml"
|
||||
ln -sf ${cfg.rbac.credentialsFile} "$extDir/conf/credentials.xml"
|
||||
|
||||
rm -rf ${stateDir}/extensions/hivemq-allow-all-extension
|
||||
'';
|
||||
|
||||
serviceConfig = {
|
||||
Type = "simple";
|
||||
ExecStart = "${pkgs.coreutils}/bin/true"; # TODO: real HiveMQ command
|
||||
User = "gebos-hivemq";
|
||||
Group = "gebos-hivemq";
|
||||
StateDirectory = "gebos-hivemq";
|
||||
WorkingDirectory = stateDir;
|
||||
ExecStart = ''
|
||||
${pkgs.jre_headless}/bin/java \
|
||||
-Dhivemq.home=${cfg.package} \
|
||||
-Dhivemq.config.folder=${confDir} \
|
||||
-Dhivemq.log.folder=${stateDir}/log \
|
||||
-Dhivemq.data.folder=${stateDir}/data \
|
||||
-Dhivemq.extensions.folder=${stateDir}/extensions \
|
||||
${javaOpts} \
|
||||
-jar ${cfg.package}/bin/hivemq.jar
|
||||
'';
|
||||
Restart = "on-failure";
|
||||
RestartSec = "5s";
|
||||
# HiveMQ exits 143 on SIGTERM shutdown — treat that as success.
|
||||
SuccessExitStatus = "143";
|
||||
KillSignal = "SIGTERM";
|
||||
LimitNOFILE = 65536;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
@@ -13,12 +13,20 @@ in
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
# TimescaleDB ships under the non-OSI "Timescale License" (TSL), which
|
||||
# nixpkgs flags as unfree. Allow just this package rather than opening the
|
||||
# whole config to unfree.
|
||||
nixpkgs.config.allowUnfreePredicate = pkg:
|
||||
lib.elem (lib.getName pkg) [ "timescaledb" ];
|
||||
|
||||
services.postgresql = {
|
||||
enable = true;
|
||||
package = pkgs.postgresql_17;
|
||||
enableTCPIP = true;
|
||||
# listen_addresses is managed explicitly below. The upstream postgresql
|
||||
# module also defines it (via its own default / enableTCPIP), so mkForce
|
||||
# makes this module the single source of truth and avoids the collision.
|
||||
settings = {
|
||||
listen_addresses = lib.concatStringsSep "," cfg.listenAddresses;
|
||||
listen_addresses = lib.mkForce (lib.concatStringsSep "," cfg.listenAddresses);
|
||||
shared_preload_libraries = "timescaledb,pg_stat_statements";
|
||||
};
|
||||
extensions = ps: with ps; [
|
||||
|
||||
@@ -17,6 +17,76 @@ let
|
||||
];
|
||||
ingesterKeys = [ "ingester_postgres_password" ];
|
||||
postgresAdminKeys = [ "postgres_admin_password" ];
|
||||
hivemqKeys = [ "hivemq_ingester_password" "hivemq_admin_password" "hivemq_device_password" ];
|
||||
|
||||
# HiveMQ file-RBAC credentials.xml. Only the passwords are secret; the role/
|
||||
# topic policy is structural. Devices publish under `acrios/<IMSI>/<metric>`
|
||||
# (set by the sender firmware's mqtt_topic_base), so that is the namespace the
|
||||
# ingester consumes and the device role is scoped to.
|
||||
#
|
||||
# Roles:
|
||||
# ingest — the Go ingester; SUBSCRIBE the whole device tree.
|
||||
# device — physical senders; full access under acrios/ (publish telemetry
|
||||
# + the LWT/status and any downlink topics the firmware uses).
|
||||
# superuser — operational break-glass.
|
||||
#
|
||||
# Option A (current): one shared `device` user (`bender`) for the whole fleet.
|
||||
# Tenant isolation is enforced downstream by the ingester's IMSI→tenant
|
||||
# registry lookup, NOT at the broker — any device could publish under any IMSI.
|
||||
#
|
||||
# TODO(Option B / per-device auth): give each sender its own RBAC user with
|
||||
# username == IMSI and scope its role to `acrios/${{username}}/#`, so the
|
||||
# broker enforces that a device can only publish under its own IMSI (real
|
||||
# anti-spoofing + per-device revocation). Requires setting mqtt_user = <IMSI>
|
||||
# on each device and a per-device password provisioning flow (generate → sops
|
||||
# → push to device). file-RBAC hot-reloads, so added users need no restart.
|
||||
hivemqCredentialsXml = ingesterPassword: adminPassword: devicePassword: ''
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<file-rbac>
|
||||
<users>
|
||||
<user>
|
||||
<name>ingester</name>
|
||||
<password>${ingesterPassword}</password>
|
||||
<roles><id>ingest</id></roles>
|
||||
</user>
|
||||
<user>
|
||||
<name>admin</name>
|
||||
<password>${adminPassword}</password>
|
||||
<roles><id>superuser</id></roles>
|
||||
</user>
|
||||
<user>
|
||||
<name>bender</name>
|
||||
<password>${devicePassword}</password>
|
||||
<roles><id>device</id></roles>
|
||||
</user>
|
||||
</users>
|
||||
<roles>
|
||||
<role>
|
||||
<id>ingest</id>
|
||||
<permissions>
|
||||
<permission>
|
||||
<topic>acrios/#</topic>
|
||||
<activity>SUBSCRIBE</activity>
|
||||
</permission>
|
||||
</permissions>
|
||||
</role>
|
||||
<role>
|
||||
<id>device</id>
|
||||
<permissions>
|
||||
<!-- No <activity> = PUBLISH and SUBSCRIBE; covers telemetry,
|
||||
the LWT/status topic, and any downlink under acrios/. -->
|
||||
<permission><topic>acrios/#</topic></permission>
|
||||
</permissions>
|
||||
</role>
|
||||
<role>
|
||||
<id>superuser</id>
|
||||
<permissions>
|
||||
<permission><topic>#</topic></permission>
|
||||
</permissions>
|
||||
</role>
|
||||
</roles>
|
||||
</file-rbac>
|
||||
'';
|
||||
|
||||
# Render an env file body from a list of (key, envVarName) pairs.
|
||||
envBody = pairs: lib.concatStringsSep "\n" (map
|
||||
@@ -41,7 +111,8 @@ let
|
||||
enabledKeys =
|
||||
lib.optionals cfg.supabase supabaseKeys
|
||||
++ lib.optionals cfg.ingester ingesterKeys
|
||||
++ lib.optionals cfg.postgresAdmin postgresAdminKeys;
|
||||
++ lib.optionals cfg.postgresAdmin postgresAdminKeys
|
||||
++ lib.optionals cfg.hivemq hivemqKeys;
|
||||
in
|
||||
{
|
||||
options.services.gebos.secrets = {
|
||||
@@ -55,6 +126,26 @@ in
|
||||
postgresAdmin = lib.mkEnableOption ''
|
||||
Postgres bootstrap admin password applied via a one-shot ALTER ROLE unit.'';
|
||||
|
||||
hivemq = lib.mkEnableOption ''
|
||||
HiveMQ file-RBAC credentials.xml (ingester + admin MQTT users), rendered
|
||||
0400/gebos-hivemq for the broker's file-RBAC extension.'';
|
||||
|
||||
hivemqCredentialsFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
readOnly = true;
|
||||
description = ''
|
||||
Path of the rendered HiveMQ file-RBAC credentials.xml consumed by the
|
||||
gebos-hivemq broker. Resolves to the sops-templated secret when sops
|
||||
material is present and the hivemq toggle is on, else to a baked-in
|
||||
dev-defaults file (so first-boot eval and local VMs work).
|
||||
'';
|
||||
default =
|
||||
if hasSecrets && cfg.hivemq
|
||||
then config.sops.templates."hivemq-credentials".path
|
||||
else "${pkgs.writeText "hivemq-credentials-dev.xml"
|
||||
(hivemqCredentialsXml "dev-ingester" "dev-admin" "dev-device")}";
|
||||
};
|
||||
|
||||
envFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
readOnly = true;
|
||||
@@ -90,6 +181,18 @@ in
|
||||
mode = "0400";
|
||||
owner = "root";
|
||||
};
|
||||
|
||||
# Rendered as a standalone file (not env) — the file-RBAC extension reads
|
||||
# credentials.xml directly. Owned by the broker user so preStart can link
|
||||
# it into the extension's conf/ dir.
|
||||
templates."hivemq-credentials" = lib.mkIf cfg.hivemq {
|
||||
content = hivemqCredentialsXml
|
||||
config.sops.placeholder.hivemq_ingester_password
|
||||
config.sops.placeholder.hivemq_admin_password
|
||||
config.sops.placeholder.hivemq_device_password;
|
||||
mode = "0400";
|
||||
owner = "gebos-hivemq";
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# Template for nix/secrets/secrets.yaml — the sops-encrypted store.
|
||||
#
|
||||
# To bootstrap:
|
||||
# 1. cp nix/secrets/secrets.yaml.example nix/secrets/secrets.yaml
|
||||
# 2. Replace the placeholder values below with real ones.
|
||||
# (See nix/secrets/README.md for how to generate JWT_SECRET / ANON_KEY /
|
||||
# SERVICE_ROLE_KEY etc.)
|
||||
# 3. sops -e -i nix/secrets/secrets.yaml
|
||||
# (requires sops + age + the recipients in .sops.yaml to be real keys)
|
||||
#
|
||||
# Once encrypted, edit in place with: sops nix/secrets/secrets.yaml
|
||||
#
|
||||
# Local dev does NOT need this file at all — process-compose.nix bakes in
|
||||
# dev defaults and the NixOS modules fall back to a writeText env file when
|
||||
# nix/secrets/secrets.yaml is absent. This file is only consumed at activation
|
||||
# time on real hosts.
|
||||
# Supabase compose stack (consumed on app-host by gebos-supabase.service)
|
||||
supabase_postgres_password: ENC[AES256_GCM,data:jXXxqvd15zPe5gDwnrFohsl8qrPny0IFV8UE2oyN9hGOWQ7KheltWm2r0IOL,iv:ZYBfmtduP2rSTezFwwv/NAavaF6/fvXBrdrF0MM0jY4=,tag:TiodVCwbCXDGafeaOsLURg==,type:str]
|
||||
supabase_jwt_secret: ENC[AES256_GCM,data:j17p4R3JiLWlO4C8iYE2SXthzqB4kGbkypFMPTI30J2bI9c=,iv:AAamwue0Vs1Ns7OlujNtjU5IY43Z1NVJxJW5S9ftGbk=,tag:LWX3h0QMRBNNzUPWLxe3HQ==,type:str]
|
||||
supabase_anon_key: ENC[AES256_GCM,data:zpeMG5L5GHlhGrHETnSY3cG+0rgC52HmSTDfdr0Il6cuEb0/KiUA9iyQOybT7Oo=,iv:NIMbKi14WlrTkLUt0ybBOFCklpBZ8TdFq+LkvqeElqA=,tag:rOQ1ETG13pt00kkCQQRJbQ==,type:str]
|
||||
supabase_service_role_key: ENC[AES256_GCM,data:MDacdgnZhC3mx/YUoQT8j2LGHyCYGmW2TMBdcSp91WmoWh4/WuYy5BtpBfgDLC1S+6taTPmfjA==,iv:a7mSPzr/JYs6036NGlI75NqFm0Y1WMTJJGp8YX1PIOs=,tag:Q3ygfgexT6Ju9GHIf+jHjw==,type:str]
|
||||
supabase_dashboard_password: ENC[AES256_GCM,data:i9Pl87BtOICcKwJxDp9QH1CtPaUbTb3IObrdPg/xcsqSMim9+RyO+g==,iv:qDQH1//2ZKNXAgAmFir2TG6auUEh6AFE00a/gP/1+Aw=,tag:llYW7o9XK5vmAIs45QQJpQ==,type:str]
|
||||
# Ingester (consumed on mqtt-ingest by gebos-ingester.service)
|
||||
ingester_postgres_password: ENC[AES256_GCM,data:Aq5gMK4RaYloOoYURXaGbb3jVVfSl8ZiXaabKkcrs+/iaHm1KBWtrw==,iv:07b7SF8khDBXVTVZGoqEuDsD+k9sOsvnunJAtKpfchg=,tag:DnFLwyB5Mt3NRo/Jklcntg==,type:str]
|
||||
# Postgres bootstrap (consumed on db-host by a one-shot ALTER ROLE unit)
|
||||
postgres_admin_password: ENC[AES256_GCM,data:Ien5PaD37D7c6EAq3dgh5+cpDANIA7j+V2SLiUJeIUXpqi1EjtvjTIm4,iv:PZa0Of3c2ozZ/ZE57sw6TMV7ze19A3eHKYcf6hc0kKo=,tag:YDVbdMr7gRmbp1ma2VWrmw==,type:str]
|
||||
# HiveMQ file-RBAC broker users (consumed on mqtt-ingest by gebos-hivemq).
|
||||
# PLAIN passwords (password-type defaults to PLAIN; the whole file is sops-
|
||||
# encrypted). The ingester will authenticate as `ingester`; `admin` is for ops.
|
||||
hivemq_ingester_password: ENC[AES256_GCM,data:0Kfqp5V03fyZU6ulSlCmIbV5x0Wrot8wRxtut8rxFzxsygprLkEV462R+g==,iv:6Lb7PhmJoa2JbS1g/VaPCq3HOXBpX3/8eaww/A6EJqY=,tag:Ny4pygCw3wfWObccWfOBxQ==,type:str]
|
||||
hivemq_admin_password: ENC[AES256_GCM,data:I9g7/PcZb9TMf81SKNppMDf/8Je/9V+TSG932KGi+BXZjU7IsocM8HuY,iv:AlyAI44Ryl60W3HFxExC3i92WqMKA1myW+sLsBD50mo=,tag:JYO7YtDTouCZ5frTlDofBw==,type:str]
|
||||
# Shared device login (Option A) — must match mqtt_user/mqtt_password on the
|
||||
# senders. Per-device users (Option B) are a TODO in gebos-secrets.nix.
|
||||
hivemq_device_password: ENC[AES256_GCM,data:27kqmePshQ==,iv:5m2e7Hc583jvsSjiKbl63cY9cTFs0aAY/L6jV5Vh9MU=,tag:nGhgrVpNU8Z2Z7cHTbBSSQ==,type:str]
|
||||
sops:
|
||||
age:
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRL1VHOWVreFdsQ0JWK3o0
|
||||
aFZYd0JITHU0RHg2a09DeVdFRXFiUC95OHlFCnZzaDVGeFdJa2l6S1Zxbng1K2tr
|
||||
ZkEzQzdHeXRWWi9OUTdDYXJBeTZUTWcKLS0tIDBKdEZTSlZFeFRFSGtXclV0MkxR
|
||||
RVFNbnQxbDdGSnpNendBbmJnditLMDAKNrn1/8gIngi0pM+DEf6ov/SGNQrXnFby
|
||||
1a2iAqBP+EDG0z52dcZns53Mj6XDgeh5Z2W5hDpITxDNhSVXEKmdHQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1cx8ul285kjkzmnhw6skdstnzrxnnme4xkflknzn7yhv52fgxqevqkd66cn
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1Z3JpRnZIUG90b0xkdnU0
|
||||
aHpBQm5BMjF1YVNhOHVQcXJEWnhxbU5FQUM0Cml3SUJvWjAydWFwcGVHQlo0SjhU
|
||||
bzM4N3J0YjdBWURJbER5OENsR3VMYWsKLS0tIElQWTZqL1lscUZoOXArR014ZDVJ
|
||||
Z0JidnBZSWt0QmdBb1l0MlUxSngyUjQK6EuSmh77Gcc9aow16fgeZV+/GlJw1Zaz
|
||||
aH3GqG7lw6hvn0HWSqJj8shTROqj+sWi1fUCIgMo1kcTBhKQC+ErEQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1wpl3vz60tlt880p5fmfedr8c59dm4kf0czsacv0w5my706twuf5q9p0x43
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjZEw0UC82azY2c3NBVk42
|
||||
M1RORGszMDVWbG5UVkRZOFVEL05hcVpzSlVvCklhOThsMDRGelBtTlFSZ3hWT01G
|
||||
ZkErVTdhT2t2dFpzOHI2Z1p3a2c0NTgKLS0tIFNpNE53VEVPczVSRTJNWlEyMU5X
|
||||
RFp0VzZaeG5hcTNmZ21GTENBM09zZTQK1QswQul0EbX1UnuF1iL4WXXZN3nOSBjp
|
||||
inIv5inJhjcZ+Vo82NDeQcWnk+5Ns43sKqQjHf0XpfgyzRyjV6SHmw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1g57pznep69nlyv3sltz6k0sml47v68m03gh68jkqwcq4jdx68vtsvnefq0
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTRE4wSUxMMjA2cUNEQXFL
|
||||
ZE9ha2dsL0tKK2tTTTJpd25EUUFlQnJiQ1RBCjNTcUk2cGxvUWF5b2MrVi81bVZr
|
||||
dDBoSFhRdDR0UnZsMWdRYnJhNGtMYjQKLS0tIFBiM3pweFgzRWNCSHozV01XbCs3
|
||||
elhHMFE0dnZHU2pFaEJ2ekJ0U2FhQTAKNZGdxEMPf7pG11zS233wFYfDKySMqL/m
|
||||
7syeofAs00911pOubBhRa1G6zvwblevDnZuAqhHBTiKfKtGrpCG4Jg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age190gu75rf3ra89mhk27xe3tv87tad087altqhugjlhkerqwe2jfqsnu738d
|
||||
encrypted_regex: ^(supabase_|ingester_|postgres_admin_|hivemq_)
|
||||
lastmodified: "2026-06-29T15:20:36Z"
|
||||
mac: ENC[AES256_GCM,data:ds4eFZC9fjU+0NjvpaRKWq8HSbslFSPVGLlPtVGlvLm1kpZ8OtC6D6H4VtZZdIKvie/SahcA/3oMGMC66UDADrem7OGKWX5UFRrtuwrctenxteu6Q16sNqlmYBUSkd1YNAH3K4m4TR1DyvMNVC6hTbpCP2hVv7k9fy0o5UZT7bQ=,iv:vRBvBsypuurce0yq3Nu4zvqpzL7+84O3SkbOPoHllt8=,tag:ANZ7TYppVgX36SU1WvLlow==,type:str]
|
||||
version: 3.13.1
|
||||
@@ -27,3 +27,12 @@ ingester_postgres_password: "replace-me-openssl-rand-hex-32"
|
||||
|
||||
# Postgres bootstrap (consumed on db-host by a one-shot ALTER ROLE unit)
|
||||
postgres_admin_password: "replace-me-openssl-rand-hex-32"
|
||||
|
||||
# HiveMQ file-RBAC broker users (consumed on mqtt-ingest by gebos-hivemq).
|
||||
# PLAIN passwords (password-type defaults to PLAIN; the whole file is sops-
|
||||
# encrypted). The ingester will authenticate as `ingester`; `admin` is for ops.
|
||||
hivemq_ingester_password: "replace-me-openssl-rand-hex-32"
|
||||
hivemq_admin_password: "replace-me-openssl-rand-hex-32"
|
||||
# Shared device login (Option A) — must match mqtt_user/mqtt_password on the
|
||||
# senders. Per-device users (Option B) are a TODO in gebos-secrets.nix.
|
||||
hivemq_device_password: "replace-me-to-match-the-sender-mqtt_password"
|
||||
|
||||
@@ -16,7 +16,7 @@ services:
|
||||
image: kong:2.8.1
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:8000:8000/tcp" # Caddy proxies api.ge-bos.de here
|
||||
- "127.0.0.1:8000:8000/tcp" # Caddy proxies api.gebos.online here
|
||||
environment:
|
||||
KONG_DATABASE: "off"
|
||||
KONG_DECLARATIVE_CONFIG: /home/kong/kong.yml
|
||||
@@ -29,10 +29,10 @@ services:
|
||||
environment:
|
||||
GOTRUE_DB_DRIVER: postgres
|
||||
GOTRUE_DB_DATABASE_URL: postgres://supabase_auth_admin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
|
||||
GOTRUE_SITE_URL: https://app.ge-bos.de
|
||||
GOTRUE_SITE_URL: https://app.gebos.online
|
||||
GOTRUE_JWT_SECRET: ${JWT_SECRET}
|
||||
GOTRUE_JWT_EXP: "3600"
|
||||
API_EXTERNAL_URL: https://api.ge-bos.de
|
||||
API_EXTERNAL_URL: https://api.gebos.online
|
||||
|
||||
rest:
|
||||
image: postgrest/postgrest:v12.0.2
|
||||
@@ -52,7 +52,7 @@ services:
|
||||
STUDIO_PG_META_URL: http://meta:8080
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
DEFAULT_ORGANIZATION_NAME: Gebos
|
||||
SUPABASE_URL: https://api.ge-bos.de
|
||||
SUPABASE_URL: https://api.gebos.online
|
||||
DASHBOARD_USERNAME: gebos
|
||||
DASHBOARD_PASSWORD: ${DASHBOARD_PASSWORD}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user