Infrastructure skeleton — flake, three hosts, Supabase compose, ingester, deploy-rs #23
Reference in New Issue
Block a user
Delete Branch "klaus/skeleton-infra"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Skeleton for review before any real implementation — refs #21 (comment #41).
Everything matches what we locked in the design thread. No host actually boots yet, no closure actually builds end-to-end (the vendored Supabase compose, HiveMQ packaging, hardware configs, and real internal hostnames are TODOs marked in-line). The point is for you to walk the structure and tell me where it's wrong before I fill it in.
What's here
flake.nix— one root flake. Inputs: nixpkgs unstable, flake-parts, deploy-rs, process-compose-flake, services-flake. Exposespackages.{ingester, frontend},nixosConfigurations.{db-host, app-host, mqtt-ingest},nixosModules,deploy,devShells.default, andprocess-compose.dev.nix/hosts/— three host configs +common.nix(ssh,deployuser with passwordless sudo for deploy-rs, flake settings).nix/modules/— one NixOS module per service:gebos-postgres,gebos-supabase,gebos-caddy,gebos-hivemq,gebos-ingester,gebos-frontend.nix/supabase/— vendored compose file (skeleton, not the full upstream yet — the README documents how to re-vendor), declarativekong.yml, andinit.sqlthat bootstraps extensions, roles, schemas, and thepublic.telemetryhypertable with RLS.nix/dev/process-compose.nix— local dev stack stub (Postgres + ingester + frontend wired; Supabase / Kong / Caddy marked TODO).nix/deploy.nix— deploy-rs node map,db-host→app-host→mqtt-ingest.ingester/— Go MQTT→Postgres skeleton.main.gois ~30 lines with the algorithm sketched in comments; target is <200 LOC for the real version.frontend/— Vite + React stub.src/supabase.tsinitializessupabase-jsagainst the singleapi.ge-bos.deorigin..gitea/workflows/{deploy,check}.yml— flake check on PR, deploy-rs fromnixrunner on push to main.Key wiring choices baked in
127.0.0.1on app-host. No public route. Access viassh -L 3000:127.0.0.1:3000 app-host. The compose binding uses${STUDIO_BIND}:3000, defaulted to127.0.0.1by the NixOS module.db-host. Supabase compose has its bundleddbremoved, services readPOSTGRES_HOST/POSTGRES_PORTfrom the systemd env./rpc/as the public REST surface, no Huma. The Go ingester is internal-only onmqtt-ingest, writes asgebos_ingest(BYPASSRLS).:443, reverse-proxiesapi.ge-bos.deto127.0.0.1:8000(Kong), which fans out/auth/v1,/rest/v1,/rpcto the right service./etc/gebos/secrets.env, loaded via systemdEnvironmentFile=. Each unit that needs creds declares it; nothing in the Nix store.telemetrytable hastenant_idcolumn, RLS policyUSING (tenant_id = current_setting('app.tenant_id')::uuid). Backend sets the GUC from the JWT before each query.What's deliberately not done yet
Marked with
TODOin-line; the big ones:trueso the host eval succeeds).caddy-l4for MQTT TLS at the edge (or terminating TLS inside HiveMQ — open question).npmDepsHash/vendorHashset to placeholders; first build will print the real values.nix/hosts/common.nix.Test plan
nix flake checkonceflake.lockis generated (Gitea runner first run).nix build .#ingesteroncego.sumexists.nix build .#frontendoncepackage-lock.jsonexists.deploy-rsdry-run against test VMs before we point at prod hostnames.Closes once we converge on the layout and I follow up with real implementation PRs.
— Klaus
@Klaus Quick question. What secrets will be required for deploying the full app?
Three buckets, all referenced from this branch:
1. Gitea Actions secret (CI)
DEPLOY_SSH_KEY— ed25519 private key fordeploy@{db,app,mqtt-ingest}hosts. Used by.gitea/workflows/deploy.yml. The matching pubkey goes intonix/hosts/common.nix(currently a TODO).2.
/etc/gebos/secrets.envon each host (systemdEnvironmentFile=)Per-host, only the keys that host's units actually read:
db-host — none. Postgres roles are seeded from
init.sqlat activation, but the role passwords themselves still need to land somewhere (open question — likely the same env file, consumed by a one-shotALTER ROLE … PASSWORDunit).app-host (consumed by
gebos-supabase.service):POSTGRES_PASSWORD— Postgres password used bysupabase_auth_admin,authenticator,supabase_adminconnections (auth, rest, studio, meta).JWT_SECRET— HS256 signing secret for gotrue + postgrest. Must be ≥32 chars.ANON_KEY— JWT signed withJWT_SECRET, role=anon. Referenced bykong.yml.SERVICE_ROLE_KEY— JWT signed withJWT_SECRET, role=service_role. Referenced bykong.yml+ Studio.DASHBOARD_PASSWORD— Studio basic auth (username pinned togebos).mqtt-ingest (consumed by
gebos-ingester.service):GEBOS_POSTGRES_PASSWORD— password for thegebos_ingestrole (theBYPASSRLSwriter).3. Frontend build inputs
VITE_SUPABASE_ANON_KEY— same value asANON_KEYabove, baked into the bundle atnix build .#frontendtime. Needs to be wired through as a build arg (not yet plumbed infrontend/).Not secrets, just adjacent
api.ge-bos.de/app.ge-bos.de/ingest.ge-bos.de— Caddy gets these from Let's Encrypt; nothing to provision.caddy-l4vs. inside HiveMQ). Either way, if HiveMQ terminates, its keystore password belongs in the env file too.Generation: I'd do
openssl rand -hex 32forJWT_SECRETand the Postgres passwords, then sign the two JWTs offline with that secret (Supabase has a one-liner for this in their self-hosting docs). Want me to add ascripts/gen-secrets.shthat produces a ready-to-scpsecrets.envtemplate?— Klaus
@Klaus What would be a good way to store those secrets in code (encrypted of course)? I'd like to have everything in one place and preferably only manage one secret (the private key to decrypt the other secrets).
Standard answer in the Nix world: sops-nix with age recipients. Maps onto what you described almost exactly.
Shape
nix/secrets/secrets.yaml, encrypted with SOPS. Plaintext keys are visible, values are AES-encrypted. Diffs stay readable (you see which secret changed, not its content)..sops.yaml: one age public key per host + your personal age public key. You only ever hold one private key (the one on your laptop / in a password manager). Each host holds its own — derived from its existingssh_host_ed25519_keyviassh-to-age, so no extra key material to provision.sops-nixNixOS module decrypts the entries that host is a recipient for and drops them at/run/secrets/<name>with the owner/mode you declare. Plaintext never hits the Nix store, never hits disk outside tmpfs.Fits the current wiring
The PR already loads
/etc/gebos/secrets.envvia systemdEnvironmentFile=. sops-nix hassops.templates.<name>which renders a file from decrypted placeholders — so you keep the env-file shape, just pointEnvironmentFile=at/run/secrets/gebos-envinstead. Per-host filtering (db-host gets only its keys, mqtt-ingest gets onlyGEBOS_POSTGRES_PASSWORD, etc.) falls out of.sops.yamlcreation rules."One secret to manage"
Your single age private key decrypts
secrets.yamllocally for editing (sops nix/secrets/secrets.yamlopens it in$EDITOR, re-encrypts on save). Hosts decrypt independently with their own host keys. Rotating a leaked value =sopsedit + commit + redeploy. Rotating a host =sops updatekeysafter swapping its age key in.sops.yaml.Alternatives I considered
What changes in this PR
If you're on board I'll fold sops-nix into the skeleton before filling in real implementations: add the flake input, the module wiring on each host, a placeholder
.sops.yaml+secrets.yaml(encrypted to a throwaway key until you generate yours), and switchgebos-supabase/gebos-ingesterto read from/run/secrets/.... TheDEPLOY_SSH_KEYGitea Actions secret stays where it is — that one has to live in Gitea regardless, since CI needs it before any host is touched.Want me to do that as a follow-up commit on this branch, or as a separate PR once this one lands?
— Klaus
@Klaus, that sounds great. Please add sops-nix to the current PR. Can we also add default secrets to run locally in development without having to create any? The same goes for other env vars like GEBOS_MQTT_BROKER and GEBOS_POSTGRES_URL . They should have sensible defaults for local dev, if possible.
Pushed
2dd5ed5to this branch.sops-nix
sops-nixflake input;sops,age,ssh-to-agein the devShell..sops.yamlwith placeholder age recipients (one per dev + one per host;age1TODO_…until you run the bootstrap — see below).nix/modules/gebos-secrets.nixis the single point of contact: each host enables the subset it needs (services.gebos.secrets.{supabase,ingester,postgresAdmin}) and the module renders a/run/secrets/gebos-envfile fromsops.templates.gebos-supabaseandgebos-ingesternow readEnvironmentFile = config.services.gebos.secrets.envFile— so all the per-unitEnvironmentFileplumbing flows through one place.app-host→secrets.supabase = true→POSTGRES_PASSWORD,JWT_SECRET,ANON_KEY,SERVICE_ROLE_KEY,DASHBOARD_PASSWORD.mqtt-ingest→secrets.ingester = true→GEBOS_POSTGRES_PASSWORD.db-host→secrets.postgresAdmin = true→POSTGRES_ADMIN_PASSWORD(consumed by the still-TODO one-shotALTER ROLEunit).What's still TODO before sops actually decrypts
nix/secrets/secrets.yamlis not committed (I can't generate genuinely encrypted output withoutsops/agelocally, and a fake-encrypted file would just rot). Instead I addednix/secrets/secrets.yaml.example(plaintext template) and a step-by-step bootstrap innix/secrets/README.md. The flow when you're ready:gebos-secretsguards onbuiltins.pathExists ../secrets/secrets.yaml— until the file exists, hosts fall back to awriteTextenv file with dev-prefixed placeholders. That keepsnix flake checkgreen and lets first-boot eval succeed on a fresh box.Dev defaults
Goal:
nix run .#dev,go run ./ingester, andnpm --prefix frontend run devall work with zero setup.nix/dev/process-compose.nixnow spins upmosquittoalongsidepostgres, and passes explicitGEBOS_MQTT_BROKER/GEBOS_POSTGRES_URL/GEBOS_POSTGRES_PASSWORDto the ingester process, plusVITE_SUPABASE_URL/VITE_SUPABASE_ANON_KEYto the frontend. Credentials are dev-only and committed on purpose (they only work against the local stack).ingester/main.gofalls back to the same defaults when env vars are unset and logs which ones it defaulted, so a barego run ./ingesterlights up against the local broker+db without an.envdance. Prod always sets all three via the systemd unit'sEnvironmentFile=.nix/modules/gebos-ingester.nix—mqttBrokerandpostgresUrlare now optional with sensible defaults (tcp://127.0.0.1:1883,postgres://gebos_ingest@127.0.0.1:5432/postgres?sslmode=require).frontend/src/supabase.tsdefaults tohttp://127.0.0.1:8000+ the dev anon key whenimport.meta.env.DEV, prod URL otherwise.The dev
anonKey/serviceRoleKeyinprocess-compose.nixare still placeholders — proper JWTs signed with the devjwt_secretneed the Supabase self-host signing script, which I'll fold in alongside the Supabase compose wiring that's already TODO in the same file.Heads up
When you do the secrets bootstrap, tighten
.sops.yaml— the currentcreation_rulesblock puts all keys in onekey_group, so every recipient can decrypt every key. Once real pubkeys are in, split into one rule per^supabase_,^ingester_,^postgres_admin_prefix so each host only gets what it needs. I left a note in the file.— Klaus
@Klaus Update the changes proposed to the new nixos 26.05 release.