Project Setup #21
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Setup the infrastructure required for the project. The project shall be hosted on a single bare metal NixOS server.
If possible I would like to avoid docker compose and use more linux native services like systemd in order to run services.
Open ended questions: How does the deployment work? I'd like a Continous deployment action in Gitea that can deploy the code.
Imported from Linear GEB-20 · Priority: No priority
@Klaus for now let's ideate on this together in this issue. Once I approve, you can start with the implementation.
Happy to ideate before I touch any code. Here's a strawman so we have something concrete to push back on — nothing below is decided.
Proposed shape
One
flake.nixat the repo root exposing:packages.{backend, frontend}— the Go binary and the built Vite bundlenixosModules.{gebos-backend, gebos-frontend, gebos-supabase}— drop-in service modulesnixosConfigurations.prod— the full server definition (composes the modules above + Postgres + Caddy)devShells.default— Go, Node, psql, supabase CLI, etc.nix run .#dev(likely backed byprocess-compose-flakeordevenv) that brings the full stack up locally without touching the host's systemdRepo layout I'd suggest:
Service map (all systemd units, no containers):
postgresql.service— nixpkgsservices.postgresql, with the Supabase extensions (pgjwt,pg_graphql,pgsodium,supabase_vault) enabledpostgrest.service— nixpkgs already ships PostgRESTgotrue.service— package from source if not already in nixpkgsrealtime.service— Elixir/Phoenix app; packageable but the heaviest liftcaddy.service— routes/→ static React,/rest/*→ PostgREST,/auth/*→ GoTrue,/realtime/*→ Realtime,/api/*→ Go backend; ACME for TLSgebos-backend.service— the Go binaryCD via Gitea Actions, two viable paths:
deploy-rsorcolmena— runner builds the system closure, signs it, copies to the server, activates with automatic rollback on failure. My preferred choice; declarative and atomic.nixos-rebuild switch --flake gitea://lars/Gebos#prodon new commits tomain. Simpler, less infra, but no pre-deploy build verification.Decisions I'd like your input on before I draft anything more concrete
psql+ Supabase CLI from your laptop?gebos.example.com/rest/...) or subdomains per service (api.,auth.,db.)? Which domain, and is Caddy doing ACME?net/http(1.22+ patterns) or a router likechi? Database access through pgx, or always through PostgREST? Any opinion onsqlcfor typed queries?agenix,sops-nix, or just an out-of-band/etc/gebos/secrets.envthat systemd loads viaEnvironmentFile=?deploy-rs/colmena(my recommendation) vs. pull-basednixos-rebuildtimer vs. plain SSH +nixos-rebuild switch --target-host?supabase migrationCLI,goose, plain SQL files applied by a systemdoneshot, or something else?If you'd rather not answer all of these up front, the highest-leverage ones to unblock a meaningful prototype are 1, 2, 5, and 6 — the rest can be deferred to when we hit them.
Let me know which directions feel right (or wrong) and I'll iterate. Once we converge I'll wait for your green light before opening an MR.
Ok thank you. I've made a more detailed plan about this. We need multiple different machines (microservces) in order to separate failure modes.
Architecture
authschemasupabase/auth(Docker)JWT_SECRETshared between Go backend and GoTrueThe other assumptions still holding:
To your questions
Let's ideate on this together!
@Klaus Please respond to my request
@Klaus Can you hear me?
Loud and clear. Ready when you are.
— Klaus
@Klaus Please continue our discussion about the system architecture above
Good — splitting failure domains across three hosts makes sense. A few things to refine before we commit.
Things I'd push back on or want to confirm
1. HiveMQ CE HA isn't really HA. Clustering is an Enterprise feature; CE is single-node. "Two instances behind a TCP LB" with QoS 1/2 means duplicated messages (best case) or lost sessions on failover (worst case) — devices reconnect to whichever node the LB picks and their persistent session lives on the other one. Realistic options:
My recommendation: single
mqtt-ingestwith persistent sessions, accept ~30s of reconnection on restart. Revisit if device count crosses ~10k.2. Supabase Auth in Docker contradicts "avoid docker compose / prefer systemd." Either:
systemd-nspawnunit if Docker is purely about the upstream image, orI'd vote (a). Same for
postgrest,realtime,storage-api,imgproxy— all already packageable or in nixpkgs.3. RLS on ingested telemetry — the actual mechanic. Two viable patterns:
t/<tenant_id>/d/<device_id>/.... Telegraf'stopic_parserextractstenant_idas a tag, lands as a column on the hypertable. RLS policy:USING (tenant_id = current_setting('app.tenant_id')::uuid). The Go backend setsapp.tenant_idfrom the JWT before each query (viaSET LOCALin a transaction or pgxBeforeAcquire). Telegraf itself writes as aBYPASSRLSrole — it has to, because at write time there's no session user.devices(device_id, tenant_id)table, RLS uses a subquery. Slower, but doesn't trust the topic structure.I'd start with the first. The trust boundary is "what tenants can publish to what topics" which we enforce at the broker via ACL — HiveMQ CE supports file-based ACLs, or we can write a small auth extension that hits Postgres.
4. Where does the Go/Huma backend sit relative to PostgREST? Original sketch had PostgREST as the main REST surface. If you want Huma for OpenAPI generation, that implies the Go backend is the public REST API and PostgREST goes away (or is internal-only). I'd actually drop PostgREST entirely in favor of Huma + pgx + sqlc — cleaner trust boundary, real types, no PostgREST schema-cache reload gymnastics. Confirm?
5. Synchronous replica on db-host. Sync replication blocks writes when the replica is unreachable, which is bad for an MQTT ingest pipeline that can't pause. Recommended setup: one async streaming replica + pgBackRest WAL archive to S3-compatible (Hetzner Object Storage or Backblaze B2). RPO ≈ archive interval, RTO ≈ restore time. Sync only if you genuinely need RPO=0; for telemetry, async is almost always correct.
New decisions needed
A. Subdomain map. I'd propose:
app.ge-bos.de→ static SPA (Caddy on app-host)api.ge-bos.de→ Go/Huma backendauth.ge-bos.de→ GoTruerealtime.ge-bos.de→ Realtime (if we keep it)storage.ge-bos.de→ Storage API (if we keep it)ingest.ge-bos.de:8883→ MQTT/TLS (mqtt-ingest)ge-bos.de→ marketing redirect toapp? or same as app?Or collapse api/auth/realtime/storage under
app.ge-bos.de/{api,auth,realtime,storage}— fewer certs, simpler CORS. Preference?B. Local dev model. Three-node prod, but local dev was "easy to start on a NixOS machine." Two options:
process-compose-flake, fast iteration. Doesn't exercise the network topology.nixosTest-style three-VM setup mirroring prod. Realistic but slow to spin up.Recommend single-VM for day-to-day, with the multi-VM setup available as
nix run .#integration-testfor pre-deploy smoke. Agree?C. Secrets — "simpler route." Concretely that means
EnvironmentFile=/etc/gebos/secrets.envper host, mode 0400, provisioned manually on each host the first time. The file is NOT in the repo or the Nix store. Deploys read it at service start. Cheap and works; downside is rotation is a manual ssh + restart. OK with this?D. Telegraf vs. small Go ingester. Telegraf is fine if the payload is "JSON or line protocol → columns." If devices send anything that needs decoding (CBOR, protobuf, custom binary, calibration, deduplication), a 200-line Go service is easier to reason about than a Telegraf config. What's the actual device payload format?
E. Object storage for backups + Supabase Storage. Both pgBackRest and Supabase Storage want an S3-compatible bucket. Same provider, separate buckets? Hetzner Object Storage is the obvious pick since the hosts are likely Hetzner — confirm?
F. Deploy topology.
deploy-rswith one flake exposing:Gitea Actions workflow on push to
main: build all three closures, rundeploy-rsper host in dependency order (db → app → ingest), each with auto-rollback. The Gitea runner needs SSH + Nix-store push to each host. Sound right?My recommended highest-leverage decisions to lock first
A (subdomains), C (secrets file path), D (payload format — drives whether Telegraf survives), and the answer to #4 above (PostgREST stays or goes). The rest can iterate.
Once those four are settled I think we have enough to draft the actual flake layout and you can green-light implementation.
— Klaus
@Klaus These are good points:
Subdomain map
Your recommendation looks good, we do not need realtime or storage endpoints for now. And yes, ge-bos.de without subdomain will be used for marketing.
Local dev model
Single-VM dev sounds like the easier approach.
Secrets — "simpler route." EnvironmentFile=/etc/gebos/secrets.env per host is good
Telegraf vs. small Go ingester.
Let's go with our own GO ingester. This way we have full control. I'd think this should be less than 200 lines total.
Object storage for backups + Supabase Storage.
Both is not required for now.
Deploy topology
Yes, "workflow on push to main: build all three closures" sounds good
Locked in:
tenant_idcolumn on the hypertable, standard RLS (USING (tenant_id = current_setting('app.tenant_id')::uuid)). Backend setsapp.tenant_idfrom the JWT per-request. Yes, that's exactly the mechanic.app.,api.,auth.,ingest.— no realtime/storage.ge-bos.deapex = marketing (separate concern, not in this repo's deploy)process-compose-flakeEnvironmentFile=/etc/gebos/secrets.env, provisioned out-of-band, mode 0400deploy-rsfrom Gitea Actions on push tomain, three closures, db → app → ingest orderThree things "vanilla Supabase via Docker" forces us to settle before I draft the flake — they're not nits, they change the topology:
1. Where does the Supabase stack actually run?
The official self-hosting bundle is a
docker-compose.ymlthat brings its own Postgres container. Two choices:app-host, including its bundled Postgres. Thendb-hostbecomes the dedicated TimescaleDB for telemetry only, and Supabase's auth/metadata lives in the compose-bundled Postgres onapp-host. Two Postgres instances, two backup stories later.db-hostPostgres (overridePOSTGRES_HOSTetc., disable the bundleddbservice). Less "vanilla" — you're editing the compose file — but one Postgres to back up.I'd recommend (b) despite the "vanilla" friction; one Postgres is much simpler operationally and RLS on telemetry needs to share the auth schema anyway. Your call though.
2. PostgREST vs. Huma — pick one public REST surface.
Vanilla Supabase ships PostgREST at
/rest/v1/*. You also want Huma for OpenAPI. Running both as public APIs is confusing (two ways to read the same data, two auth integrations to keep in sync). Options:api.ge-bos.de. Clean.I'd recommend the first. Confirm?
3. GoTrue exposure.
Vanilla compose has Kong fronting all Supabase services on a single port. Do we:
auth.ge-bos.de→ Kong → GoTrue, orBypassing Kong is simpler and we don't use most of what Kong gives us (rate limiting, API keys for anon/service roles — actually we do use the anon/service role keys; Kong injects them). So: keep Kong.
auth.ge-bos.de→ Kong's auth route. Same pattern for any other Supabase service we expose later.Answer 1/2/3 and I'll draft the flake layout + a skeleton PR for your review before any real implementation.
— Klaus
@Klaus
Where does the Supabase stack actually run?
My choice is (b) Run the compose stack but point its services at external db-host Postgres.
PostgREST vs. Huma — pick one public REST surface.
I'd like to use PostgREST for most queries from the frontend. But it cannot handle all requests, for example custom aggregations of chart data. Is there a way that we can unify this better? Perhaps there is a better choice in this case than Huma?
GoTrue exposure.
I mainly want to use Caddy for the easy TLS certificate management. Are there any downsides to but Caddy in front of Kong? What does Kong provide for us?
Good questions — 2 and 3 are worth digging into.
1. Supabase on external
db-host— lockedConcretely: in the compose file, remove (or
profiles:out) the bundleddbservice, then override on every Supabase service that talks to Postgres:db-hostPostgres needs the Supabase-required extensions enabled (pgjwt,pgcrypto,pgsodium,supabase_vault,pg_graphql) and theauth,storage,realtime,_analytics,_supavisorschemas pre-created with the right owners. The Supabase docker repo has aninitSQL bundle we'll copy into the db-host NixOS module'sinitialScript.One Postgres, one backup story when we get there. Good call.
2. PostgREST as primary, with a clean escape hatch for aggregations
You don't need Huma for this. The Supabase-native answer is
/rpc/endpoints backed by SQL functions — you stay in one API, one client (supabase-js/postgrest-js), one auth model, RLS still applies.Pattern:
Frontend then calls:
That covers the large majority of "PostgREST can't express this" cases:
viewis queryable directly; PostgREST treats it like a table.security invokerfunction in/rpc/.When you'd actually want a separate Go service instead of
/rpc/:For Gebos right now, I think none of those apply except possibly (2) eventually. So my recommendation:
api.ge-bos.de).api.ge-bos.de/rpc/*for custom queries via SQL functions.BYPASSRLSrole.OpenAPI: PostgREST auto-generates an OpenAPI 2.0 spec at
GET /of its base URL. Not as nice as Huma's output but it exists and stays in sync automatically.If we ever do need (1)–(3) above, we add a single Go endpoint then — easy to bolt on later, no need to design for it now.
3. Kong — keep it, Caddy in front is fine
What Kong does in the vanilla Supabase compose stack:
apikey/Authorization: Bearer <anon|service_role>header against the two well-known JWTs, and rejects unauthenticated traffic before it ever reaches PostgREST/GoTrue. Without Kong, anything internet-facing is wide open at the network layer (it'd still hit RLS, but no defence in depth)./auth/v1/*→ GoTrue,/rest/v1/*→ PostgREST,/storage/v1/*→ Storage,/realtime/v1/*→ Realtime.supabase-jsis hardcoded to this layout.apikeyand forwards the JWT asAuthorizationto upstreams in the form they expect.Downsides of Caddy → Kong → service:
Caddyfile+kong.yml). Real but small.Downsides of bypassing Kong (Caddy → GoTrue directly):
supabase-jswill still send theapikeyheader, but nothing validates it. Effectively anyone on the internet can hit GoTrue's signup/login endpoints unauthenticated-by-apikey. (GoTrue itself has rate limits and captcha hooks, so not catastrophic, but it's an unforced downgrade.)supabase-jsto point at non-standard paths, or (b) replicate Kong's path layout in Caddy's reverse proxy block. Doable, but you're now maintaining a Caddy file that mimics Kong, defeating "vanilla."Recommendation: Caddy terminates TLS for
api.ge-bos.deandauth.ge-bos.de, proxies to Kong on a private port (e.g.,127.0.0.1:8000), Kong fans out to GoTrue / PostgREST / etc. You still get the Caddy ACME flow you wanted; Kong does the API-gateway job it's designed for. The "two proxies" cost is real but tiny, and you don't lose features.Concretely:
(Or collapse onto a single
api.ge-bos.dewith/auth/v1and/rest/v1paths — that's actually what the Supabase JS client expects out of the box if you set oneSUPABASE_URL. Worth considering — simpler client config, one cert, one CORS origin. Up to you.)What I'd need from you to start drafting
/rpc/is the public surface.api.ge-bos.defor everything Supabase) or two (api.+auth.)? I lean one — matchessupabase-jsdefaults.Once those three are settled I'll open the skeleton PR (flake, three host modules, compose unit for Supabase on app-host pointed at db-host, ingester stub) for you to review before any real implementation lands.
— Klaus
@Klaus I think the choice of using RPC and ditching Huma is good. PostgREST + /rpc/ is the public surface.
Yes, then let's go with the setup of Caddy and Kong chained. In order to get the best of both worlds. The TLS termination via Caddy and the Auth via Kong. Also let's do a single subdomain for everything supabase. (What about the supabase dashboard?)
Locked:
/rpc/is the public REST surface, Huma droppedapi.ge-bos.defor everything Supabase.supabase-jsinitialized withSUPABASE_URL=https://api.ge-bos.de, paths/auth/v1/*,/rest/v1/*,/rpc/*all routed by KongStudio (Supabase dashboard)
It's bundled in the vanilla compose stack (Next.js, port 3000 inside the network). Three real options, and the right answer depends on whether you want web access from anywhere or only from your laptop.
A. Don't expose it publicly — SSH tunnel when you need it. My recommendation.
ssh -L 3000:127.0.0.1:3000 app-hostthen openhttp://localhost:3000.service_roleJWT — it intentionally bypasses RLS so admins can do admin things. Anything that reaches Studio's UI can read/write any tenant's data. The only thing in front of it in vanilla compose is HTTP Basic Auth withDASHBOARD_USERNAME/DASHBOARD_PASSWORDenv vars, which is weak as the sole defence on the public internet.B. Public at
studio.ge-bos.de, behind real auth.forward_authto something stronger than basic — e.g., an OAuth2 proxy (oauth2-proxyagainst GitHub/Google), or Caddy's built-inbasic_authwith an argon2 hash if you accept basic-auth's limits.C. Public at
api.ge-bos.de/studio/*behind the same Caddy+Kong chain.basePathand it's fiddly. Not worth it vs. (B).My vote: (A) for now. Studio is in the compose file, but the unit is bound to
127.0.0.1:3000onapp-host, no public route, no Kong route. If you later want browser access from anywhere, we add (B) — it's a 20-line Caddy block and anoauth2-proxysystemd unit, no other changes.If you want (B) from day one, tell me which IdP (GitHub? Google? a static password is also OK if you accept the limits) and I'll wire it in the initial PR.
Assuming (A), I have everything I need. Want me to open the skeleton PR now — flake at root, three
nixosConfigurations(mqtt-ingest,db-host,app-host), Supabase compose unit onapp-hostpointed atdb-host, Caddy→Kong wiring, Go ingester stub,deploy-rsconfig, Gitea Actions workflow — for review before any real implementation?— Klaus
@Klaus For the studio: "Don't expose it publicly — SSH tunnel when you need it.". Yes please create a PR
Skeleton PR up as #23 (draft).
Studio is bound to
127.0.0.1onapp-host— no public route. Access from your laptop withssh -L 3000:127.0.0.1:3000 app-host, as agreed.PR description walks the layout and flags the TODOs that are intentionally deferred (real hardware configs, HiveMQ packaging, full upstream Supabase compose, etc.). Review the structure there and let me know what should move before I start filling in real implementation.
— Klaus