debugging gateway internet access

ingester/gateway/DEBUGGING.md

@@ -0,0 +1,106 @@
+# Gateway MQTT connectivity — debugging findings (July 2026)
+
+Why the ACRIOS `ACR_CV_101N_W_D2` gateway (SIM7022 NB-IoT modem) could not
+connect to `ingest.gebos.online:8883`, how we proved it, and what to do next.
+
+**TL;DR: the bundled SIM lives in a private ACRIOS APN (`acrios.iot`) with no
+public-internet breakout. The broker, TLS setup, and gateway firmware are all
+fine.** No packet from the gateway ever reached our host; every failure
+happened inside the carrier/APN network.
+
+## Root cause
+
+The modem's own PDP-context diagnostics (see step 6) show:
+
+```
++CGDCONT:   0,"IP","acrios.iot","10.10.111.184"
++CGPADDR:   0,"10.10.111.184"
++CGCONTRDP: 0,5,"acrios.iot","10.10.111.184.255.255.255.0",,"10.65.0.1","10.65.0.2",,,,,1430
+```
+
+- APN `acrios.iot` — a private APN; the SIM (IMSI prefix 901-28, a global
+  IoT-roaming range) is provisioned for ACRIOS's walled garden only.
+- Private device IP (`10.10.111.184`) and private DNS servers (`10.65.0.x`).
+- Result: DNS for public names fails (`+CMQTTCONNECT: 0,25`) and raw TCP to a
+  public IP fails (`+CMQTTCONNECT: 0,3`). Signalling-plane features (network
+  registration, NITZ time sync, CSQ) all work, which made the SIM *look* fine.
+- `APN = "auto"` in the Lua config is not the bug: `acrios.iot` is almost
+  certainly the only APN this SIM subscription allows, so overriding the APN
+  string alone cannot fix it.
+
+This also explains the firmware's `acrios/` topic prefix — these gateways are
+normally sold talking to ACRIOS's own MQTT cloud inside that APN.
+
+## Debugging steps (in order, with what each eliminated)
+
+1. **Verified the broker from the outside.** From a dev machine:
+   `openssl s_client -connect ingest.gebos.online:8883` (valid Let's Encrypt
+   RSA-2048 chain, TLS 1.2 + 1.3 OK) and
+   `mosquitto_pub ... -u bender -P <device pw>` → `CONNACK 0`, publish
+   accepted. ⇒ EMQX, its ACME cert, and the device credentials all work; the
+   problem is specific to the gateway's path.
+
+2. **Read the serial log symptom precisely.** `AT+CMQTTCONNECT` returned `OK`
+   (command accepted) but the result URC never arrived before the script's
+   20 s timeout — indistinguishable from a TLS stall at that point.
+
+3. **Checked EMQX for handshake errors.** `docker logs` showed nothing — but a
+   deliberately failed cipher probe from the dev machine *also* logged nothing,
+   proving EMQX's default log level swallows TLS failures. Inconclusive, not
+   exonerating. (`emqx ctl listeners` shutdown counters were similarly
+   ambiguous; `emqx ctl log set-level debug` enables visibility.)
+
+4. **tcpdump on the ingest host** (`tcpdump -i any 'tcp port 8883'`) while the
+   gateway retried: zero packets from the gateway, ever. ⇒ nothing above the
+   IP layer (TLS version, ciphers, cert chain, EMQX config) could be the cause.
+
+5. **Script changes to force better diagnostics** (v1.8 → v1.10, see below):
+   - `mqtt_timeout` 20 s → 60 s: the modem finally had time to report real
+     result codes instead of being abandoned mid-attempt.
+     ⇒ surfaced `+CMQTTCONNECT: 0,25` = **DNS error**.
+   - `mqtt_server` hostname → IP literal `178.104.178.108`: bypassed DNS.
+     ⇒ error changed to `+CMQTTCONNECT: 0,3` = **socket connect fail**.
+     DNS *and* TCP both dead, registration fine → no data breakout.
+   - `authmode 1 → 0` (skip server-cert verification): ruled out the modem
+     rejecting our CA chain; made no difference (never reached TLS). Note the
+     stock ACRIOS script enables cert verification whenever an MQTT username
+     is set, yet configures no CA — with this SIM it never got far enough to
+     matter, but it could never have verified successfully anyway.
+   - Added PDP-context dump (`AT+CGDCONT?`, `AT+CGPADDR`, `AT+CGCONTRDP`)
+     before every connect attempt. ⇒ produced the root-cause evidence above.
+
+6. **SIMCom CMQTT result codes seen** (for future reference):
+   `+CMQTTCONNECT: 0,25` = DNS error · `0,3` = socket connect fail ·
+   `+CMQTTPUB: 0,26` = socket closed / not connected (follow-on noise, not a
+   distinct fault). No URC before timeout usually means the command was still
+   waiting — raise the wait, don't guess.
+
+## Current state of `mbus_mqtt_gebos.lua` (v1.10) — temporary changes to revert
+
+| Change | Why it's there | Revert when |
+| --- | --- | --- |
+| `mqtt_server = "178.104.178.108"` | bypass private-APN DNS | breakout exists **and** public DNS resolves |
+| `AT+CSSLCFG="authmode",0,0` | no CA provisioned on modem | ISRG Root X1 loaded (`AT+CCERTDOWN`) + `cacert` configured → set authmode 1 |
+| `mqtt_timeout = 60000` | see real modem result codes | after measuring real handshake time on a working network |
+| PDP-context dump in `MQTT_SSL_Start()` | root-cause evidence | once connectivity is stable (costs battery/airtime per connect) |
+
+## Options to get data flowing
+
+1. **Preferred: ask ACRIOS** (SIM contract holder) to enable internet breakout
+   on the APN, or whitelist `ingest.gebos.online` / `178.104.178.108:8883`
+   for our SIMs. Hardware untouched.
+2. **Own SIM:** any IoT SIM with public breakout (1NCE, EMnify, local carrier
+   NB-IoT, …); set `APN = "<provider apn>"` explicitly in the Lua config.
+   Check provider NB-IoT coverage on bands 3/5/8/20 (what the firmware
+   configures).
+3. Fallback: ACRIOS cloud bridge/forwarding — reintroduces the vendor
+   dependency the self-hosted ingest stack exists to avoid.
+
+## Verified-good along the way
+
+- EMQX 5.8.6 (oci-container, host network) serving MQTTS :8883 with the ACME
+  (HTTP-01) RSA-2048 cert; hot-reloads renewed PEMs.
+- Broker auth: `bender` device login + ACL, external end-to-end publish test.
+- Note for MQTTX/dashboard tests: EMQX default log level logs neither TLS
+  handshake failures nor connects; use `emqx ctl log set-level debug`
+  temporarily, and `emqx ctl listeners` for connection/shutdown counters.