debugging gateway internet access

ingester/gateway/DEBUGGING.md

@@ -0,0 +1,106 @@
+# Gateway MQTT connectivity — debugging findings (July 2026)
+
+Why the ACRIOS `ACR_CV_101N_W_D2` gateway (SIM7022 NB-IoT modem) could not
+connect to `ingest.gebos.online:8883`, how we proved it, and what to do next.
+
+**TL;DR: the bundled SIM lives in a private ACRIOS APN (`acrios.iot`) with no
+public-internet breakout. The broker, TLS setup, and gateway firmware are all
+fine.** No packet from the gateway ever reached our host; every failure
+happened inside the carrier/APN network.
+
+## Root cause
+
+The modem's own PDP-context diagnostics (see step 6) show:
+
+```
++CGDCONT:   0,"IP","acrios.iot","10.10.111.184"
++CGPADDR:   0,"10.10.111.184"
++CGCONTRDP: 0,5,"acrios.iot","10.10.111.184.255.255.255.0",,"10.65.0.1","10.65.0.2",,,,,1430
+```
+
+- APN `acrios.iot` — a private APN; the SIM (IMSI prefix 901-28, a global
+  IoT-roaming range) is provisioned for ACRIOS's walled garden only.
+- Private device IP (`10.10.111.184`) and private DNS servers (`10.65.0.x`).
+- Result: DNS for public names fails (`+CMQTTCONNECT: 0,25`) and raw TCP to a
+  public IP fails (`+CMQTTCONNECT: 0,3`). Signalling-plane features (network
+  registration, NITZ time sync, CSQ) all work, which made the SIM *look* fine.
+- `APN = "auto"` in the Lua config is not the bug: `acrios.iot` is almost
+  certainly the only APN this SIM subscription allows, so overriding the APN
+  string alone cannot fix it.
+
+This also explains the firmware's `acrios/` topic prefix — these gateways are
+normally sold talking to ACRIOS's own MQTT cloud inside that APN.
+
+## Debugging steps (in order, with what each eliminated)
+
+1. **Verified the broker from the outside.** From a dev machine:
+   `openssl s_client -connect ingest.gebos.online:8883` (valid Let's Encrypt
+   RSA-2048 chain, TLS 1.2 + 1.3 OK) and
+   `mosquitto_pub ... -u bender -P <device pw>` → `CONNACK 0`, publish
+   accepted. ⇒ EMQX, its ACME cert, and the device credentials all work; the
+   problem is specific to the gateway's path.
+
+2. **Read the serial log symptom precisely.** `AT+CMQTTCONNECT` returned `OK`
+   (command accepted) but the result URC never arrived before the script's
+   20 s timeout — indistinguishable from a TLS stall at that point.
+
+3. **Checked EMQX for handshake errors.** `docker logs` showed nothing — but a
+   deliberately failed cipher probe from the dev machine *also* logged nothing,
+   proving EMQX's default log level swallows TLS failures. Inconclusive, not
+   exonerating. (`emqx ctl listeners` shutdown counters were similarly
+   ambiguous; `emqx ctl log set-level debug` enables visibility.)
+
+4. **tcpdump on the ingest host** (`tcpdump -i any 'tcp port 8883'`) while the
+   gateway retried: zero packets from the gateway, ever. ⇒ nothing above the
+   IP layer (TLS version, ciphers, cert chain, EMQX config) could be the cause.
+
+5. **Script changes to force better diagnostics** (v1.8 → v1.10, see below):
+   - `mqtt_timeout` 20 s → 60 s: the modem finally had time to report real
+     result codes instead of being abandoned mid-attempt.
+     ⇒ surfaced `+CMQTTCONNECT: 0,25` = **DNS error**.
+   - `mqtt_server` hostname → IP literal `178.104.178.108`: bypassed DNS.
+     ⇒ error changed to `+CMQTTCONNECT: 0,3` = **socket connect fail**.
+     DNS *and* TCP both dead, registration fine → no data breakout.
+   - `authmode 1 → 0` (skip server-cert verification): ruled out the modem
+     rejecting our CA chain; made no difference (never reached TLS). Note the
+     stock ACRIOS script enables cert verification whenever an MQTT username
+     is set, yet configures no CA — with this SIM it never got far enough to
+     matter, but it could never have verified successfully anyway.
+   - Added PDP-context dump (`AT+CGDCONT?`, `AT+CGPADDR`, `AT+CGCONTRDP`)
+     before every connect attempt. ⇒ produced the root-cause evidence above.
+
+6. **SIMCom CMQTT result codes seen** (for future reference):
+   `+CMQTTCONNECT: 0,25` = DNS error · `0,3` = socket connect fail ·
+   `+CMQTTPUB: 0,26` = socket closed / not connected (follow-on noise, not a
+   distinct fault). No URC before timeout usually means the command was still
+   waiting — raise the wait, don't guess.
+
+## Current state of `mbus_mqtt_gebos.lua` (v1.10) — temporary changes to revert
+
+| Change | Why it's there | Revert when |
+| --- | --- | --- |
+| `mqtt_server = "178.104.178.108"` | bypass private-APN DNS | breakout exists **and** public DNS resolves |
+| `AT+CSSLCFG="authmode",0,0` | no CA provisioned on modem | ISRG Root X1 loaded (`AT+CCERTDOWN`) + `cacert` configured → set authmode 1 |
+| `mqtt_timeout = 60000` | see real modem result codes | after measuring real handshake time on a working network |
+| PDP-context dump in `MQTT_SSL_Start()` | root-cause evidence | once connectivity is stable (costs battery/airtime per connect) |
+
+## Options to get data flowing
+
+1. **Preferred: ask ACRIOS** (SIM contract holder) to enable internet breakout
+   on the APN, or whitelist `ingest.gebos.online` / `178.104.178.108:8883`
+   for our SIMs. Hardware untouched.
+2. **Own SIM:** any IoT SIM with public breakout (1NCE, EMnify, local carrier
+   NB-IoT, …); set `APN = "<provider apn>"` explicitly in the Lua config.
+   Check provider NB-IoT coverage on bands 3/5/8/20 (what the firmware
+   configures).
+3. Fallback: ACRIOS cloud bridge/forwarding — reintroduces the vendor
+   dependency the self-hosted ingest stack exists to avoid.
+
+## Verified-good along the way
+
+- EMQX 5.8.6 (oci-container, host network) serving MQTTS :8883 with the ACME
+  (HTTP-01) RSA-2048 cert; hot-reloads renewed PEMs.
+- Broker auth: `bender` device login + ACL, external end-to-end publish test.
+- Note for MQTTX/dashboard tests: EMQX default log level logs neither TLS
+  handshake failures nor connects; use `emqx ctl log set-level debug`
+  temporarily, and `emqx ctl listeners` for connection/shutdown counters.

ingester/gateway/mbus_mqtt_gebos.lua

@@ -1,11 +1,15 @@
-ver="1.8"
+ver="1.11"
 ----- CONFIGURATION -----
 -------- NB-IoT ---------
 APN = "auto"  -- APN for the SIM card
 PLMNID = 0  -- PLMNID for the SIM card
 reinitOnWakeGather = 1  -- 1 for reinitialization of the NB Iot module before gathering
 --------- MQTT ---------
-mqtt_server = "ingest.gebos.online"  -- broker
+-- DIAGNOSTIC: connect by IP to bypass carrier DNS — the modem fails with
+-- +CMQTTCONNECT: 0,25 (DNS error) when given the hostname. Revert to
+-- "ingest.gebos.online" once DNS works on this SIM/APN (fix the APN or set
+-- modem DNS servers); a hardcoded IP breaks silently if the host ever moves.
+mqtt_server = "178.104.178.108"  -- broker (= ingest.gebos.online)
 mqtt_user_base = "acrcv-"  -- client name template
 mqtt_client_name = nil  -- client name - set to some value, otherwise will be mqtt_user_base .. imsi
 mqtt_topic_base = "acrios/"  -- topic template, will add IMSI/ .. topic name
@@ -13,7 +17,7 @@ mqtt_user = "bender"  -- user name (for login)
 mqtt_password = "hancock"  -- password (for login)
 mqtt_port = 8883  -- port (8883 using TLS/SSL)
 mqtt_mode = "auto" -- "TCP" or "SSL" or autodetection based on port (8883 for SSL, else TCP)
-mqtt_timeout = 20000  -- timeout used for connecting and waiting for subscription
+mqtt_timeout = 60000  -- timeout used for connecting and waiting for subscription; generous because the TLS handshake (~6 kB cert chain) is slow over NB-IoT
 reportScheduled = true  -- true for reporting settings to the MQTT. List of days on which gathering occurs
 willMessage = true  -- true to enable MQTT disconnect message
 atDebug = true  -- true for debugging AT commands
@@ -87,6 +91,21 @@ function MQTT_SSL_Start()
     end
 
 	wSc(SCmV)
+	-- DIAGNOSTIC: dump PDP context / assigned IP / DNS before every connect
+	-- attempt. No IP in AT+CGPADDR = no data session (wrong APN) — explains
+	-- +CMQTTCONNECT: 0,25 (DNS) and 0,3 (socket connect fail).
+	at("AT+CGDCONT?", 5000)
+	at("AT+CGPADDR", 5000)
+	at("AT+CGCONTRDP", 5000)
+
+	-- DIAGNOSTIC: eUICC probe (run once, then remove). ICCID identifies the
+	-- SIM issuer; the AT+CSIM APDUs ask the card whether it has an eUICC
+	-- ISD-R applet and, if so, its EID.
+	--   select ISD-R: response ending 9000 or 61xx = eUICC; 6A82 = plain SIM.
+	--   GetEID:       response contains BF3E..5A10<32 hex digits> = the EID.
+	at("AT+CCID", 5000)
+	at('AT+CSIM=42,"00A4040410A0000005591010FFFFFFFF8900000100"', 10000)
+	at('AT+CSIM=22,"81E2910006BF3E035C015A"', 10000)
 	at("AT+QCPMUCFG=1,2")
 	at('AT+CEDRXS=2,5,"1111"')
 	at('AT+QCPTWEDRXS=2,5,"0000","1111"')
@@ -94,10 +113,14 @@ function MQTT_SSL_Start()
     if mqtt_mode == "SSL" then
 	    at("AT+CSSLCFG=\"sslversion\",0,4")
     end
-    if mqtt_mode == "SSL" and (mqtt_user == nil or mqtt_user == "") then
+    if mqtt_mode == "SSL" then
+        -- authmode 0 = TLS without server-certificate verification. The modem
+        -- has no CA certificate configured (no AT+CSSLCFG="cacert",...), so
+        -- authmode 1 could never validate the broker's Let's Encrypt chain and
+        -- the handshake stalled with no CONNACK. To restore verification:
+        -- provision ISRG Root X1 on the module (AT+CCERTDOWN), point
+        -- AT+CSSLCFG="cacert",0,"<file>" at it, then set authmode back to 1.
         at("AT+CSSLCFG=\"authmode\",0,0")
-    else
-        at("AT+CSSLCFG=\"authmode\",0,1")
     end
 
 	if mqtt_client_name == nil then
@@ -985,6 +1008,13 @@ function onStartup()
 	at("APN=" .. APN)
 	at("PLMNID=" .. PLMNID)
 
+	-- DIAGNOSTIC: dump the PDP context, assigned IP and DNS servers to the
+	-- serial log. If AT+CGPADDR shows no address (or 0.0.0.0) the SIM has no
+	-- data path — wrong APN — and no socket/DNS can ever work.
+	at("AT+CGDCONT?", 5000)
+	at("AT+CGPADDR", 5000)
+	at("AT+CGCONTRDP", 5000)
+
 	ex("setSwWdogNoComReloadValue", noComWdg)
 
 	mqtt_topic_base = mqtt_topic_base .. getIMSI() .. '/'