diff --git a/ingester/gateway/DEBUGGING.md b/ingester/gateway/DEBUGGING.md new file mode 100644 index 0000000..ba17d3c --- /dev/null +++ b/ingester/gateway/DEBUGGING.md @@ -0,0 +1,106 @@ +# Gateway MQTT connectivity — debugging findings (July 2026) + +Why the ACRIOS `ACR_CV_101N_W_D2` gateway (SIM7022 NB-IoT modem) could not +connect to `ingest.gebos.online:8883`, how we proved it, and what to do next. + +**TL;DR: the bundled SIM lives in a private ACRIOS APN (`acrios.iot`) with no +public-internet breakout. The broker, TLS setup, and gateway firmware are all +fine.** No packet from the gateway ever reached our host; every failure +happened inside the carrier/APN network. + +## Root cause + +The modem's own PDP-context diagnostics (see step 6) show: + +``` ++CGDCONT: 0,"IP","acrios.iot","10.10.111.184" ++CGPADDR: 0,"10.10.111.184" ++CGCONTRDP: 0,5,"acrios.iot","10.10.111.184.255.255.255.0",,"10.65.0.1","10.65.0.2",,,,,1430 +``` + +- APN `acrios.iot` — a private APN; the SIM (IMSI prefix 901-28, a global + IoT-roaming range) is provisioned for ACRIOS's walled garden only. +- Private device IP (`10.10.111.184`) and private DNS servers (`10.65.0.x`). +- Result: DNS for public names fails (`+CMQTTCONNECT: 0,25`) and raw TCP to a + public IP fails (`+CMQTTCONNECT: 0,3`). Signalling-plane features (network + registration, NITZ time sync, CSQ) all work, which made the SIM *look* fine. +- `APN = "auto"` in the Lua config is not the bug: `acrios.iot` is almost + certainly the only APN this SIM subscription allows, so overriding the APN + string alone cannot fix it. + +This also explains the firmware's `acrios/` topic prefix — these gateways are +normally sold talking to ACRIOS's own MQTT cloud inside that APN. + +## Debugging steps (in order, with what each eliminated) + +1. **Verified the broker from the outside.** From a dev machine: + `openssl s_client -connect ingest.gebos.online:8883` (valid Let's Encrypt + RSA-2048 chain, TLS 1.2 + 1.3 OK) and + `mosquitto_pub ... -u bender -P ` → `CONNACK 0`, publish + accepted. ⇒ EMQX, its ACME cert, and the device credentials all work; the + problem is specific to the gateway's path. + +2. **Read the serial log symptom precisely.** `AT+CMQTTCONNECT` returned `OK` + (command accepted) but the result URC never arrived before the script's + 20 s timeout — indistinguishable from a TLS stall at that point. + +3. **Checked EMQX for handshake errors.** `docker logs` showed nothing — but a + deliberately failed cipher probe from the dev machine *also* logged nothing, + proving EMQX's default log level swallows TLS failures. Inconclusive, not + exonerating. (`emqx ctl listeners` shutdown counters were similarly + ambiguous; `emqx ctl log set-level debug` enables visibility.) + +4. **tcpdump on the ingest host** (`tcpdump -i any 'tcp port 8883'`) while the + gateway retried: zero packets from the gateway, ever. ⇒ nothing above the + IP layer (TLS version, ciphers, cert chain, EMQX config) could be the cause. + +5. **Script changes to force better diagnostics** (v1.8 → v1.10, see below): + - `mqtt_timeout` 20 s → 60 s: the modem finally had time to report real + result codes instead of being abandoned mid-attempt. + ⇒ surfaced `+CMQTTCONNECT: 0,25` = **DNS error**. + - `mqtt_server` hostname → IP literal `178.104.178.108`: bypassed DNS. + ⇒ error changed to `+CMQTTCONNECT: 0,3` = **socket connect fail**. + DNS *and* TCP both dead, registration fine → no data breakout. + - `authmode 1 → 0` (skip server-cert verification): ruled out the modem + rejecting our CA chain; made no difference (never reached TLS). Note the + stock ACRIOS script enables cert verification whenever an MQTT username + is set, yet configures no CA — with this SIM it never got far enough to + matter, but it could never have verified successfully anyway. + - Added PDP-context dump (`AT+CGDCONT?`, `AT+CGPADDR`, `AT+CGCONTRDP`) + before every connect attempt. ⇒ produced the root-cause evidence above. + +6. **SIMCom CMQTT result codes seen** (for future reference): + `+CMQTTCONNECT: 0,25` = DNS error · `0,3` = socket connect fail · + `+CMQTTPUB: 0,26` = socket closed / not connected (follow-on noise, not a + distinct fault). No URC before timeout usually means the command was still + waiting — raise the wait, don't guess. + +## Current state of `mbus_mqtt_gebos.lua` (v1.10) — temporary changes to revert + +| Change | Why it's there | Revert when | +| --- | --- | --- | +| `mqtt_server = "178.104.178.108"` | bypass private-APN DNS | breakout exists **and** public DNS resolves | +| `AT+CSSLCFG="authmode",0,0` | no CA provisioned on modem | ISRG Root X1 loaded (`AT+CCERTDOWN`) + `cacert` configured → set authmode 1 | +| `mqtt_timeout = 60000` | see real modem result codes | after measuring real handshake time on a working network | +| PDP-context dump in `MQTT_SSL_Start()` | root-cause evidence | once connectivity is stable (costs battery/airtime per connect) | + +## Options to get data flowing + +1. **Preferred: ask ACRIOS** (SIM contract holder) to enable internet breakout + on the APN, or whitelist `ingest.gebos.online` / `178.104.178.108:8883` + for our SIMs. Hardware untouched. +2. **Own SIM:** any IoT SIM with public breakout (1NCE, EMnify, local carrier + NB-IoT, …); set `APN = ""` explicitly in the Lua config. + Check provider NB-IoT coverage on bands 3/5/8/20 (what the firmware + configures). +3. Fallback: ACRIOS cloud bridge/forwarding — reintroduces the vendor + dependency the self-hosted ingest stack exists to avoid. + +## Verified-good along the way + +- EMQX 5.8.6 (oci-container, host network) serving MQTTS :8883 with the ACME + (HTTP-01) RSA-2048 cert; hot-reloads renewed PEMs. +- Broker auth: `bender` device login + ACL, external end-to-end publish test. +- Note for MQTTX/dashboard tests: EMQX default log level logs neither TLS + handshake failures nor connects; use `emqx ctl log set-level debug` + temporarily, and `emqx ctl listeners` for connection/shutdown counters. diff --git a/ingester/gateway/mbus_mqtt_gebos.lua b/ingester/gateway/mbus_mqtt_gebos.lua index 854cac2..2b7588f 100644 --- a/ingester/gateway/mbus_mqtt_gebos.lua +++ b/ingester/gateway/mbus_mqtt_gebos.lua @@ -1,11 +1,15 @@ -ver="1.8" +ver="1.11" ----- CONFIGURATION ----- -------- NB-IoT --------- APN = "auto" -- APN for the SIM card PLMNID = 0 -- PLMNID for the SIM card reinitOnWakeGather = 1 -- 1 for reinitialization of the NB Iot module before gathering --------- MQTT --------- -mqtt_server = "ingest.gebos.online" -- broker +-- DIAGNOSTIC: connect by IP to bypass carrier DNS — the modem fails with +-- +CMQTTCONNECT: 0,25 (DNS error) when given the hostname. Revert to +-- "ingest.gebos.online" once DNS works on this SIM/APN (fix the APN or set +-- modem DNS servers); a hardcoded IP breaks silently if the host ever moves. +mqtt_server = "178.104.178.108" -- broker (= ingest.gebos.online) mqtt_user_base = "acrcv-" -- client name template mqtt_client_name = nil -- client name - set to some value, otherwise will be mqtt_user_base .. imsi mqtt_topic_base = "acrios/" -- topic template, will add IMSI/ .. topic name @@ -13,7 +17,7 @@ mqtt_user = "bender" -- user name (for login) mqtt_password = "hancock" -- password (for login) mqtt_port = 8883 -- port (8883 using TLS/SSL) mqtt_mode = "auto" -- "TCP" or "SSL" or autodetection based on port (8883 for SSL, else TCP) -mqtt_timeout = 20000 -- timeout used for connecting and waiting for subscription +mqtt_timeout = 60000 -- timeout used for connecting and waiting for subscription; generous because the TLS handshake (~6 kB cert chain) is slow over NB-IoT reportScheduled = true -- true for reporting settings to the MQTT. List of days on which gathering occurs willMessage = true -- true to enable MQTT disconnect message atDebug = true -- true for debugging AT commands @@ -87,6 +91,21 @@ function MQTT_SSL_Start() end wSc(SCmV) + -- DIAGNOSTIC: dump PDP context / assigned IP / DNS before every connect + -- attempt. No IP in AT+CGPADDR = no data session (wrong APN) — explains + -- +CMQTTCONNECT: 0,25 (DNS) and 0,3 (socket connect fail). + at("AT+CGDCONT?", 5000) + at("AT+CGPADDR", 5000) + at("AT+CGCONTRDP", 5000) + + -- DIAGNOSTIC: eUICC probe (run once, then remove). ICCID identifies the + -- SIM issuer; the AT+CSIM APDUs ask the card whether it has an eUICC + -- ISD-R applet and, if so, its EID. + -- select ISD-R: response ending 9000 or 61xx = eUICC; 6A82 = plain SIM. + -- GetEID: response contains BF3E..5A10<32 hex digits> = the EID. + at("AT+CCID", 5000) + at('AT+CSIM=42,"00A4040410A0000005591010FFFFFFFF8900000100"', 10000) + at('AT+CSIM=22,"81E2910006BF3E035C015A"', 10000) at("AT+QCPMUCFG=1,2") at('AT+CEDRXS=2,5,"1111"') at('AT+QCPTWEDRXS=2,5,"0000","1111"') @@ -94,10 +113,14 @@ function MQTT_SSL_Start() if mqtt_mode == "SSL" then at("AT+CSSLCFG=\"sslversion\",0,4") end - if mqtt_mode == "SSL" and (mqtt_user == nil or mqtt_user == "") then - at("AT+CSSLCFG=\"authmode\",0,0") - else - at("AT+CSSLCFG=\"authmode\",0,1") + if mqtt_mode == "SSL" then + -- authmode 0 = TLS without server-certificate verification. The modem + -- has no CA certificate configured (no AT+CSSLCFG="cacert",...), so + -- authmode 1 could never validate the broker's Let's Encrypt chain and + -- the handshake stalled with no CONNACK. To restore verification: + -- provision ISRG Root X1 on the module (AT+CCERTDOWN), point + -- AT+CSSLCFG="cacert",0,"" at it, then set authmode back to 1. + at("AT+CSSLCFG=\"authmode\",0,0") end if mqtt_client_name == nil then @@ -985,6 +1008,13 @@ function onStartup() at("APN=" .. APN) at("PLMNID=" .. PLMNID) + -- DIAGNOSTIC: dump the PDP context, assigned IP and DNS servers to the + -- serial log. If AT+CGPADDR shows no address (or 0.0.0.0) the SIM has no + -- data path — wrong APN — and no socket/DNS can ever work. + at("AT+CGDCONT?", 5000) + at("AT+CGPADDR", 5000) + at("AT+CGCONTRDP", 5000) + ex("setSwWdogNoComReloadValue", noComWdg) mqtt_topic_base = mqtt_topic_base .. getIMSI() .. '/'