Gebos/ingester/gateway/DEBUGGING.md

Gateway MQTT connectivity — debugging findings (July 2026)

Why the ACRIOS ACR_CV_101N_W_D2 gateway (SIM7022 NB-IoT modem) could not connect to ingest.gebos.online:8883, how we proved it, and what to do next.

TL;DR: the bundled SIM lives in a private ACRIOS APN (acrios.iot) with no public-internet breakout. The broker, TLS setup, and gateway firmware are all fine. No packet from the gateway ever reached our host; every failure happened inside the carrier/APN network.

Root cause

The modem's own PDP-context diagnostics (see step 6) show:

+CGDCONT:   0,"IP","acrios.iot","10.10.111.184"
+CGPADDR:   0,"10.10.111.184"
+CGCONTRDP: 0,5,"acrios.iot","10.10.111.184.255.255.255.0",,"10.65.0.1","10.65.0.2",,,,,1430

• APN acrios.iot — a private APN; the SIM (IMSI prefix 901-28, a global IoT-roaming range) is provisioned for ACRIOS's walled garden only.
• Private device IP (10.10.111.184) and private DNS servers (10.65.0.x).
• Result: DNS for public names fails (+CMQTTCONNECT: 0,25) and raw TCP to a public IP fails (+CMQTTCONNECT: 0,3). Signalling-plane features (network registration, NITZ time sync, CSQ) all work, which made the SIM look fine.
• APN = "auto" in the Lua config is not the bug: acrios.iot is almost certainly the only APN this SIM subscription allows, so overriding the APN string alone cannot fix it.

This also explains the firmware's acrios/ topic prefix — these gateways are normally sold talking to ACRIOS's own MQTT cloud inside that APN.

Debugging steps (in order, with what each eliminated)

1. Verified the broker from the outside. From a dev machine: openssl s_client -connect ingest.gebos.online:8883 (valid Let's Encrypt RSA-2048 chain, TLS 1.2 + 1.3 OK) and mosquitto_pub ... -u bender -P <device pw> → CONNACK 0, publish accepted. ⇒ EMQX, its ACME cert, and the device credentials all work; the problem is specific to the gateway's path.

2. Read the serial log symptom precisely. AT+CMQTTCONNECT returned OK (command accepted) but the result URC never arrived before the script's 20 s timeout — indistinguishable from a TLS stall at that point.

3. Checked EMQX for handshake errors. docker logs showed nothing — but a deliberately failed cipher probe from the dev machine also logged nothing, proving EMQX's default log level swallows TLS failures. Inconclusive, not exonerating. (emqx ctl listeners shutdown counters were similarly ambiguous; emqx ctl log set-level debug enables visibility.)

4. tcpdump on the ingest host (tcpdump -i any 'tcp port 8883') while the gateway retried: zero packets from the gateway, ever. ⇒ nothing above the IP layer (TLS version, ciphers, cert chain, EMQX config) could be the cause.

5. Script changes to force better diagnostics (v1.8 → v1.10, see below):

   • mqtt_timeout 20 s → 60 s: the modem finally had time to report real result codes instead of being abandoned mid-attempt. ⇒ surfaced +CMQTTCONNECT: 0,25 = DNS error.
   • mqtt_server hostname → IP literal 178.104.178.108: bypassed DNS. ⇒ error changed to +CMQTTCONNECT: 0,3 = socket connect fail. DNS and TCP both dead, registration fine → no data breakout.
   • authmode 1 → 0 (skip server-cert verification): ruled out the modem rejecting our CA chain; made no difference (never reached TLS). Note the stock ACRIOS script enables cert verification whenever an MQTT username is set, yet configures no CA — with this SIM it never got far enough to matter, but it could never have verified successfully anyway.
   • Added PDP-context dump (AT+CGDCONT?, AT+CGPADDR, AT+CGCONTRDP) before every connect attempt. ⇒ produced the root-cause evidence above.

6. SIMCom CMQTT result codes seen (for future reference): +CMQTTCONNECT: 0,25 = DNS error · 0,3 = socket connect fail · +CMQTTPUB: 0,26 = socket closed / not connected (follow-on noise, not a distinct fault). No URC before timeout usually means the command was still waiting — raise the wait, don't guess.

Current state of mbus_mqtt_gebos.lua (v1.10) — temporary changes to revert

Change	Why it's there	Revert when
mqtt_server = "178.104.178.108"	bypass private-APN DNS	breakout exists and public DNS resolves
AT+CSSLCFG="authmode",0,0	no CA provisioned on modem	ISRG Root X1 loaded (AT+CCERTDOWN) + cacert configured → set authmode 1
mqtt_timeout = 60000	see real modem result codes	after measuring real handshake time on a working network
PDP-context dump in MQTT_SSL_Start()	root-cause evidence	once connectivity is stable (costs battery/airtime per connect)

Options to get data flowing

1. Preferred: ask ACRIOS (SIM contract holder) to enable internet breakout on the APN, or whitelist ingest.gebos.online / 178.104.178.108:8883 for our SIMs. Hardware untouched.
2. Own SIM: any IoT SIM with public breakout (1NCE, EMnify, local carrier NB-IoT, …); set APN = "<provider apn>" explicitly in the Lua config. Check provider NB-IoT coverage on bands 3/5/8/20 (what the firmware configures).
3. Fallback: ACRIOS cloud bridge/forwarding — reintroduces the vendor dependency the self-hosted ingest stack exists to avoid.

Verified-good along the way

• EMQX 5.8.6 (oci-container, host network) serving MQTTS :8883 with the ACME (HTTP-01) RSA-2048 cert; hot-reloads renewed PEMs.
• Broker auth: bender device login + ACL, external end-to-end publish test.
• Note for MQTTX/dashboard tests: EMQX default log level logs neither TLS handshake failures nor connects; use emqx ctl log set-level debug temporarily, and emqx ctl listeners for connection/shutdown counters.