debugging gateway internet access
This commit is contained in:
Lars Nolden
@@ -0,0 +1,106 @@
|
||||
# Gateway MQTT connectivity — debugging findings (July 2026)
|
||||
|
||||
Why the ACRIOS `ACR_CV_101N_W_D2` gateway (SIM7022 NB-IoT modem) could not
|
||||
connect to `ingest.gebos.online:8883`, how we proved it, and what to do next.
|
||||
|
||||
**TL;DR: the bundled SIM lives in a private ACRIOS APN (`acrios.iot`) with no
|
||||
public-internet breakout. The broker, TLS setup, and gateway firmware are all
|
||||
fine.** No packet from the gateway ever reached our host; every failure
|
||||
happened inside the carrier/APN network.
|
||||
|
||||
## Root cause
|
||||
|
||||
The modem's own PDP-context diagnostics (see step 6) show:
|
||||
|
||||
```
|
||||
+CGDCONT: 0,"IP","acrios.iot","10.10.111.184"
|
||||
+CGPADDR: 0,"10.10.111.184"
|
||||
+CGCONTRDP: 0,5,"acrios.iot","10.10.111.184.255.255.255.0",,"10.65.0.1","10.65.0.2",,,,,1430
|
||||
```
|
||||
|
||||
- APN `acrios.iot` — a private APN; the SIM (IMSI prefix 901-28, a global
|
||||
IoT-roaming range) is provisioned for ACRIOS's walled garden only.
|
||||
- Private device IP (`10.10.111.184`) and private DNS servers (`10.65.0.x`).
|
||||
- Result: DNS for public names fails (`+CMQTTCONNECT: 0,25`) and raw TCP to a
|
||||
public IP fails (`+CMQTTCONNECT: 0,3`). Signalling-plane features (network
|
||||
registration, NITZ time sync, CSQ) all work, which made the SIM *look* fine.
|
||||
- `APN = "auto"` in the Lua config is not the bug: `acrios.iot` is almost
|
||||
certainly the only APN this SIM subscription allows, so overriding the APN
|
||||
string alone cannot fix it.
|
||||
|
||||
This also explains the firmware's `acrios/` topic prefix — these gateways are
|
||||
normally sold talking to ACRIOS's own MQTT cloud inside that APN.
|
||||
|
||||
## Debugging steps (in order, with what each eliminated)
|
||||
|
||||
1. **Verified the broker from the outside.** From a dev machine:
|
||||
`openssl s_client -connect ingest.gebos.online:8883` (valid Let's Encrypt
|
||||
RSA-2048 chain, TLS 1.2 + 1.3 OK) and
|
||||
`mosquitto_pub ... -u bender -P <device pw>` → `CONNACK 0`, publish
|
||||
accepted. ⇒ EMQX, its ACME cert, and the device credentials all work; the
|
||||
problem is specific to the gateway's path.
|
||||
|
||||
2. **Read the serial log symptom precisely.** `AT+CMQTTCONNECT` returned `OK`
|
||||
(command accepted) but the result URC never arrived before the script's
|
||||
20 s timeout — indistinguishable from a TLS stall at that point.
|
||||
|
||||
3. **Checked EMQX for handshake errors.** `docker logs` showed nothing — but a
|
||||
deliberately failed cipher probe from the dev machine *also* logged nothing,
|
||||
proving EMQX's default log level swallows TLS failures. Inconclusive, not
|
||||
exonerating. (`emqx ctl listeners` shutdown counters were similarly
|
||||
ambiguous; `emqx ctl log set-level debug` enables visibility.)
|
||||
|
||||
4. **tcpdump on the ingest host** (`tcpdump -i any 'tcp port 8883'`) while the
|
||||
gateway retried: zero packets from the gateway, ever. ⇒ nothing above the
|
||||
IP layer (TLS version, ciphers, cert chain, EMQX config) could be the cause.
|
||||
|
||||
5. **Script changes to force better diagnostics** (v1.8 → v1.10, see below):
|
||||
- `mqtt_timeout` 20 s → 60 s: the modem finally had time to report real
|
||||
result codes instead of being abandoned mid-attempt.
|
||||
⇒ surfaced `+CMQTTCONNECT: 0,25` = **DNS error**.
|
||||
- `mqtt_server` hostname → IP literal `178.104.178.108`: bypassed DNS.
|
||||
⇒ error changed to `+CMQTTCONNECT: 0,3` = **socket connect fail**.
|
||||
DNS *and* TCP both dead, registration fine → no data breakout.
|
||||
- `authmode 1 → 0` (skip server-cert verification): ruled out the modem
|
||||
rejecting our CA chain; made no difference (never reached TLS). Note the
|
||||
stock ACRIOS script enables cert verification whenever an MQTT username
|
||||
is set, yet configures no CA — with this SIM it never got far enough to
|
||||
matter, but it could never have verified successfully anyway.
|
||||
- Added PDP-context dump (`AT+CGDCONT?`, `AT+CGPADDR`, `AT+CGCONTRDP`)
|
||||
before every connect attempt. ⇒ produced the root-cause evidence above.
|
||||
|
||||
6. **SIMCom CMQTT result codes seen** (for future reference):
|
||||
`+CMQTTCONNECT: 0,25` = DNS error · `0,3` = socket connect fail ·
|
||||
`+CMQTTPUB: 0,26` = socket closed / not connected (follow-on noise, not a
|
||||
distinct fault). No URC before timeout usually means the command was still
|
||||
waiting — raise the wait, don't guess.
|
||||
|
||||
## Current state of `mbus_mqtt_gebos.lua` (v1.10) — temporary changes to revert
|
||||
|
||||
| Change | Why it's there | Revert when |
|
||||
| --- | --- | --- |
|
||||
| `mqtt_server = "178.104.178.108"` | bypass private-APN DNS | breakout exists **and** public DNS resolves |
|
||||
| `AT+CSSLCFG="authmode",0,0` | no CA provisioned on modem | ISRG Root X1 loaded (`AT+CCERTDOWN`) + `cacert` configured → set authmode 1 |
|
||||
| `mqtt_timeout = 60000` | see real modem result codes | after measuring real handshake time on a working network |
|
||||
| PDP-context dump in `MQTT_SSL_Start()` | root-cause evidence | once connectivity is stable (costs battery/airtime per connect) |
|
||||
|
||||
## Options to get data flowing
|
||||
|
||||
1. **Preferred: ask ACRIOS** (SIM contract holder) to enable internet breakout
|
||||
on the APN, or whitelist `ingest.gebos.online` / `178.104.178.108:8883`
|
||||
for our SIMs. Hardware untouched.
|
||||
2. **Own SIM:** any IoT SIM with public breakout (1NCE, EMnify, local carrier
|
||||
NB-IoT, …); set `APN = "<provider apn>"` explicitly in the Lua config.
|
||||
Check provider NB-IoT coverage on bands 3/5/8/20 (what the firmware
|
||||
configures).
|
||||
3. Fallback: ACRIOS cloud bridge/forwarding — reintroduces the vendor
|
||||
dependency the self-hosted ingest stack exists to avoid.
|
||||
|
||||
## Verified-good along the way
|
||||
|
||||
- EMQX 5.8.6 (oci-container, host network) serving MQTTS :8883 with the ACME
|
||||
(HTTP-01) RSA-2048 cert; hot-reloads renewed PEMs.
|
||||
- Broker auth: `bender` device login + ACL, external end-to-end publish test.
|
||||
- Note for MQTTX/dashboard tests: EMQX default log level logs neither TLS
|
||||
handshake failures nor connects; use `emqx ctl log set-level debug`
|
||||
temporarily, and `emqx ctl listeners` for connection/shutdown counters.
|
||||
@@ -1,11 +1,15 @@
|
||||
ver="1.8"
|
||||
ver="1.11"
|
||||
----- CONFIGURATION -----
|
||||
-------- NB-IoT ---------
|
||||
APN = "auto" -- APN for the SIM card
|
||||
PLMNID = 0 -- PLMNID for the SIM card
|
||||
reinitOnWakeGather = 1 -- 1 for reinitialization of the NB Iot module before gathering
|
||||
--------- MQTT ---------
|
||||
mqtt_server = "ingest.gebos.online" -- broker
|
||||
-- DIAGNOSTIC: connect by IP to bypass carrier DNS — the modem fails with
|
||||
-- +CMQTTCONNECT: 0,25 (DNS error) when given the hostname. Revert to
|
||||
-- "ingest.gebos.online" once DNS works on this SIM/APN (fix the APN or set
|
||||
-- modem DNS servers); a hardcoded IP breaks silently if the host ever moves.
|
||||
mqtt_server = "178.104.178.108" -- broker (= ingest.gebos.online)
|
||||
mqtt_user_base = "acrcv-" -- client name template
|
||||
mqtt_client_name = nil -- client name - set to some value, otherwise will be mqtt_user_base .. imsi
|
||||
mqtt_topic_base = "acrios/" -- topic template, will add IMSI/ .. topic name
|
||||
@@ -13,7 +17,7 @@ mqtt_user = "bender" -- user name (for login)
|
||||
mqtt_password = "hancock" -- password (for login)
|
||||
mqtt_port = 8883 -- port (8883 using TLS/SSL)
|
||||
mqtt_mode = "auto" -- "TCP" or "SSL" or autodetection based on port (8883 for SSL, else TCP)
|
||||
mqtt_timeout = 20000 -- timeout used for connecting and waiting for subscription
|
||||
mqtt_timeout = 60000 -- timeout used for connecting and waiting for subscription; generous because the TLS handshake (~6 kB cert chain) is slow over NB-IoT
|
||||
reportScheduled = true -- true for reporting settings to the MQTT. List of days on which gathering occurs
|
||||
willMessage = true -- true to enable MQTT disconnect message
|
||||
atDebug = true -- true for debugging AT commands
|
||||
@@ -87,6 +91,21 @@ function MQTT_SSL_Start()
|
||||
end
|
||||
|
||||
wSc(SCmV)
|
||||
-- DIAGNOSTIC: dump PDP context / assigned IP / DNS before every connect
|
||||
-- attempt. No IP in AT+CGPADDR = no data session (wrong APN) — explains
|
||||
-- +CMQTTCONNECT: 0,25 (DNS) and 0,3 (socket connect fail).
|
||||
at("AT+CGDCONT?", 5000)
|
||||
at("AT+CGPADDR", 5000)
|
||||
at("AT+CGCONTRDP", 5000)
|
||||
|
||||
-- DIAGNOSTIC: eUICC probe (run once, then remove). ICCID identifies the
|
||||
-- SIM issuer; the AT+CSIM APDUs ask the card whether it has an eUICC
|
||||
-- ISD-R applet and, if so, its EID.
|
||||
-- select ISD-R: response ending 9000 or 61xx = eUICC; 6A82 = plain SIM.
|
||||
-- GetEID: response contains BF3E..5A10<32 hex digits> = the EID.
|
||||
at("AT+CCID", 5000)
|
||||
at('AT+CSIM=42,"00A4040410A0000005591010FFFFFFFF8900000100"', 10000)
|
||||
at('AT+CSIM=22,"81E2910006BF3E035C015A"', 10000)
|
||||
at("AT+QCPMUCFG=1,2")
|
||||
at('AT+CEDRXS=2,5,"1111"')
|
||||
at('AT+QCPTWEDRXS=2,5,"0000","1111"')
|
||||
@@ -94,10 +113,14 @@ function MQTT_SSL_Start()
|
||||
if mqtt_mode == "SSL" then
|
||||
at("AT+CSSLCFG=\"sslversion\",0,4")
|
||||
end
|
||||
if mqtt_mode == "SSL" and (mqtt_user == nil or mqtt_user == "") then
|
||||
if mqtt_mode == "SSL" then
|
||||
-- authmode 0 = TLS without server-certificate verification. The modem
|
||||
-- has no CA certificate configured (no AT+CSSLCFG="cacert",...), so
|
||||
-- authmode 1 could never validate the broker's Let's Encrypt chain and
|
||||
-- the handshake stalled with no CONNACK. To restore verification:
|
||||
-- provision ISRG Root X1 on the module (AT+CCERTDOWN), point
|
||||
-- AT+CSSLCFG="cacert",0,"<file>" at it, then set authmode back to 1.
|
||||
at("AT+CSSLCFG=\"authmode\",0,0")
|
||||
else
|
||||
at("AT+CSSLCFG=\"authmode\",0,1")
|
||||
end
|
||||
|
||||
if mqtt_client_name == nil then
|
||||
@@ -985,6 +1008,13 @@ function onStartup()
|
||||
at("APN=" .. APN)
|
||||
at("PLMNID=" .. PLMNID)
|
||||
|
||||
-- DIAGNOSTIC: dump the PDP context, assigned IP and DNS servers to the
|
||||
-- serial log. If AT+CGPADDR shows no address (or 0.0.0.0) the SIM has no
|
||||
-- data path — wrong APN — and no socket/DNS can ever work.
|
||||
at("AT+CGDCONT?", 5000)
|
||||
at("AT+CGPADDR", 5000)
|
||||
at("AT+CGCONTRDP", 5000)
|
||||
|
||||
ex("setSwWdogNoComReloadValue", noComWdg)
|
||||
|
||||
mqtt_topic_base = mqtt_topic_base .. getIMSI() .. '/'
|
||||
|
||||
Reference in New Issue
Block a user