# Gateway MQTT connectivity — debugging findings (July 2026)

Why the ACRIOS `ACR_CV_101N_W_D2` gateway (SIM7022 NB-IoT modem) could not
connect to `ingest.gebos.online:8883`, how we proved it, and what to do next.

**TL;DR: the bundled SIM lives in a private ACRIOS APN (`acrios.iot`) with no
public-internet breakout. The broker, TLS setup, and gateway firmware are all
fine.** No packet from the gateway ever reached our host; every failure
happened inside the carrier/APN network.

## Root cause

The modem's own PDP-context diagnostics (see step 6) show:

```
+CGDCONT:   0,"IP","acrios.iot","10.10.111.184"
+CGPADDR:   0,"10.10.111.184"
+CGCONTRDP: 0,5,"acrios.iot","10.10.111.184.255.255.255.0",,"10.65.0.1","10.65.0.2",,,,,1430
```

- APN `acrios.iot` — a private APN; the SIM (IMSI prefix 901-28, a global
  IoT-roaming range) is provisioned for ACRIOS's walled garden only.
- Private device IP (`10.10.111.184`) and private DNS servers (`10.65.0.x`).
- Result: DNS for public names fails (`+CMQTTCONNECT: 0,25`) and raw TCP to a
  public IP fails (`+CMQTTCONNECT: 0,3`). Signalling-plane features (network
  registration, NITZ time sync, CSQ) all work, which made the SIM *look* fine.
- `APN = "auto"` in the Lua config is not the bug: `acrios.iot` is almost
  certainly the only APN this SIM subscription allows, so overriding the APN
  string alone cannot fix it.

This also explains the firmware's `acrios/` topic prefix — these gateways are
normally sold talking to ACRIOS's own MQTT cloud inside that APN.

## Debugging steps (in order, with what each eliminated)

1. **Verified the broker from the outside.** From a dev machine:
   `openssl s_client -connect ingest.gebos.online:8883` (valid Let's Encrypt
   RSA-2048 chain, TLS 1.2 + 1.3 OK) and
   `mosquitto_pub ... -u bender -P <device pw>` → `CONNACK 0`, publish
   accepted. ⇒ EMQX, its ACME cert, and the device credentials all work; the
   problem is specific to the gateway's path.

2. **Read the serial log symptom precisely.** `AT+CMQTTCONNECT` returned `OK`
   (command accepted) but the result URC never arrived before the script's
   20 s timeout — indistinguishable from a TLS stall at that point.

3. **Checked EMQX for handshake errors.** `docker logs` showed nothing — but a
   deliberately failed cipher probe from the dev machine *also* logged nothing,
   proving EMQX's default log level swallows TLS failures. Inconclusive, not
   exonerating. (`emqx ctl listeners` shutdown counters were similarly
   ambiguous; `emqx ctl log set-level debug` enables visibility.)

4. **tcpdump on the ingest host** (`tcpdump -i any 'tcp port 8883'`) while the
   gateway retried: zero packets from the gateway, ever. ⇒ nothing above the
   IP layer (TLS version, ciphers, cert chain, EMQX config) could be the cause.

5. **Script changes to force better diagnostics** (v1.8 → v1.10, see below):
   - `mqtt_timeout` 20 s → 60 s: the modem finally had time to report real
     result codes instead of being abandoned mid-attempt.
     ⇒ surfaced `+CMQTTCONNECT: 0,25` = **DNS error**.
   - `mqtt_server` hostname → IP literal `178.104.178.108`: bypassed DNS.
     ⇒ error changed to `+CMQTTCONNECT: 0,3` = **socket connect fail**.
     DNS *and* TCP both dead, registration fine → no data breakout.
   - `authmode 1 → 0` (skip server-cert verification): ruled out the modem
     rejecting our CA chain; made no difference (never reached TLS). Note the
     stock ACRIOS script enables cert verification whenever an MQTT username
     is set, yet configures no CA — with this SIM it never got far enough to
     matter, but it could never have verified successfully anyway.
   - Added PDP-context dump (`AT+CGDCONT?`, `AT+CGPADDR`, `AT+CGCONTRDP`)
     before every connect attempt. ⇒ produced the root-cause evidence above.

6. **SIMCom CMQTT result codes seen** (for future reference):
   `+CMQTTCONNECT: 0,25` = DNS error · `0,3` = socket connect fail ·
   `+CMQTTPUB: 0,26` = socket closed / not connected (follow-on noise, not a
   distinct fault). No URC before timeout usually means the command was still
   waiting — raise the wait, don't guess.

## Current state of `mbus_mqtt_gebos.lua` (v1.10) — temporary changes to revert

| Change | Why it's there | Revert when |
| --- | --- | --- |
| `mqtt_server = "178.104.178.108"` | bypass private-APN DNS | breakout exists **and** public DNS resolves |
| `AT+CSSLCFG="authmode",0,0` | no CA provisioned on modem | ISRG Root X1 loaded (`AT+CCERTDOWN`) + `cacert` configured → set authmode 1 |
| `mqtt_timeout = 60000` | see real modem result codes | after measuring real handshake time on a working network |
| PDP-context dump in `MQTT_SSL_Start()` | root-cause evidence | once connectivity is stable (costs battery/airtime per connect) |

## Options to get data flowing

1. **Preferred: ask ACRIOS** (SIM contract holder) to enable internet breakout
   on the APN, or whitelist `ingest.gebos.online` / `178.104.178.108:8883`
   for our SIMs. Hardware untouched.
2. **Own SIM:** any IoT SIM with public breakout (1NCE, EMnify, local carrier
   NB-IoT, …); set `APN = "<provider apn>"` explicitly in the Lua config.
   Check provider NB-IoT coverage on bands 3/5/8/20 (what the firmware
   configures).
3. Fallback: ACRIOS cloud bridge/forwarding — reintroduces the vendor
   dependency the self-hosted ingest stack exists to avoid.

## Verified-good along the way

- EMQX 5.8.6 (oci-container, host network) serving MQTTS :8883 with the ACME
  (HTTP-01) RSA-2048 cert; hot-reloads renewed PEMs.
- Broker auth: `bender` device login + ACL, external end-to-end publish test.
- Note for MQTTX/dashboard tests: EMQX default log level logs neither TLS
  handshake failures nor connects; use `emqx ctl log set-level debug`
  temporarily, and `emqx ctl listeners` for connection/shutdown counters.