# Testing ## One-command HIL test ``` nix develop --command ./hil/run.sh --duration 300 ``` Performs: device discovery (aborts on ambiguity) -> SDR USB reset -> firmware build + flash -> DUT configuration -> concurrent serial + SDR capture -> correlation -> report (text + report.json). Exit 0 = PASS. Useful flags: - `--no-flash` reuse the already-flashed firmware - `--duration N` capture window seconds - `--tx-power N` configured SX1262 dBm (-9..+2; FEM adds gain!) - `--delay MS` repeat delay (20..1000; note: run.sh sets it on the DUT, which persists) - `--gain G` SDR tuner gain - `--min-repeats N` PASS threshold ## Artifacts Each run writes `captures/-hil/`: ``` serial.log DUT JSON-lines (wmbus_rx / wmbus_tx / heartbeats / stats) rtlwmbus.log rtl_wmbus decodes report.json machine-readable correlation result report.txt human-readable report setup.log DUT configuration acks ``` ## Manual pieces ``` ./tools/discover_devices.sh -v # device inventory ./tools/flash.sh # build + flash; no SDR required ./hil/capture_sdr.sh 60 /tmp/sdr.log # SDR-only capture ./result-py/bin/python3 hil/dut_cmd.py PORT --listen 10 status stats ./result-py/bin/python3 hil/iq_energy.py iq.raw # burst energy probe ./result-py/bin/python3 hil/iq_spectrum.py iq.raw # spectrum sanity ./result-py/bin/python3 hil/sdr_health.py # observer sensitivity ./firmware/test/run_tests.sh # frame + duty-ledger tests (sanitizers) ``` Hop-marker regression coverage in the host frame suite checks ELL precedence, short/long TPL headers, H/R and security-mode rejection, truncated frames, format A/B block boundaries, unchanged payload bytes and T1 encode/decode. The 2026-09-09 host smoke also marked three captured meter frames and passed them through `wmbusmeters`: ELL was reported as `relayed`, mode 5 as `hc=1`, and the unencrypted water telegram still decoded its readings. Encrypted captures were not decrypted (no meter keys); ciphertext preservation was checked byte-for-byte. Target firmware build and HIL CLI smoke passed; the new marking behavior has not been flashed or RF-verified. ## DUT serial commands All output is single-line JSON; every line the HIL parses starts with `{`. ``` status current configuration + duty-cycle usage stats all counters (incl. tx_airtime_us, tx_duty_cycle_ppm) clear_stats radio_status chip mode, command status, device errors, RSSI radio_reset queue a full radio re-init probe low-level scripted SPI/IRQ/TX diagnostic with raw dumps test_tx transmit one synthetic C1 frame test_tx_t1 transmit one synthetic T1 frame (3-of-6, 50 kHz dev) test_burst transmit the same frame twice, repeat_delay_ms apart inject standby fault injection: leave the radio out of RX inject sleep fault injection: destroy the radio configuration reboot set repeat_delay_ms N 20..1000. PERSISTED across reboots (NVS) set tx_power N -9..+2 (configured SX1262 level; FEM adds gain) set repeat_enabled 0|1 set verbose 0|1 set fem_rx_csd 0|1 KCT8103L CSD level while in RX (LNA experiment) set preamble_bits N 0|8|16|24|32 preamble detector length set duty_budget_ppm N rolling-hour TX airtime cap; 1000 = 0.1 %, 0 = off. 0..100000. PERSISTED across reboots (NVS), like delay. Re-enable after guard-off tests! set test_lfield N 19..200, length of synthetic test frames set beacon_period_ms N 0 (off) or 250..600000; periodic synthetic frames set whitelist ID[,ID...] | set whitelist clear (up to 8 unique eight-hex IDs) ``` The ten `set` commands, OLED toggles and web writes share validation and revisioned configuration. Integers/booleans must be complete, not valid-looking prefixes; malformed or duplicate IDs are rejected rather than truncated. `status` retains its previous fields and adds all missing settings, `revision`, `applied_revision` and the complete whitelist. Accepted values can briefly precede application by the radio task. A TX-power or preamble change requires radio reinitialization; a copied revision is not hardware/RF proof. ## On-device UI (OLED + PRG button) Standalone debug without a host (docs/FUTURE_UI.md for design + pins): - Pages (short press cycles, wraps): live status w/ duty bar -> health counters -> last telegram -> effective config -> Wi-Fi credentials -> Wi-Fi join QR. - Hold ~1.5 s for the page action, each logged as `{"event":"ui_action",...}`: status = toggle repeat, health = clear stats, config = toggle beacon at 3000 ms (duty-safe at L=19), either Wi-Fi page = enable/disable the debug access point. - Display blanks after 3 min; any press wakes it without acting. - `{"event":"ui_init","ok":...}` at boot reports display bring-up; a missing/broken display never blocks the repeater. - Config shown is the applied RAM snapshot. Delay and duty budget are restored from NVS on boot; other settings reset. Unplugging USB with LiPo attached does not reboot the device. ## Wi-Fi dashboard checks Enable **WLAN-DIAGNOSE** using the OLED, join the displayed WPA2 network, and open `http://192.168.4.1`. The following page provides a standard `WIFI:` join QR (SSID and password, not the dashboard URL). Wi-Fi starts off on reboot; its password persists in the separate `wifi_debug` NVS namespace. On enable/disable, serial emits `wifi_debug` states: 0 off, 1 starting, 2 on, 3 stopping, 4 error, plus the ESP-IDF error code. Credentials are not logged. Never change the test workstation's Wi-Fi without permission; phone-based testing avoids disturbing its network. The history API remains read-only: `GET /api/events?after=N`. Start at zero; responses carry `boot_id`, cursor, retention/eviction metadata, cached status and immutable final rows. A stale/ahead cursor causes `reset`. Clients must fetch from zero on a new boot ID. The browser polls once per second and keeps at most 256 rows. Pause freezes the event display, not recording/polling. RX/repeat counters exclude synthetic events and survive OLED/serial `clear_stats`, but not reboot. ### Writable settings API Open **Einstellungen am Heltec-Gerät ändern**. Each field has its own draft, accepted device value, persistence indication and explicit submit action. The ten fields match the serial commands above; T1 synthetic sends still cap the effective test L-field at 140. The browser warns before duty 0 or >1000 and before disabling the AP. CSV names remain browser-only. - `GET /api/settings` returns `boot_id`, `revision`, `applied_revision`, `csrf_token` and `settings` containing all ten keys. - `POST /api/settings` accepts exactly `{boot_id,revision,key,value}`, one setting at a time. Values are JSON integers, booleans or an array of up to eight unique eight-hex-digit whitelist strings; an empty list allows all. The response has the same snapshot shape. - `POST /api/wifi/disable` accepts exactly `{boot_id}` and returns `{ok:true}` before requesting an idempotent stop from the separate Wi-Fi control task. A lost response leaves the client uncertain; it never retries automatically. - Writes require `Content-Type: application/json` and `X-CSRF-Token` from GET. Host must be `192.168.4.1` (optional port 80); foreign Origin/Fetch-Site values are rejected. The random CSRF token rotates when the AP starts. WPA2 AP access is the authorization boundary; there is no separate per-user login. - Bodies are bounded to 512 bytes and shallow JSON, with duplicate/unknown fields and embedded NULs rejected. Invalid inputs return 400, stale boot or revision 409, origin/token failures 403, and busy/persistence failures 503. No failed write is reported as saved. - Pending settings are copied coherently by the radio owner, never written into a half-processed reception from HTTP/OLED/USB. NVS runs under the writer mutex outside the radio task; short spinlocks protect only memory copies. Settings GETs while pending are capped at 1 Hz and 30 attempts, with no overlap or blind POST retry. Conflicts preserve drafts and require review. Static assets are gzip-compressed at build time by `tools/compress_web_asset.py`, then served directly with `Content-Encoding: gzip`; no runtime compression or partition-layout change is required. Acceptance scenarios: - See each reception's CRC/decode result and repeat completion, failure or explicit skip. A TX timestamp is software-operation entry (including driver setup), not an independently measured RF-start timestamp. A TX outcome timestamp can represent an error, not successful completion. - Load the downloaded CSV template or observed-ID export, preserving leading zeros. Quoted commas/newlines are supported. A duplicate ID or malformed CSV must leave all previous mappings unchanged. Refresh clears mappings. Labels apply only to CRC-valid received identities. - Check manufacturer-scoped type meanings (e.g. TCH), unknown raw codes, version bytes and ambiguity warnings. Identity refers to the link-layer sender, not necessarily a separately addressed transport-layer meter. - Pause while new events arrive, reconnect after history eviction, and reboot: no cross-boot or cross-reception joins are allowed. - Compare radio timing/health with Wi-Fi off, on and actively serving clients. Verify disable/re-enable and password persistence. Independently observe RF if claiming unchanged sensitivity or downstream reception. - Scan the OLED QR with a phone. Its integer-scaled, positive-polarity modules retain at least a four-module quiet zone. On this small OLED, camera focus can still be limiting; use text credentials if needed. - On 320/390-pixel phone viewports, check that the current signals are easy to find, controls are tappable and there is no horizontal page overflow. Open **?** explanations by touch, including near the bottom of the page. Close with the same button, outside tap, explicit close and Escape. Explanations must remain usable while live rows update. All authored visible UI/accessibility text is German; technical raw codes, exact CSV/API headers and registered company names are intentionally kept. 2026-09-09 integration checks: test/debug builds and existing host suites passed. A sanitized history smoke exercised 200,000 writes with concurrent snapshots, overflow, cursor reset and synthetic/counter separation. The actual browser assets were exercised on a localhost fixture server: mobile layout, labels/filtering, atomic CSV rejection, downloaded template, pause, ambiguity, eviction, disconnect/reconnect, reboot and refresh. An independent ZBar decoder recovered the expected join payload from the exact 128x64 framebuffer generated by the production QR renderer. The German UI was checked at 320/390 px, including popup scrolling, 44 px help/button targets, local German CSV labels, outside/same-button/ explicit-close/Escape dismissal and focus restoration. A detail popup remained open while the live history grew from five to six rows. Expanded rows initially created a phantom fourth column on phones; the corrected column spans were exercised across 320 -> 760 -> 390 px without reload. Page/header widths stayed aligned and there was no horizontal overflow. Browser checks used coordinate mouse input on narrow/mobile-emulated viewports; native touchscreen automation timed out, so a physical phone touch/QR check is still required. Evidence is retained locally under `captures/2026-09-09-wifi-dashboard/`. Pre-settings image sizes were test 927,360 bytes and debug 1,041,264 bytes. The writable-settings build with compressed assets is now 862,672 bytes (test, 18% app space free) and 978,656 bytes (debug, 7% free). The partition layout and NVS locations were not changed. After flashing the earlier, pre-settings German test build, USB telemetry at 59.74 s showed six reception candidates, one CRC-valid reception, one CRC rejection, no TX errors and no recoveries. Heartbeats reported RX mode and intact radio configuration. This confirms live firmware operation, not phone access. The test-profile firmware was flashed to the attached ESP32-S3. In the pre-QR hardware smoke, the OLED action started the AP successfully; seven CRC-valid receptions produced five completed repeats with no recoveries or TX errors. Four repeats while the AP was on had software delays of 50.001–50.005 ms for a 50 ms setting. This small sample is not a controlled RX-sensitivity comparison, RF delivery proof, or proof of active HTTP load. Physical phone scanning and end-to-end HTTP checks require a phone report or an explicitly authorized network connection; do not infer them from the localhost fixture checks. Writable-settings checks: 49 sanitized host scenarios ran the production `settings.c`, `settings_http.c` and ESP-IDF cJSON parser through small host HTTP/FreeRTOS/NVS adapters. They exercised all ten accepted settings, non-mutating validation/security failures, two concurrent revision-CAS writers, simulated storage failure/reboot and radio-copy acknowledgement. This is not physical NVS or radio-handoff verification. Chromium exercised all ten forms against that coordinator, preserved unrelated drafts and edits made while a POST was delayed, rejected malformed input, handled 409/503 without automatic writes, and gated dangerous duty changes and Wi-Fi-off with native confirm/cancel dialogs. Confirmed Wi-Fi-off stopped automatic polling while retaining CSV names. Layout/help were inspected at 320/390 px; setting inputs are 16 px and buttons at least 44 px high. All five built gzip assets decompressed byte-for-byte to their sources and loaded in Chromium. Existing frame, duty-ledger, long-T1 and font suites passed. The settings firmware is built but **not yet flashed**: the Heltec USB device disappeared before the flash command could open it; both serial discovery and USB enumeration found it absent. Reconnect USB for the final on-device setting, radio-handoff and reboot/persistence smoke. Workstation Wi-Fi was not changed. Physical phone HTTP/QR checks remain separate. Local proof artifacts are in `captures/2026-09-09-web-settings/`. ## Known pitfalls - SDR PLL wedge -> gain-independent white noise; `usbreset 0bda:2838`. - Never wrap the rtl pipeline in `timeout bash -c` (orphans children); apply timeout to `rtl_sdr` itself. - ESP32-S3 USB re-enumerates between app CDC and ROM bootloader; flash scripts rediscover the port on failure. ## LNA / FEM A/B experiment (no SDR needed) ``` nix develop --command ./hil/lna_ab.sh 600 # seconds per arm ``` Runs two RX-only arms (CSD high vs CSD low) with retransmission disabled and reports frame counts, CRC success, unique meters and RSSI per arm. Result: docs/RF_FRONTEND_TESTS.md. ## Interpreting observer state Every run records `SDR sensitivity: full | degraded` from `hil/sdr_health.py`. Treat cross-run comparisons as invalid unless both runs report the same state -- this dongle loses ~17 dB within minutes of streaming (docs/HARDWARE.md). ## Meter-anchored TX verification When serial telemetry and raw IQ were captured together: ``` ./result-py/bin/python3 hil/meter_anchored.py \ --iq iq.raw --serial serial.log --json ``` The verifier first fits the DUT/IQ clocks only from strong meter receptions, which prevents a repeat from borrowing its own original burst. It then compares the median spectrum at exact TX times with nearby non-TX control windows. `rf_state` is: - `strong`: event-correlated FSK is present and median event energy is at least 6 dB over the capture floor; - `weak`: event-correlated FSK is present below that energy threshold; - `not_observed`: no event-correlated FSK signature; - `no_events`: the serial log contained no in-range TX events. `tx_above_6db` is only an energy count; it is not an RF verdict by itself. `fsk_excess_db` is the in-band (20-80 kHz offset) event/control excess after subtracting wideband (250-700 kHz offset) event/control change. At least five analyzable TX/control pairs are required. For unattended beacon runs, `hil/beacon_soak.sh` discards raw IQ only when the result is `strong`. Weak, absent, or unclassified segments retain IQ for reanalysis. ## Long reliability soak (DUT only, no SDR) ``` nix develop --command ./hil/soak.sh 9000 # seconds ``` Measures firmware-side reliability over hours: repeat success rate, timing stability, radio recoveries, config-integrity failures and MCU resets, and reports progress against the milestone targets. Deliberately does not use the SDR, which cannot stream for hours without faulting. ## Fault injection (verifying the recovery ladder) ``` ./result-py/bin/python3 hil/dut_cmd.py --listen 26 "inject standby" ./result-py/bin/python3 hil/dut_cmd.py --listen 26 "inject sleep" ``` `standby` leaves the radio out of RX with its configuration intact; `sleep` additionally destroys the configuration. The heartbeat probe must detect both and the recovery ladder must restore RX. Verified behaviour: ``` inject standby -> heartbeat chip_mode=2, config_intact=true -> radio_recovery level 1 -> rx_restored inject sleep -> heartbeat chip_mode=0, config_intact=false -> radio_recovery level 1 -> failed (correctly refuses) -> radio_recovery level 2 -> rx_restored ``` ## Build profiles ``` ./tools/build.sh test # default; the configuration all HIL numbers use ./tools/build.sh debug # verbose diagnostics, assertions, per-TX dumps ./tools/flash.sh [test|debug] ``` Differences are behavioural, not cosmetic: | | test | debug | |---|---|---| | `verbose` at boot | off | on | | `tx_diag` after every TX | only on failure | always | | raw frame hex in events | yes (HIL needs it) | yes | | optimisation / log level | size, warn | debug, info | | task watchdog | 15 s | 30 s | `hil/run.sh` always builds and flashes the `test` profile, so reported timing is never perturbed by debug serial traffic. Build directories are separate (`build-test`, `build-debug`) so profiles cannot silently reuse a stale sdkconfig. ## Verifying a standalone repeater (no serial connection) When the DUT runs on external power with no USB host, firmware telemetry is unavailable and RF is the only evidence. Record IQ once, then derive both decodes and RF-level repeat pairs from the same samples: ``` timeout 80 rtl_sdr -f 868950000 -s 1600000 -g 42.1 -n 96000000 iq.raw cat iq.raw | rtl_wmbus -f > decodes.log ./result-py/bin/python3 hil/verify_standalone.py --iq iq.raw --decodes decodes.log ``` Reports decoded C1/T1 repeats (a supported H=0 original followed by its H=1 counterpart, with every other cooked byte identical) and RF-level repeat pairs (burst pairs of matching duration inside the delay window). Decoded pairs use occurrence order because SDR timestamps are quantized; each original is matched once. Same-H duplicate decodes are not repeat proof. The RF-level check matters because our transmission usually overloads the dongle and fails CRC even though the RF is correct. Note: only one process can hold the RTL-SDR. If another tool (e.g. an oms-tui / wmbusmeters pipeline) is running, `rtl_sdr` fails with `usb_claim_interface error -6` and it must be stopped first.