Files
heltec-repeater/docs/TESTING.md
T

19 KiB
Raw Blame History

Testing

One-command HIL test

nix develop --command ./hil/run.sh --duration 300

Performs: device discovery (aborts on ambiguity) -> SDR USB reset -> firmware build + flash -> DUT configuration -> concurrent serial + SDR capture -> correlation -> report (text + report.json). Exit 0 = PASS.

Useful flags:

  • --no-flash reuse the already-flashed firmware
  • --duration N capture window seconds
  • --tx-power N configured SX1262 dBm (-9..+2; FEM adds gain!)
  • --delay MS repeat delay (20..1000; note: run.sh sets it on the DUT, which persists)
  • --gain G SDR tuner gain
  • --min-repeats N PASS threshold

Artifacts

Each run writes captures/<timestamp>-hil/:

serial.log      DUT JSON-lines (wmbus_rx / wmbus_tx / heartbeats / stats)
rtlwmbus.log    rtl_wmbus decodes
report.json     machine-readable correlation result
report.txt      human-readable report
setup.log       DUT configuration acks

Manual pieces

./tools/discover_devices.sh -v                 # device inventory
./tools/flash.sh                               # build + flash; no SDR required
./hil/capture_sdr.sh 60 /tmp/sdr.log           # SDR-only capture
./result-py/bin/python3 hil/dut_cmd.py PORT --listen 10 status stats
./result-py/bin/python3 hil/iq_energy.py iq.raw     # burst energy probe
./result-py/bin/python3 hil/iq_spectrum.py iq.raw   # spectrum sanity
./result-py/bin/python3 hil/sdr_health.py           # observer sensitivity
./firmware/test/run_tests.sh                        # frame + duty-ledger tests (sanitizers)

Hop-marker regression coverage in the host frame suite checks ELL precedence, short/long TPL headers, H/R and security-mode rejection, truncated frames, format A/B block boundaries, unchanged payload bytes and T1 encode/decode. The 2026-09-09 host smoke also marked three captured meter frames and passed them through wmbusmeters: ELL was reported as relayed, mode 5 as hc=1, and the unencrypted water telegram still decoded its readings. Encrypted captures were not decrypted (no meter keys); ciphertext preservation was checked byte-for-byte. Target firmware build and HIL CLI smoke passed; the new marking behavior has not been flashed or RF-verified.

DUT serial commands

All output is single-line JSON; every line the HIL parses starts with {.

status              current configuration + duty-cycle usage
stats               all counters (incl. tx_airtime_us, tx_duty_cycle_ppm)
clear_stats
radio_status        chip mode, command status, device errors, RSSI
radio_reset         queue a full radio re-init
probe               low-level scripted SPI/IRQ/TX diagnostic with raw dumps
test_tx             transmit one synthetic C1 frame
test_tx_t1          transmit one synthetic T1 frame (3-of-6, 50 kHz dev)
test_burst          transmit the same frame twice, repeat_delay_ms apart
inject standby      fault injection: leave the radio out of RX
inject sleep        fault injection: destroy the radio configuration
reboot

set repeat_delay_ms N     20..1000. PERSISTED across reboots (NVS)
set tx_power N            -9..+2 (configured SX1262 level; FEM adds gain)
set repeat_enabled 0|1
set verbose 0|1
set fem_rx_csd 0|1        KCT8103L CSD level while in RX (LNA experiment)
set preamble_bits N       0|8|16|24|32 preamble detector length
set duty_budget_ppm N     rolling-hour TX airtime cap; 1000 = 0.1 %, 0 = off.
                          0..100000. PERSISTED across reboots (NVS), like delay.
                          Re-enable after guard-off tests!
set test_lfield N         19..200, length of synthetic test frames
set beacon_period_ms N    0 (off) or 250..600000; periodic synthetic frames
set whitelist ID[,ID...] | set whitelist clear  (up to 8 unique eight-hex IDs)

The ten set commands, OLED toggles and web writes share validation and revisioned configuration. Integers/booleans must be complete, not valid-looking prefixes; malformed or duplicate IDs are rejected rather than truncated. status retains its previous fields and adds all missing settings, revision, applied_revision and the complete whitelist. Accepted values can briefly precede application by the radio task. A TX-power or preamble change requires radio reinitialization; a copied revision is not hardware/RF proof.

On-device UI (OLED + PRG button)

Standalone debug without a host (docs/FUTURE_UI.md for design + pins):

  • Pages (short press cycles, wraps): live status w/ duty bar -> health counters -> last telegram -> effective config -> Wi-Fi credentials -> Wi-Fi join QR.
  • Hold ~1.5 s for the page action, each logged as {"event":"ui_action",...}: status = toggle repeat, health = clear stats, config = toggle beacon at 3000 ms (duty-safe at L=19), either Wi-Fi page = enable/disable the debug access point.
  • Display blanks after 3 min; any press wakes it without acting.
  • {"event":"ui_init","ok":...} at boot reports display bring-up; a missing/broken display never blocks the repeater.
  • Config shown is the applied RAM snapshot. Delay and duty budget are restored from NVS on boot; other settings reset. Unplugging USB with LiPo attached does not reboot the device.

Wi-Fi dashboard checks

Enable WLAN-DIAGNOSE using the OLED, join the displayed WPA2 network, and open http://192.168.4.1. The following page provides a standard WIFI: join QR (SSID and password, not the dashboard URL). Wi-Fi starts off on reboot; its password persists in the separate wifi_debug NVS namespace. On enable/disable, serial emits wifi_debug states: 0 off, 1 starting, 2 on, 3 stopping, 4 error, plus the ESP-IDF error code. Credentials are not logged. Never change the test workstation's Wi-Fi without permission; phone-based testing avoids disturbing its network.

The history API remains read-only: GET /api/events?after=N. Start at zero; responses carry boot_id, cursor, retention/eviction metadata, cached status and immutable final rows. A stale/ahead cursor causes reset. Clients must fetch from zero on a new boot ID. The browser polls once per second and keeps at most 256 rows. Pause freezes the event display, not recording/polling. RX/repeat counters exclude synthetic events and survive OLED/serial clear_stats, but not reboot.

Writable settings API

Open Einstellungen am Heltec-Gerät ändern. Each field has its own draft, accepted device value, persistence indication and explicit submit action. The ten fields match the serial commands above; T1 synthetic sends still cap the effective test L-field at 140. The browser warns before duty 0 or >1000 and before disabling the AP. CSV names remain browser-only.

  • GET /api/settings returns boot_id, revision, applied_revision, csrf_token and settings containing all ten keys.
  • POST /api/settings accepts exactly {boot_id,revision,key,value}, one setting at a time. Values are JSON integers, booleans or an array of up to eight unique eight-hex-digit whitelist strings; an empty list allows all. The response has the same snapshot shape.
  • POST /api/wifi/disable accepts exactly {boot_id} and returns {ok:true} before requesting an idempotent stop from the separate Wi-Fi control task. A lost response leaves the client uncertain; it never retries automatically.
  • Writes require Content-Type: application/json and X-CSRF-Token from GET. Host must be 192.168.4.1 (optional port 80); foreign Origin/Fetch-Site values are rejected. The random CSRF token rotates when the AP starts. WPA2 AP access is the authorization boundary; there is no separate per-user login.
  • Bodies are bounded to 512 bytes and shallow JSON, with duplicate/unknown fields and embedded NULs rejected. Invalid inputs return 400, stale boot or revision 409, origin/token failures 403, and busy/persistence failures 503. No failed write is reported as saved.
  • Pending settings are copied coherently by the radio owner, never written into a half-processed reception from HTTP/OLED/USB. NVS runs under the writer mutex outside the radio task; short spinlocks protect only memory copies. Settings GETs while pending are capped at 1 Hz and 30 attempts, with no overlap or blind POST retry. Conflicts preserve drafts and require review.

Static assets are gzip-compressed at build time by tools/compress_web_asset.py, then served directly with Content-Encoding: gzip; no runtime compression or partition-layout change is required.

Acceptance scenarios:

  • See each reception's CRC/decode result and repeat completion, failure or explicit skip. A TX timestamp is software-operation entry (including driver setup), not an independently measured RF-start timestamp. A TX outcome timestamp can represent an error, not successful completion.
  • Load the downloaded CSV template or observed-ID export, preserving leading zeros. Quoted commas/newlines are supported. A duplicate ID or malformed CSV must leave all previous mappings unchanged. Refresh clears mappings. Labels apply only to CRC-valid received identities.
  • Check manufacturer-scoped type meanings (e.g. TCH), unknown raw codes, version bytes and ambiguity warnings. Identity refers to the link-layer sender, not necessarily a separately addressed transport-layer meter.
  • Pause while new events arrive, reconnect after history eviction, and reboot: no cross-boot or cross-reception joins are allowed.
  • Compare radio timing/health with Wi-Fi off, on and actively serving clients. Verify disable/re-enable and password persistence. Independently observe RF if claiming unchanged sensitivity or downstream reception.
  • Scan the OLED QR with a phone. Its integer-scaled, positive-polarity modules retain at least a four-module quiet zone. On this small OLED, camera focus can still be limiting; use text credentials if needed.
  • On 320/390-pixel phone viewports, check that the current signals are easy to find, controls are tappable and there is no horizontal page overflow. Open ? explanations by touch, including near the bottom of the page. Close with the same button, outside tap, explicit close and Escape. Explanations must remain usable while live rows update. All authored visible UI/accessibility text is German; technical raw codes, exact CSV/API headers and registered company names are intentionally kept.

2026-09-09 integration checks: test/debug builds and existing host suites passed. A sanitized history smoke exercised 200,000 writes with concurrent snapshots, overflow, cursor reset and synthetic/counter separation. The actual browser assets were exercised on a localhost fixture server: mobile layout, labels/filtering, atomic CSV rejection, downloaded template, pause, ambiguity, eviction, disconnect/reconnect, reboot and refresh. An independent ZBar decoder recovered the expected join payload from the exact 128x64 framebuffer generated by the production QR renderer.

The German UI was checked at 320/390 px, including popup scrolling, 44 px help/button targets, local German CSV labels, outside/same-button/ explicit-close/Escape dismissal and focus restoration. A detail popup remained open while the live history grew from five to six rows. Expanded rows initially created a phantom fourth column on phones; the corrected column spans were exercised across 320 -> 760 -> 390 px without reload. Page/header widths stayed aligned and there was no horizontal overflow. Browser checks used coordinate mouse input on narrow/mobile-emulated viewports; native touchscreen automation timed out, so a physical phone touch/QR check is still required. Evidence is retained locally under captures/2026-09-09-wifi-dashboard/.

Pre-settings image sizes were test 927,360 bytes and debug 1,041,264 bytes. The writable-settings build with compressed assets is now 862,672 bytes (test, 18% app space free) and 978,656 bytes (debug, 7% free). The partition layout and NVS locations were not changed.

After flashing the earlier, pre-settings German test build, USB telemetry at 59.74 s showed six reception candidates, one CRC-valid reception, one CRC rejection, no TX errors and no recoveries. Heartbeats reported RX mode and intact radio configuration. This confirms live firmware operation, not phone access.

The test-profile firmware was flashed to the attached ESP32-S3. In the pre-QR hardware smoke, the OLED action started the AP successfully; seven CRC-valid receptions produced five completed repeats with no recoveries or TX errors. Four repeats while the AP was on had software delays of 50.001–50.005 ms for a 50 ms setting. This small sample is not a controlled RX-sensitivity comparison, RF delivery proof, or proof of active HTTP load. Physical phone scanning and end-to-end HTTP checks require a phone report or an explicitly authorized network connection; do not infer them from the localhost fixture checks.

Writable-settings checks: 49 sanitized host scenarios ran the production settings.c, settings_http.c and ESP-IDF cJSON parser through small host HTTP/FreeRTOS/NVS adapters. They exercised all ten accepted settings, non-mutating validation/security failures, two concurrent revision-CAS writers, simulated storage failure/reboot and radio-copy acknowledgement. This is not physical NVS or radio-handoff verification.

Chromium exercised all ten forms against that coordinator, preserved unrelated drafts and edits made while a POST was delayed, rejected malformed input, handled 409/503 without automatic writes, and gated dangerous duty changes and Wi-Fi-off with native confirm/cancel dialogs. Confirmed Wi-Fi-off stopped automatic polling while retaining CSV names. Layout/help were inspected at 320/390 px; setting inputs are 16 px and buttons at least 44 px high. All five built gzip assets decompressed byte-for-byte to their sources and loaded in Chromium. Existing frame, duty-ledger, long-T1 and font suites passed.

The settings firmware is built but not yet flashed: the Heltec USB device disappeared before the flash command could open it; both serial discovery and USB enumeration found it absent. Reconnect USB for the final on-device setting, radio-handoff and reboot/persistence smoke. Workstation Wi-Fi was not changed. Physical phone HTTP/QR checks remain separate. Local proof artifacts are in captures/2026-09-09-web-settings/.

Known pitfalls

  • SDR PLL wedge -> gain-independent white noise; usbreset 0bda:2838.
  • Never wrap the rtl pipeline in timeout bash -c (orphans children); apply timeout to rtl_sdr itself.
  • ESP32-S3 USB re-enumerates between app CDC and ROM bootloader; flash scripts rediscover the port on failure.

LNA / FEM A/B experiment (no SDR needed)

nix develop --command ./hil/lna_ab.sh 600     # seconds per arm

Runs two RX-only arms (CSD high vs CSD low) with retransmission disabled and reports frame counts, CRC success, unique meters and RSSI per arm. Result: docs/RF_FRONTEND_TESTS.md.

Interpreting observer state

Every run records SDR sensitivity: full | degraded from hil/sdr_health.py. Treat cross-run comparisons as invalid unless both runs report the same state -- this dongle loses ~17 dB within minutes of streaming (docs/HARDWARE.md).

Meter-anchored TX verification

When serial telemetry and raw IQ were captured together:

./result-py/bin/python3 hil/meter_anchored.py \
  --iq iq.raw --serial serial.log --json

The verifier first fits the DUT/IQ clocks only from strong meter receptions, which prevents a repeat from borrowing its own original burst. It then compares the median spectrum at exact TX times with nearby non-TX control windows. rf_state is:

  • strong: event-correlated FSK is present and median event energy is at least 6 dB over the capture floor;
  • weak: event-correlated FSK is present below that energy threshold;
  • not_observed: no event-correlated FSK signature;
  • no_events: the serial log contained no in-range TX events.

tx_above_6db is only an energy count; it is not an RF verdict by itself. fsk_excess_db is the in-band (20-80 kHz offset) event/control excess after subtracting wideband (250-700 kHz offset) event/control change. At least five analyzable TX/control pairs are required.

For unattended beacon runs, hil/beacon_soak.sh discards raw IQ only when the result is strong. Weak, absent, or unclassified segments retain IQ for reanalysis.

Long reliability soak (DUT only, no SDR)

nix develop --command ./hil/soak.sh 9000     # seconds

Measures firmware-side reliability over hours: repeat success rate, timing stability, radio recoveries, config-integrity failures and MCU resets, and reports progress against the milestone targets. Deliberately does not use the SDR, which cannot stream for hours without faulting.

Fault injection (verifying the recovery ladder)

./result-py/bin/python3 hil/dut_cmd.py <port> --listen 26 "inject standby"
./result-py/bin/python3 hil/dut_cmd.py <port> --listen 26 "inject sleep"

standby leaves the radio out of RX with its configuration intact; sleep additionally destroys the configuration. The heartbeat probe must detect both and the recovery ladder must restore RX. Verified behaviour:

inject standby -> heartbeat chip_mode=2, config_intact=true
               -> radio_recovery level 1 -> rx_restored
inject sleep   -> heartbeat chip_mode=0, config_intact=false
               -> radio_recovery level 1 -> failed   (correctly refuses)
               -> radio_recovery level 2 -> rx_restored

Build profiles

./tools/build.sh test      # default; the configuration all HIL numbers use
./tools/build.sh debug     # verbose diagnostics, assertions, per-TX dumps
./tools/flash.sh [test|debug]

Differences are behavioural, not cosmetic:

test debug
verbose at boot off on
tx_diag after every TX only on failure always
raw frame hex in events yes (HIL needs it) yes
optimisation / log level size, warn debug, info
task watchdog 15 s 30 s

hil/run.sh always builds and flashes the test profile, so reported timing is never perturbed by debug serial traffic. Build directories are separate (build-test, build-debug) so profiles cannot silently reuse a stale sdkconfig.

Verifying a standalone repeater (no serial connection)

When the DUT runs on external power with no USB host, firmware telemetry is unavailable and RF is the only evidence. Record IQ once, then derive both decodes and RF-level repeat pairs from the same samples:

timeout 80 rtl_sdr -f 868950000 -s 1600000 -g 42.1 -n 96000000 iq.raw
cat iq.raw | rtl_wmbus -f > decodes.log
./result-py/bin/python3 hil/verify_standalone.py --iq iq.raw --decodes decodes.log

Reports decoded C1/T1 repeats (a supported H=0 original followed by its H=1 counterpart, with every other cooked byte identical) and RF-level repeat pairs (burst pairs of matching duration inside the delay window). Decoded pairs use occurrence order because SDR timestamps are quantized; each original is matched once. Same-H duplicate decodes are not repeat proof. The RF-level check matters because our transmission usually overloads the dongle and fails CRC even though the RF is correct.

Note: only one process can hold the RTL-SDR. If another tool (e.g. an oms-tui / wmbusmeters pipeline) is running, rtl_sdr fails with usb_claim_interface error -6 and it must be stopped first.