390 lines
19 KiB
Markdown
390 lines
19 KiB
Markdown
# Testing
|
||
|
||
## One-command HIL test
|
||
|
||
```
|
||
nix develop --command ./hil/run.sh --duration 300
|
||
```
|
||
|
||
Performs: device discovery (aborts on ambiguity) -> SDR USB reset ->
|
||
firmware build + flash -> DUT configuration -> concurrent serial + SDR
|
||
capture -> correlation -> report (text + report.json). Exit 0 = PASS.
|
||
|
||
Useful flags:
|
||
- `--no-flash` reuse the already-flashed firmware
|
||
- `--duration N` capture window seconds
|
||
- `--tx-power N` configured SX1262 dBm (-9..+2; FEM adds gain!)
|
||
- `--delay MS` repeat delay (20..1000; note: run.sh sets it on the DUT, which persists)
|
||
- `--gain G` SDR tuner gain
|
||
- `--min-repeats N` PASS threshold
|
||
|
||
## Artifacts
|
||
|
||
Each run writes `captures/<timestamp>-hil/`:
|
||
```
|
||
serial.log DUT JSON-lines (wmbus_rx / wmbus_tx / heartbeats / stats)
|
||
rtlwmbus.log rtl_wmbus decodes
|
||
report.json machine-readable correlation result
|
||
report.txt human-readable report
|
||
setup.log DUT configuration acks
|
||
```
|
||
|
||
## Manual pieces
|
||
|
||
```
|
||
./tools/discover_devices.sh -v # device inventory
|
||
./tools/flash.sh # build + flash; no SDR required
|
||
./hil/capture_sdr.sh 60 /tmp/sdr.log # SDR-only capture
|
||
./result-py/bin/python3 hil/dut_cmd.py PORT --listen 10 status stats
|
||
./result-py/bin/python3 hil/iq_energy.py iq.raw # burst energy probe
|
||
./result-py/bin/python3 hil/iq_spectrum.py iq.raw # spectrum sanity
|
||
./result-py/bin/python3 hil/sdr_health.py # observer sensitivity
|
||
./firmware/test/run_tests.sh # frame + duty-ledger tests (sanitizers)
|
||
```
|
||
|
||
Hop-marker regression coverage in the host frame suite checks ELL precedence,
|
||
short/long TPL headers, H/R and security-mode rejection, truncated frames,
|
||
format A/B block boundaries, unchanged payload bytes and T1 encode/decode.
|
||
The 2026-09-09 host smoke also marked three captured meter frames and passed
|
||
them through `wmbusmeters`: ELL was reported as `relayed`, mode 5 as `hc=1`,
|
||
and the unencrypted water telegram still decoded its readings. Encrypted
|
||
captures were not decrypted (no meter keys); ciphertext preservation was
|
||
checked byte-for-byte. Target firmware build and HIL CLI smoke passed;
|
||
the new marking behavior has not been flashed or RF-verified.
|
||
|
||
## DUT serial commands
|
||
|
||
All output is single-line JSON; every line the HIL parses starts with `{`.
|
||
|
||
```
|
||
status current configuration + duty-cycle usage
|
||
stats all counters (incl. tx_airtime_us, tx_duty_cycle_ppm)
|
||
clear_stats
|
||
radio_status chip mode, command status, device errors, RSSI
|
||
radio_reset queue a full radio re-init
|
||
probe low-level scripted SPI/IRQ/TX diagnostic with raw dumps
|
||
test_tx transmit one synthetic C1 frame
|
||
test_tx_t1 transmit one synthetic T1 frame (3-of-6, 50 kHz dev)
|
||
test_burst transmit the same frame twice, repeat_delay_ms apart
|
||
inject standby fault injection: leave the radio out of RX
|
||
inject sleep fault injection: destroy the radio configuration
|
||
reboot
|
||
|
||
set repeat_delay_ms N 20..1000. PERSISTED across reboots (NVS)
|
||
set tx_power N -9..+2 (configured SX1262 level; FEM adds gain)
|
||
set repeat_enabled 0|1
|
||
set verbose 0|1
|
||
set fem_rx_csd 0|1 KCT8103L CSD level while in RX (LNA experiment)
|
||
set preamble_bits N 0|8|16|24|32 preamble detector length
|
||
set duty_budget_ppm N rolling-hour TX airtime cap; 1000 = 0.1 %, 0 = off.
|
||
0..100000. PERSISTED across reboots (NVS), like delay.
|
||
Re-enable after guard-off tests!
|
||
set test_lfield N 19..200, length of synthetic test frames
|
||
set beacon_period_ms N 0 (off) or 250..600000; periodic synthetic frames
|
||
set whitelist ID[,ID...] | set whitelist clear (up to 8 unique eight-hex IDs)
|
||
```
|
||
|
||
The ten `set` commands, OLED toggles and web writes share validation and
|
||
revisioned configuration. Integers/booleans must be complete, not valid-looking
|
||
prefixes; malformed or duplicate IDs are rejected rather than truncated.
|
||
`status` retains its previous fields and adds all missing settings,
|
||
`revision`, `applied_revision` and the complete whitelist. Accepted values can
|
||
briefly precede application by the radio task. A TX-power or preamble change
|
||
requires radio reinitialization; a copied revision is not hardware/RF proof.
|
||
|
||
## On-device UI (OLED + PRG button)
|
||
|
||
Standalone debug without a host (docs/FUTURE_UI.md for design + pins):
|
||
|
||
- Pages (short press cycles, wraps): live status w/ duty bar -> health
|
||
counters -> last telegram -> effective config -> Wi-Fi credentials ->
|
||
Wi-Fi join QR.
|
||
- Hold ~1.5 s for the page action, each logged as
|
||
`{"event":"ui_action",...}`: status = toggle repeat, health = clear
|
||
stats, config = toggle beacon at 3000 ms (duty-safe at L=19), either
|
||
Wi-Fi page = enable/disable the debug access point.
|
||
- Display blanks after 3 min; any press wakes it without acting.
|
||
- `{"event":"ui_init","ok":...}` at boot reports display bring-up; a
|
||
missing/broken display never blocks the repeater.
|
||
- Config shown is the applied RAM snapshot. Delay and duty budget are restored
|
||
from NVS on boot; other settings reset. Unplugging USB with LiPo attached
|
||
does not reboot the device.
|
||
|
||
## Wi-Fi dashboard checks
|
||
|
||
Enable **WLAN-DIAGNOSE** using the OLED, join the displayed WPA2 network, and
|
||
open `http://192.168.4.1`. The following page provides a standard `WIFI:`
|
||
join QR (SSID and password, not the dashboard URL). Wi-Fi starts off on
|
||
reboot; its password persists in the separate `wifi_debug` NVS namespace.
|
||
On enable/disable, serial emits `wifi_debug` states: 0 off, 1 starting,
|
||
2 on, 3 stopping, 4 error, plus the ESP-IDF error code. Credentials are
|
||
not logged. Never change the test workstation's Wi-Fi without permission;
|
||
phone-based testing avoids disturbing its network.
|
||
|
||
The history API remains read-only: `GET /api/events?after=N`. Start at zero;
|
||
responses carry `boot_id`, cursor, retention/eviction metadata, cached
|
||
status and immutable final rows. A stale/ahead cursor causes `reset`.
|
||
Clients must fetch from zero on a new boot ID. The browser polls once per
|
||
second and keeps at most 256 rows. Pause freezes the event display, not
|
||
recording/polling. RX/repeat counters exclude synthetic events and survive
|
||
OLED/serial `clear_stats`, but not reboot.
|
||
|
||
### Writable settings API
|
||
|
||
Open **Einstellungen am Heltec-Gerät ändern**. Each field has its own draft,
|
||
accepted device value, persistence indication and explicit submit action.
|
||
The ten fields match the serial commands above; T1 synthetic sends still cap
|
||
the effective test L-field at 140. The browser warns before duty 0 or >1000
|
||
and before disabling the AP. CSV names remain browser-only.
|
||
|
||
- `GET /api/settings` returns `boot_id`, `revision`, `applied_revision`,
|
||
`csrf_token` and `settings` containing all ten keys.
|
||
- `POST /api/settings` accepts exactly `{boot_id,revision,key,value}`, one
|
||
setting at a time. Values are JSON integers, booleans or an array of up to
|
||
eight unique eight-hex-digit whitelist strings; an empty list allows all.
|
||
The response has the same snapshot shape.
|
||
- `POST /api/wifi/disable` accepts exactly `{boot_id}` and returns `{ok:true}`
|
||
before requesting an idempotent stop from the separate Wi-Fi control task.
|
||
A lost response leaves the client uncertain; it never retries automatically.
|
||
- Writes require `Content-Type: application/json` and `X-CSRF-Token` from GET.
|
||
Host must be `192.168.4.1` (optional port 80); foreign Origin/Fetch-Site values
|
||
are rejected. The random CSRF token rotates when the AP starts. WPA2 AP
|
||
access is the authorization boundary; there is no separate per-user login.
|
||
- Bodies are bounded to 512 bytes and shallow JSON, with duplicate/unknown
|
||
fields and embedded NULs rejected. Invalid inputs return 400, stale boot or
|
||
revision 409, origin/token failures 403, and busy/persistence failures 503.
|
||
No failed write is reported as saved.
|
||
- Pending settings are copied coherently by the radio owner, never written
|
||
into a half-processed reception from HTTP/OLED/USB. NVS runs under the writer
|
||
mutex outside the radio task; short spinlocks protect only memory copies.
|
||
Settings GETs while pending are capped at 1 Hz and 30 attempts, with no
|
||
overlap or blind POST retry. Conflicts preserve drafts and require review.
|
||
|
||
Static assets are gzip-compressed at build time by `tools/compress_web_asset.py`,
|
||
then served directly with `Content-Encoding: gzip`; no runtime compression or
|
||
partition-layout change is required.
|
||
|
||
Acceptance scenarios:
|
||
|
||
- See each reception's CRC/decode result and repeat completion, failure or
|
||
explicit skip. A TX timestamp is software-operation entry (including
|
||
driver setup), not an independently measured RF-start timestamp. A TX
|
||
outcome timestamp can represent an error, not successful completion.
|
||
- Load the downloaded CSV template or observed-ID export, preserving
|
||
leading zeros. Quoted commas/newlines are supported. A duplicate ID or
|
||
malformed CSV must leave all previous mappings unchanged. Refresh clears
|
||
mappings. Labels apply only to CRC-valid received identities.
|
||
- Check manufacturer-scoped type meanings (e.g. TCH), unknown raw codes,
|
||
version bytes and ambiguity warnings. Identity refers to the link-layer
|
||
sender, not necessarily a separately addressed transport-layer meter.
|
||
- Pause while new events arrive, reconnect after history eviction, and
|
||
reboot: no cross-boot or cross-reception joins are allowed.
|
||
- Compare radio timing/health with Wi-Fi off, on and actively serving
|
||
clients. Verify disable/re-enable and password persistence. Independently
|
||
observe RF if claiming unchanged sensitivity or downstream reception.
|
||
- Scan the OLED QR with a phone. Its integer-scaled, positive-polarity
|
||
modules retain at least a four-module quiet zone. On this small OLED,
|
||
camera focus can still be limiting; use text credentials if needed.
|
||
- On 320/390-pixel phone viewports, check that the current signals are easy
|
||
to find, controls are tappable and there is no horizontal page overflow.
|
||
Open **?** explanations by touch, including near the bottom of the page.
|
||
Close with the same button, outside tap, explicit close and Escape.
|
||
Explanations must remain usable while live rows update.
|
||
All authored visible UI/accessibility text is German; technical raw codes,
|
||
exact CSV/API headers and registered company names are intentionally kept.
|
||
|
||
2026-09-09 integration checks: test/debug builds and existing host suites
|
||
passed. A sanitized history smoke exercised 200,000 writes with concurrent
|
||
snapshots, overflow, cursor reset and synthetic/counter separation.
|
||
The actual browser assets were exercised on a localhost fixture server:
|
||
mobile layout, labels/filtering, atomic CSV rejection, downloaded template,
|
||
pause, ambiguity, eviction, disconnect/reconnect, reboot and refresh.
|
||
An independent ZBar decoder recovered the expected join payload from the
|
||
exact 128x64 framebuffer generated by the production QR renderer.
|
||
|
||
The German UI was checked at 320/390 px, including popup scrolling,
|
||
44 px help/button targets, local German CSV labels, outside/same-button/
|
||
explicit-close/Escape dismissal and focus restoration. A detail popup
|
||
remained open while the live history grew from five to six rows. Expanded
|
||
rows initially created a phantom fourth column on phones; the corrected
|
||
column spans were exercised across 320 -> 760 -> 390 px without reload.
|
||
Page/header widths stayed aligned and there was no horizontal overflow.
|
||
Browser checks used coordinate mouse input on narrow/mobile-emulated
|
||
viewports; native touchscreen automation timed out, so a physical phone
|
||
touch/QR check is still required. Evidence is retained locally under
|
||
`captures/2026-09-09-wifi-dashboard/`.
|
||
|
||
Pre-settings image sizes were test 927,360 bytes and debug 1,041,264 bytes.
|
||
The writable-settings build with compressed assets is now 862,672 bytes
|
||
(test, 18% app space free) and 978,656 bytes (debug, 7% free). The partition
|
||
layout and NVS locations were not changed.
|
||
|
||
After flashing the earlier, pre-settings German test build, USB telemetry at 59.74 s showed
|
||
six reception candidates, one CRC-valid reception, one CRC rejection, no
|
||
TX errors and no recoveries. Heartbeats reported RX mode and intact radio
|
||
configuration. This confirms live firmware operation, not phone access.
|
||
|
||
The test-profile firmware was flashed to the attached ESP32-S3. In the
|
||
pre-QR hardware smoke, the OLED action started the AP successfully; seven
|
||
CRC-valid receptions produced five completed repeats with no recoveries or
|
||
TX errors. Four repeats while the AP was on had software delays of
|
||
50.001–50.005 ms for a 50 ms setting. This small sample is not a controlled
|
||
RX-sensitivity comparison, RF delivery proof, or proof of active HTTP load.
|
||
Physical phone scanning and end-to-end HTTP checks require a phone report
|
||
or an explicitly authorized network connection; do not infer them from the
|
||
localhost fixture checks.
|
||
|
||
Writable-settings checks: 49 sanitized host scenarios ran the production
|
||
`settings.c`, `settings_http.c` and ESP-IDF cJSON parser through small host
|
||
HTTP/FreeRTOS/NVS adapters. They exercised all ten accepted settings,
|
||
non-mutating validation/security failures, two concurrent revision-CAS writers,
|
||
simulated storage failure/reboot and radio-copy acknowledgement. This is not
|
||
physical NVS or radio-handoff verification.
|
||
|
||
Chromium exercised all ten forms against that coordinator, preserved unrelated
|
||
drafts and edits made while a POST was delayed, rejected malformed input,
|
||
handled 409/503 without automatic writes, and gated dangerous duty changes and
|
||
Wi-Fi-off with native confirm/cancel dialogs. Confirmed Wi-Fi-off stopped
|
||
automatic polling while retaining CSV names. Layout/help were inspected at
|
||
320/390 px; setting inputs are 16 px and buttons at least 44 px high. All five
|
||
built gzip assets decompressed byte-for-byte to their sources and loaded in
|
||
Chromium. Existing frame, duty-ledger, long-T1 and font suites passed.
|
||
|
||
The settings firmware is built but **not yet flashed**: the Heltec USB device
|
||
disappeared before the flash command could open it; both serial discovery and
|
||
USB enumeration found it absent. Reconnect USB for the final on-device setting,
|
||
radio-handoff and reboot/persistence smoke. Workstation Wi-Fi was not changed.
|
||
Physical phone HTTP/QR checks remain separate. Local proof artifacts are in
|
||
`captures/2026-09-09-web-settings/`.
|
||
|
||
## Known pitfalls
|
||
|
||
- SDR PLL wedge -> gain-independent white noise; `usbreset 0bda:2838`.
|
||
- Never wrap the rtl pipeline in `timeout bash -c` (orphans children);
|
||
apply timeout to `rtl_sdr` itself.
|
||
- ESP32-S3 USB re-enumerates between app CDC and ROM bootloader; flash
|
||
scripts rediscover the port on failure.
|
||
|
||
## LNA / FEM A/B experiment (no SDR needed)
|
||
|
||
```
|
||
nix develop --command ./hil/lna_ab.sh 600 # seconds per arm
|
||
```
|
||
|
||
Runs two RX-only arms (CSD high vs CSD low) with retransmission disabled
|
||
and reports frame counts, CRC success, unique meters and RSSI per arm.
|
||
Result: docs/RF_FRONTEND_TESTS.md.
|
||
|
||
## Interpreting observer state
|
||
|
||
Every run records `SDR sensitivity: full | degraded` from
|
||
`hil/sdr_health.py`. Treat cross-run comparisons as invalid unless both
|
||
runs report the same state -- this dongle loses ~17 dB within minutes of
|
||
streaming (docs/HARDWARE.md).
|
||
|
||
## Meter-anchored TX verification
|
||
|
||
When serial telemetry and raw IQ were captured together:
|
||
|
||
```
|
||
./result-py/bin/python3 hil/meter_anchored.py \
|
||
--iq iq.raw --serial serial.log --json
|
||
```
|
||
|
||
The verifier first fits the DUT/IQ clocks only from strong meter receptions,
|
||
which prevents a repeat from borrowing its own original burst. It then
|
||
compares the median spectrum at exact TX times with nearby non-TX control
|
||
windows. `rf_state` is:
|
||
|
||
- `strong`: event-correlated FSK is present and median event energy is at
|
||
least 6 dB over the capture floor;
|
||
- `weak`: event-correlated FSK is present below that energy threshold;
|
||
- `not_observed`: no event-correlated FSK signature;
|
||
- `no_events`: the serial log contained no in-range TX events.
|
||
|
||
`tx_above_6db` is only an energy count; it is not an RF verdict by itself.
|
||
`fsk_excess_db` is the in-band (20-80 kHz offset) event/control excess after
|
||
subtracting wideband (250-700 kHz offset) event/control change. At least
|
||
five analyzable TX/control pairs are required.
|
||
|
||
For unattended beacon runs, `hil/beacon_soak.sh` discards raw IQ only when
|
||
the result is `strong`. Weak, absent, or unclassified segments retain IQ
|
||
for reanalysis.
|
||
|
||
## Long reliability soak (DUT only, no SDR)
|
||
|
||
```
|
||
nix develop --command ./hil/soak.sh 9000 # seconds
|
||
```
|
||
|
||
Measures firmware-side reliability over hours: repeat success rate, timing
|
||
stability, radio recoveries, config-integrity failures and MCU resets, and
|
||
reports progress against the milestone targets. Deliberately does not use
|
||
the SDR, which cannot stream for hours without faulting.
|
||
|
||
## Fault injection (verifying the recovery ladder)
|
||
|
||
```
|
||
./result-py/bin/python3 hil/dut_cmd.py <port> --listen 26 "inject standby"
|
||
./result-py/bin/python3 hil/dut_cmd.py <port> --listen 26 "inject sleep"
|
||
```
|
||
|
||
`standby` leaves the radio out of RX with its configuration intact;
|
||
`sleep` additionally destroys the configuration. The heartbeat probe must
|
||
detect both and the recovery ladder must restore RX. Verified behaviour:
|
||
|
||
```
|
||
inject standby -> heartbeat chip_mode=2, config_intact=true
|
||
-> radio_recovery level 1 -> rx_restored
|
||
inject sleep -> heartbeat chip_mode=0, config_intact=false
|
||
-> radio_recovery level 1 -> failed (correctly refuses)
|
||
-> radio_recovery level 2 -> rx_restored
|
||
```
|
||
|
||
## Build profiles
|
||
|
||
```
|
||
./tools/build.sh test # default; the configuration all HIL numbers use
|
||
./tools/build.sh debug # verbose diagnostics, assertions, per-TX dumps
|
||
./tools/flash.sh [test|debug]
|
||
```
|
||
|
||
Differences are behavioural, not cosmetic:
|
||
|
||
| | test | debug |
|
||
|---|---|---|
|
||
| `verbose` at boot | off | on |
|
||
| `tx_diag` after every TX | only on failure | always |
|
||
| raw frame hex in events | yes (HIL needs it) | yes |
|
||
| optimisation / log level | size, warn | debug, info |
|
||
| task watchdog | 15 s | 30 s |
|
||
|
||
`hil/run.sh` always builds and flashes the `test` profile, so reported
|
||
timing is never perturbed by debug serial traffic. Build directories are
|
||
separate (`build-test`, `build-debug`) so profiles cannot silently reuse a
|
||
stale sdkconfig.
|
||
|
||
## Verifying a standalone repeater (no serial connection)
|
||
|
||
When the DUT runs on external power with no USB host, firmware telemetry is
|
||
unavailable and RF is the only evidence. Record IQ once, then derive both
|
||
decodes and RF-level repeat pairs from the same samples:
|
||
|
||
```
|
||
timeout 80 rtl_sdr -f 868950000 -s 1600000 -g 42.1 -n 96000000 iq.raw
|
||
cat iq.raw | rtl_wmbus -f > decodes.log
|
||
./result-py/bin/python3 hil/verify_standalone.py --iq iq.raw --decodes decodes.log
|
||
```
|
||
|
||
Reports decoded C1/T1 repeats (a supported H=0 original followed by its H=1
|
||
counterpart, with every other cooked byte identical) and RF-level repeat
|
||
pairs (burst pairs of matching duration inside the delay window). Decoded
|
||
pairs use occurrence order because SDR timestamps are quantized; each
|
||
original is matched once. Same-H duplicate decodes are not repeat proof.
|
||
The RF-level check matters because our transmission usually overloads the
|
||
dongle and fails CRC even though the RF is correct.
|
||
|
||
Note: only one process can hold the RTL-SDR. If another tool (e.g. an
|
||
oms-tui / wmbusmeters pipeline) is running, `rtl_sdr` fails with
|
||
`usb_claim_interface error -6` and it must be stopped first.
|