From 3a0d342314bdee918c5f43d2f95e710ac6e86bc2 Mon Sep 17 00:00:00 2001 From: Lars Nolden Date: Mon, 21 Sep 2026 12:31:22 +0200 Subject: [PATCH] Document repeater validation and operating limits --- README.md | 107 ++++++++++++-- docs/FIELD_TEST_2026-09-01.md | 254 ++++++++++++++++++++-------------- docs/FINDINGS.md | 153 +++++++++++++------- docs/FUTURE_UI.md | 99 ++++++++++++- docs/OPEN_QUESTIONS.md | 26 ++-- docs/RELIABILITY.md | 62 +++++++-- docs/RF_CONFIG.md | 34 ++++- docs/RF_FRONTEND_TESTS.md | 32 +++-- docs/STANDARDS_GAPS.md | 80 +++++++++-- docs/TESTING.md | 244 ++++++++++++++++++++++++++++++-- docs/TIMING.md | 66 +++++---- 11 files changed, 890 insertions(+), 267 deletions(-) diff --git a/README.md b/README.md index 9087d56..1fc483f 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,14 @@ -# Wireless M-Bus C1 Repeater — Heltec WiFi LoRa 32 V4.3 (RF feasibility experiment) +# Wireless M-Bus C1/T1 Repeater — Heltec WiFi LoRa 32 V4.3 (RF feasibility experiment) Experimental firmware and hardware-in-the-loop test rig answering one question: **can the SX1262 + KCT8103L front-end on the Heltec V4.3 serve as the basis for a Wireless M-Bus gateway/repeater?** -Short answer: yes for RX and TX at the PHY level. It receives real C1 -meter telegrams, retransmits them bit-exact after a configurable delay, and -returns to RX reliably. Quantitative limits and caveats are below. +Short answer: yes for RX and TX at the PHY level, in both C1 and T1. +It receives C1/T1 meter telegrams and retransmits eligible OMS uplinks after +a configurable delay, setting the standard hop bit and repairing the CRC +without decrypting the payload. T-mode frames are decoded (3-of-6 in software) +and re-aired in their native mode. Quantitative limits and caveats are below. This is **not** a standards-compliant repeater. See [docs/STANDARDS_GAPS.md](docs/STANDARDS_GAPS.md). @@ -26,13 +28,28 @@ flake.nix ESP-IDF + rtl-sdr + rtl-wmbus + wmbusmeters + python env ## Quick start +### Flash firmware (Heltec only) + +Connect the Heltec over USB and run from the repository root: + +```bash +nix develop --command ./tools/flash.sh test +``` + +This builds the current local source and automatically discovers and flashes +the Heltec; no SDR is required. Use `debug` instead of `test` for verbose +diagnostics. Discovery refuses missing or ambiguous devices. If flashing +fails, the script rediscovers the USB port and retries once. + +### Hardware-in-the-loop test (Heltec + SDR) + ```bash nix build .#hil-python -o result-py # once nix develop --command ./hil/run.sh --duration 1200 ``` Exit code 0 = PASS. Artifacts land in `captures/-hil/` -(`report.txt`, `report.json`, `serial.log`, `rtlwmbus.log`, `iq_proof.txt`). +(`report.txt`, `report.json`, `serial.log`, `rtlwmbus.log`, `rf_proof.json`). Serial console (scriptable, JSON output): @@ -41,6 +58,59 @@ Serial console (scriptable, JSON output): # full command list: docs/TESTING.md ``` +## Phone dashboard (2026-09-09) + +Short-press PRG to the OLED **WLAN-DIAGNOSE** page, then hold ~1.5 s to +enable the password-protected access point. The OLED shows the SSID, +password and `http://192.168.4.1`. The next page displays a Wi-Fi join QR +code; scan it with a phone, then open the address. The 64-pixel-high OLED +requires a small QR: text credentials remain available if the camera +cannot focus. Choose **stay connected** if the phone warns about no internet. +No app, router, internet connection or captive portal is required. +The OLED and browser UI are in German. Tap a **?** beside a term for a +short beginner-friendly explanation; no mouse hover is required. +The latest signals come first; **Diagnose: Summen und Funkstatus** and +**Optional: eigene Namen für Zähler** expand only when needed. Help panels +close with **Schließen**, the same **?**, an outside tap or Escape. They +remain open while the history refreshes. + +- Wi-Fi is **off after every reboot**. Hold on either Wi-Fi page to disable + it. OLED blanking does not stop Wi-Fi. A random per-device password is + generated once and retained in NVS; it is not printed to serial. +- The dashboard shows 256 recent final reception/repeat outcomes, + CRC and decode rejections, deliberate policy skips, TX errors, RSSI, + C1/T1 mode, timing, radio health and rolling-hour duty usage. Synthetic + test/beacon transmissions are identified separately. +- **TX completed means the radio reported completion, not downstream + delivery.** Frames never detected by the Heltec cannot appear in history. + Dashboard counters cover the current boot and are unaffected by + `clear_stats`; history rolls off in RAM and is lost on reboot. +- Import an optional two-column `oms_id,label` CSV, or download a template + and export observed IDs. IDs remain eight-character strings, including + leading zeros. Labels stay only in the browser tab's memory; refresh + clears them. They are never uploaded or saved. +- Manufacturer names and device types are interpreted in offline JavaScript + from the link-layer header. Names use a bundled DLMS FLAG registry. + This is not meter-model/driver detection or encrypted-reading decoding. + Corrupt identities are untrusted; ambiguous IDs retain full raw identity + details. Labels never conceal the original ID. +- **Einstellungen am Heltec-Gerät ändern** exposes all ten existing settings: + forwarding, delay, duty budget, TX power, whitelist, beacon interval, + synthetic L-field, verbose serial output, FEM-CSD and preamble detection. + Expert controls are collapsed. Every field requires its own **Übernehmen**; + loading values never silently discards another unsent draft. +- Only delay and duty budget persist across reboots. A revision check rejects + stale writes; compare the current device value before explicitly resubmitting. + Accepted settings are copied by the radio task at a processing boundary. + TX-power/preamble changes can briefly interrupt reception for reinitialization. +- Disabling the duty guard or raising it above 1000 ppm requires confirmation, + not a new permission to exceed regulatory limits. **Heltec-WLAN ausschalten** + also requires confirmation; reconnecting afterward requires the OLED. + Anyone with access to the WPA2 AP can change settings. The same-origin/CSRF + checks prevent cross-site browser writes, not access by other joined clients. + +Details and verification limits: [docs/TESTING.md](docs/TESTING.md). + ## Results (measured, not inferred) Representative 1200 s run, `captures/2026-09-01T10-34-24-hil`: @@ -51,7 +121,7 @@ Representative 1200 s run, `captures/2026-09-01T10-34-24-hil`: | SDR-classified originals | 129 (44 meters) | | Heltec matched SDR originals | 47 (36.4 %) | | Heltec TX attempts / success | 72 / 72 (100 %) | -| TX events RF-confirmed in IQ | 5 / 5 in the IQ window (100 %) | +| TX RF confirmation | withdrawn for this run: measured with the borrow-prone aligner (see caveats) | | Firmware repeat delay (set 200 ms) | 200.001 ms min/median, 200.008 ms max | | Radio recoveries / MCU resets | 0 / 0 | @@ -64,8 +134,10 @@ Key hardware findings: and drops visible meters from 18 to 1. No overload penalty from maximum front-end gain at this site. [docs/RF_FRONTEND_TESTS.md](docs/RF_FRONTEND_TESTS.md) -- **Repeats are bit-exact**: 63/63 logged TX payloads equalled the received - raw frame plus C-mode bytes. +- **Historical bit-exact baseline**: 63/63 logged TX payloads equalled the + received raw frame plus C-mode bytes. Since 2026-09-09, received repeats + intentionally differ only in the OMS hop bit and affected CRC; see + [implementation notes](docs/STANDARDS_GAPS.md#oms-hop-marking-2026-09-09). - **RX deaf time is ~225.7 ms** per repeat at a 200 ms delay, including a 12.2 ms fixed-capture tail imposed by the SX1262 packet engine. [docs/TIMING.md](docs/TIMING.md) @@ -93,12 +165,17 @@ rtl-wmbus receives our repeat with 1-2 byte errors and fails CRC. Repeats are therefore verified two ways that are immune to this: - synthetic C1 bursts prove the framing is decodable as C1 (`verify_tx.py`); -- firmware TX timestamps are aligned to bursts in raw IQ to prove real - repeats radiate with the right timing and airtime - (`verify_repeats_iq.py`). +- `hil/meter_anchored.py` fits the serial<->IQ clock on strong meter + receptions only (a TX-fitted clock can lock at -repeat_delay and match + every repeat to its own original -- that bug produced withdrawn results), + then requires event-correlated FSK spectral excess against nearby non-TX + control windows before reporting `rf_state` strong/weak. -Closing the remaining gap needs physical antenna separation or an inline -attenuator, which is not scriptable from the host. +An earlier RF-alignment tool (`verify_repeats_iq.py`) had exactly that +clock-borrowing flaw and was removed; historical "N/N TX confirmed in IQ" +numbers from it are withdrawn (docs/FINDINGS.md). Closing the remaining +decode gap needs physical antenna separation or an inline attenuator, +which is not scriptable from the host. ## Documentation @@ -113,5 +190,5 @@ attenuator, which is not scriptable from the host. - [TESTING.md](docs/TESTING.md) — how to run things, known pitfalls - [DEVBOARD_REQUIREMENTS.md](docs/DEVBOARD_REQUIREMENTS.md) + [devboard_signals.csv](docs/devboard_signals.csv) — future PCB needs -- [FUTURE_UI.md](docs/FUTURE_UI.md) — design note: OLED + button for local - feedback (not implemented; display is untouched by this firmware) +- [FUTURE_UI.md](docs/FUTURE_UI.md) — OLED + button local UI: design, + constraints, and implementation notes (implemented in firmware/main/ui.c) diff --git a/docs/FIELD_TEST_2026-09-01.md b/docs/FIELD_TEST_2026-09-01.md index a8423c6..0a15e8d 100644 --- a/docs/FIELD_TEST_2026-09-01.md +++ b/docs/FIELD_TEST_2026-09-01.md @@ -5,18 +5,40 @@ the development bench to a real site. Goal: run the repeater standalone (no USB host) and verify it over RF alone. This log records what happened, because the sequence of wrong theories is itself a finding. +## Final correction (same date, after source and IQ audit) + +The late-evening conclusion that the board had an intermittent TX-chain +hardware fault is **withdrawn**. Two software/methodology faults explain +the evidence: + +1. The `pa_low` diagnostic used SetPaConfig + `(0x01,0x00,0x00,0x01)`. Its PA-duty and hpMax bytes came from the + SX1261 +10 dBm row, but `deviceSel=0x00` still selected SX1262. This + undocumented hybrid is not a low SX1262 row. Same-capture A/B testing + against valid SX1262 `(0x02,0x02,0x00,0x01)` measured about 32 dB more RF. +2. The beacon verdict called RF absent from total energy alone. Spectral + reanalysis of the canonical “dead” IQ capture finds weak but repeated + FSK at all 25 TX times: median +1.5 dB over floor versus +0.2 dB in + nearby controls, with +7.6 dB FSK-selective excess. + +After deleting the invalid PA row, a flashed-firmware smoke test produced +9/9 beacons at median +27.81 dB over floor (range +27.02..+27.90), nearby +controls +0.00 dB, and +40.73 dB FSK-selective excess at configured +-9 dBm. This report retains the earlier narrative as a chronology of +misdiagnosis; the final conclusions are in the corrected section below. + ## Timeline of evidence | Setup | LED (TxDone) | RF at SDR | Verdict at the time | |---|---|---|---| -| Laptop USB, board on table (~16:07) | double-blinks | **7/7 TX clock-aligned, median +37 dB over floor** | baseline: transmits | +| Laptop USB, board on table (~16:07) | double-blinks | reported 7/7 at +37 dB; later invalidated | baseline claim came from the repeat-borrowing clock-fit bug | | Power bank (USB-C, then USB-A) | went dark after seconds | none | bank auto-off at ~60 mA idle load | | UGREEN 100 W GaN wall charger, C-to-C | double-blinks "sometimes" | none in 4 captures (~8 min) | *wrongly* blamed supply sag | | LiPo pack attached | blinks, 2-4 double-pulses/90 s | none | supply theory started cracking | | LiPo + dangling USB cable (no host) | ~9 double-pulses/90 s | none | RX rate recovered; TX still invisible | | Mid-capture unplug (laptop -> LiPo, position frozen) | blinks throughout | **none in either phase** | laptop connection alone does NOT explain it: power theories all dead | | Wall power, board hanging free, no LiPo | blinks | none at prior SDR-antenna position | placement of the DUT alone is not it either | -| **SDR antenna moved to ~20 cm from the hanging board** | blinks | **same-duration burst pairs at 50-54 ms: 10-10, 17-17, 16-16 ms** | repeater IS repeating standalone; radiated/received level is simply far lower than the laptop-connected reference | +| **SDR antenna moved to ~20 cm from the hanging board** | blinks | **same-duration burst pairs at 50-54 ms: 10-10, 17-17, 16-16 ms** | standalone RF exists; comparison with the invalid laptop reference is unsupported | ## What was actually established @@ -24,23 +46,18 @@ because the sequence of wrong theories is itself a finding. RX->50 ms->TX cycle runs (LED) and its repeats are observable at RF (same-duration pairs at the configured delay) once the observer antenna is close enough. -2. **The laptop-connected "+37 dB at 50 cm" reference was not a pure - radiated-field measurement.** With the board on the host's USB, the - cable shield and shared ground with the SDR dongle add counterpoise - and possibly conducted common-mode coupling. Standalone, at a similar - claimed distance, the same transmitter was invisible; at ~20 cm it shows - up at +14..+24 dB. The gap between those numbers is the combined effect - of geometry, orientation, antenna environment and the lost cable/ground - contribution -- not a firmware or supply fault. -3. **Every power-source theory failed when tested properly.** - - Power bank: real failure, but trivial (auto-off). The LED found it - in one glance. - - Wall-adapter "sag during PA burst": WITHDRAWN. The adapter is a - 100 W GaN unit, and the frozen-position unplug test showed the - laptop connection failing at the same spot too. The earlier - conviction (committed to FINDINGS.md at the time) was wrong. - - LiPo (stiffest possible rail): same invisibility -> supply exonerated - entirely. +2. **The laptop-connected “+37 dB at 50 cm” reference was invalid.** + The old clock fit could align each expected repeat to its original + meter burst. It cannot establish a USB-cable counterpoise or conducted + coupling effect. +3. **No tested power source correlated with the weak RF.** + - Power bank: real failure, but trivial (low-load auto-off). The LED + identified it directly. + - Wall adapter and LiPo both showed chip-level TX with weak/invisible RF. + - The frozen-position unplug run did not isolate a supply effect. + These observations reject the original wall-adapter-sag theory, but + “supply exonerated entirely” was stronger than the available evidence. + The later PA-row bug is sufficient to explain the large RF deficit. 4. **TxDone still proves nothing about radiated power.** The task spec's warning (§36: "TX successful is not evidence") extends beyond FEM configuration to antenna geometry: the SX1262 completes its packet into @@ -62,110 +79,141 @@ because the sequence of wrong theories is itself a finding. - **A conducted RF test port would have ended this in minutes.** Half a day of over-the-air inference replaced one attenuated SMA measurement. Already in devboard_signals.csv; priority upgraded. -- **Antenna system design is not optional polish.** The stock whip against - a tiny PCB ground radiates so poorly, and so directionally, that - deployment placement dominates link budget. The gateway needs a real - antenna with a defined counterpoise, and the battery pack must be kept - mechanically away from the radiator. -- **On-device persistent telemetry (NVS counters, VBAT) is still the - missing piece.** The LED collapsed "is it powered/alive/repeating" from - four SDR captures to one glance, but every RF question still required - the laptop. Retained counters would have separated "not receiving" from - "not transmitting" instantly. +- **The antenna conclusion was premature.** Geometry, orientation and + counterpoise always affect over-the-air measurements, but the invalid + SX1261-derived PA configuration was not controlled and accounts + for about 32 dB by itself. A proper antenna A/B remains useful; this + session did not quantify stock-whip efficiency. +- **Persistent counters would improve, but not prove, RF diagnosis.** NVS + RX/TxDone counters and VBAT would separate power, receive, and chip-TX + failures without a host. They still could not distinguish weak RF from a + healthy radiated packet; that requires a conducted detector or external + receiver. - **LED patterns earned their place** and should carry to the gateway (with the FUTURE_UI.md caveat: an LED can only report chip state, never radiated RF). -## Still open after this session +## Questions left by the afternoon run (status after audit) -- Clock-aligned close-antenna run (device on laptop serial + SDR antenna - at 20 cm) to attribute bursts per TX event and put a number on the - standalone radiated level. Protocol ready; not yet executed. +- RESOLVED for the attached-DUT method: meter-anchored event windows plus + spectral controls now attribute RF to each TX. Absolute standalone field + strength is still uncalibrated. - Quantify laptop-cable contribution: repeat the alignment run twice at identical geometry, cable in vs out (LiPo), antenna close. - Proper antenna A/B (e.g. quarter-wave with radials vs stock whip) -- directly relevant to the gateway RF architecture. -## Location 3 (evening session) -- the beacon, and the real fault +## Location 3 (evening session) — corrected beacon postmortem -Coordinating physical actions with an async agent kept failing (missed -timestamps, tuner deaths mid-capture), so we changed instruments: a -firmware **beacon mode** (`set beacon_period_ms N`) transmits a synthetic -OMS frame (manufacturer SLN, ID 00000001, access_nr = sequence) every N ms. -No meter traffic, no serial coordination, dense uniform sampling. +Beacon mode (`set beacon_period_ms N`) was the right instrument: it emits +known frames at known times without relying on meter traffic. The first +verdict applied the wrong detector to that data. -The beacon caught what two days of repeat-chasing could not: +### What the retained IQ establishes -- **TX radiated power is intermittent at minute scale.** One 90 s window: - 0/25 beacons produced ANY RF (max +0.5 dB over floor) while 27 ambient - meter frames in the same IQ showed median +26.8 dB (observer healthy) and - the chip reported TX_DONE, mode 6, device_errors 0 for every beacon. - Thirty minutes later, same boot, same config: repeats, test_tx and - beacons all radiated at a constant ~+10 dB. -- RX stayed healthy through dead TX windows. Chip-side diagnostics are - identical in dead and live states. The fault therefore sits between the - SX1262 PA output and the antenna: KCT8103L PA path (CPS/DIO2, CTX, PA - supply) or a mechanical/thermal TX-chain fault. Firmware cannot observe - it -- exactly the RADIO_VDD_SENSE / conducted-test-port gap in - docs/devboard_signals.csv. -- **Methodology bug found and fixed:** verify_repeats_iq's single-burst - clock fit can lock at -repeat_delay and "confirm" every repeat with its - own original's burst. The "7/7 at +37 dB" laptop confirmations were this - artifact. Trustworthy methods: structural same-duration pairs at the - configured delay, content decodes (duplicate payloads), and - hil/meter_anchored.py (clock fitted on strong meter receptions; ambient - meter levels double as per-capture observer health). -- Positive control: with a healthy observer, 30 structural pairs at - 137 ms and 35 duplicate decoded payloads in 90 s -- the repeater - function itself is solid whenever the TX chain is in its live state. +- The historical capture + `captures/2026-09-01T17-35-12-beacon-diag` has a healthy observer: + 27/32 strong meter anchors, 0.7 ms median clock error, meter median + +26.8 dB. +- Beacon windows are weak in total energy (median +1.5 dB; none reaches + +6 dB), but their median spectrum contains the expected approximately + +/-45 kHz FSK energy. Relative to nearby non-TX controls, FSK-selective + excess is +7.6 dB across 25 windows. +- A known-live repeat capture + (`2026-09-01T17-41-21-repeat-pair-test`) classifies strong: 29/29 above + +6 dB, median +10.9 dB, FSK-selective excess +18.8 dB. +- Therefore “0/25 produced any RF” and “genuinely zero RF” were false. + TxDone did not prove RF, but the retained spectrum does. -## Location 3, late evening -- soak verdict and physical isolation attempt +The soak's exact historical PA-row state cannot be reconstructed: serial +telemetry logged SetTxParams power but not the `pa_low` selection, and the +old soak harness deleted IQ after reducing each segment to scalar energy. +Its claimed +8 -> +6 -> zero thermal decay is consequently not auditable. +A board-level thermal/mechanical TX fault was never demonstrated. -Antenna swap + IPEX reseat performed (TX disabled during swap). Immediate -post-swap check: 74/74 TX events radiating. Beacon soak started to look -for recurrence (2-min segments, meter-anchored verdicts, observer health -gating, dongle auto-recovery). +### Source-level root cause -Soak result: **the fault recurred post-swap.** +The firmware's alleged low-power SX1262 PA row was copied from the wrong +line of Semtech table 13-21: -- Segments 1-8 (~16 min usable): alive, TX median +7.0..+8.8 dB. -- Segment 9: marginal (median +5.9) -- levels visibly sagging. -- Segment 13: **dead** -- 106 TX attempts, median +0.5 dB, observer - independently verified healthy (56/60 meter anchors at 2.2 ms clock - error, meters at +21.8 dB). Airtight. -- The decay pattern (+8 -> +7 -> +6 -> dead over ~10 min of sustained - ~50 TX/min) suggests but does not prove a thermal mechanism. +``` +old diagnostic row: 01 00 00 01 undocumented SX1261-values/SX1262-select hybrid +fixed SX1262 row: 02 02 00 01 documented +14 dBm-optimal PA config +SetTxParams: -9 02 configured level -9 dBm, 40 us ramp +``` -**Conclusion: the old antenna and pigtail are acquitted. The intermittent -TX-kill is on the board itself**, in the chain between SX1262 PA output -and the antenna connector (KCT8103L FEM prime suspect), invisible to -every chip-level diagnostic. +Same-stream A/B measurement found about 32 dB between the two rows. The +invalid path has been removed rather than retained as a diagnostic. +Configured SX1262 power is now limited to -9..+2 dBm; the default remains +-9 dBm because the external KCT8103L adds gain and absolute e.r.p. has not +been calibrated. -Live provocation session (35 s verdict cadence): -- Baseline: marginal-alive (+5.2), died to +2.0 within ~2 min (pressure - state during death window not recorded). -- ~90 s of breath-cooling the FEM area: no recovery (+3). Weak cooling, - so thermal is not acquitted -- merely unconfirmed. -- Pressure test: not completed; session ended. +### Corrective verification -Instrument attrition tonight: the RTL-SDR dongle required one physical -replug, dropped off the USB bus once more, produced four short/empty -captures and two deaf windows. Roughly a third of tonight's wall time -went to working around the observer rather than observing the DUT. +Flashed fixed firmware, 1.6 Msps SDR, fixed 0.9 dB tuner gain: + +``` +scheduled beacons 9/9 TxDone +RF envelope over floor +27.81 dB median (+27.02..+27.90) +nearby non-TX controls +0.00 dB median +FSK-selective excess +40.73 dB +SDR clipped samples 0.000 % +computed airtime accounting 24,480 us = 9 x 2,720 us +``` + +The detector also scored nine fake event times in an off interval at +-0.16 dB FSK excess, versus +40.73 dB for the real aligned events. + +`hil/meter_anchored.py` now reports `rf_state=strong|weak|not_observed` +and requires event-correlated FSK spectral excess before claiming RF. +`hil/beacon_soak.sh` retains raw IQ for every weak, absent or unclassified +segment. Strong segments alone may be reduced to a JSON verdict. + +The unrelated duty-cycle bug found during the audit is also fixed: the +384-event ring silently forgot airtime at high rates. A 61-minute bucket +ledger now retains every event overlapping the rolling hour, and duty +usage is computed from packet bits rather than SPI/polling latency. ## Where this leaves the project -1. Repeater firmware: fully functional and standards-shaped -- structural - pairs, duplicate decodes, duty accounting (modulo ring saturation, - see OPEN_QUESTIONS), recovery ladder, LED patterns, beacon mode. -2. This particular board: not deployable unattended until the TX-chain - fault is fixed or understood. Next diagnostic steps in order of value: - a) proper cooling/heating provocation (freezer spray / hot-air at low - temp) with the live monitor; b) pressure/flex mapping; c) conducted - measurement through an attenuator to split FEM vs antenna-port; all - need bench equipment or at least controlled temperature, not breath. -3. The observer (NESDR SMArt v5) is end-of-life for serious use: replace - before the next HIL session. -4. Every hard-won methodology upgrade is now in the repo: beacon mode in - firmware, hil/meter_anchored.py (borrow-immune verification + observer - health), hil/beacon_soak.sh (unattended state mapping). +1. The connected board demonstrably transmits correct FSK after the PA-row + fix. No intermittent board-level TX fault is currently established. +2. TxDone and the LED remain chip-level evidence only. RF conclusions need + the spectral detector or an independent receiver/decode. +3. Absolute radiated power and regulatory e.r.p. remain uncalibrated. Keep + the configured SX1262 level at -9 dBm until a conducted measurement or + calibrated field-strength setup exists. +4. The NESDR observer remains unreliable for long unattended work; its + per-capture health gate and anomaly-IQ retention are mandatory. + +## Late evening — standalone battery test with fixed firmware: PASS + +First deployment after the PA-row fix and the OLED/button UI. DUT on +battery, no USB host; all verification RF-only plus the on-device screen. +Capture: `captures/2026-09-01T22-22-32-standalone-battery` (360 s IQ at +42.1 dB gain, verdicts archived alongside). + +- **Repeat function confirmed standalone** by two independent methods on + one stream: 3 decoded duplicate payloads, and 3 structural RF pairs with + matching durations (10-10, 7-7, 10-10 ms) at 53-54 ms gaps against the + 50 ms default delay. The +3-4 ms is the radio-preparation offset between + TX-routine entry and RF onset (standby + packet params + buffer write), + consistent with `tx_operation_us` measurements. +- **Observer health control in-capture**: 46 C1 decodes from 38 unique + meters; floor +2.9 dB. +- **Counter funnel closed from the OLED alone** (no laptop): RX 292 = + VALID 100 + CRCFAIL 63 + NOT-C1 130 + LEN 0 (one frame arrived between + page reads). TX 109/109 at 100%. The 34% valid ratio matches the site + benchmark; CRC failures are the characterized long-encrypted-Techem + sensitivity edge, not a fault. +- **UI-driven beacon toggle verified end to end** earlier in the session: + an independent wmbusmeters receiver decoded beacon ID 00000001 with + valid CRC every ~3.3-3.5 s (3000 ms + scheduling drift) after a button + press, on battery. Beacon was off during this capture (0 beacon decodes, + no comb -- matches the TX counter arithmetic). +- The 44 railed ~+40 dB short bursts in the capture are our repeats PLUS + the fingerprinted neighbour meter's singles; only delay-and-duration + matched pairs were attributed to the repeater. +- Duty note for future runs: the 3000 ms button beacon consumes 907 of + 1000 ppm; running it for long alongside repeats drives `DUTYBLK` up by + design. Toggle it off when measuring repeat behaviour. diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index b7c7b0b..4504689 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -19,14 +19,15 @@ - TX proof of life: SDR decoded our synthetic C1 format-A frame as valid (crc_ok=1, mfr 0x4D8E "SLN", ID 00000001) at configured -9 dBm, and the wM-Bus CRC blocks computed by firmware matched rtl_wmbus's checks. -- First real repeat observed: 100-byte Techem frame received, repeated - after 50.002 ms (configured 50 ms). -- Nooelec NESDR SMArt v5 R820T PLL wedges ("PLL not locked"), producing - gain-independent white noise on all frequencies until USB reset. - `usbreset 0bda:2838` reliably restores operation. Trigger correlated - with `rtl_test -t` direct-sampling probe. -- rtl-sdr-blog and osmocom rtl_sdr builds behave identically w.r.t. the - wedge; not a driver-fork issue. +- First real repeat observed: a 100-byte Techem frame was received and the + firmware entered its TX routine 50.002 ms after the reconstructed frame + end (configured 50 ms). This timestamp precedes radio preparation; it is + not the exact on-air start time. +- The Nooelec NESDR SMArt v5 can enter gain-independent, railed-noise + states until USB reset or physical replug. The tune-time + `[R82XX] PLL not locked!` message is benign and `rtl_test -t` was not the + cause. `hil/sdr_health.py` diagnoses the actual failure from gain response + and spectral structure. - Real C1 traffic exists at this site: 8 meters in 10 min baseline (docs/SDR_BASELINE.md); one strong plaintext meter (61471660), several marginal encrypted Techem meters. @@ -41,6 +42,18 @@ called) -> no notifications despite correct radio-side IRQ config. 3. `timeout bash -c "rtl_sdr | rtl_wmbus"` orphans the pipeline children on timeout (device stays claimed). Apply timeout to rtl_sdr directly. +4. The `pa_low` diagnostic selected SetPaConfig + `(0x01,0x00,0x00,0x01)`, documented in code as a low SX1262 row. Its + first two parameters came from the SX1261 +10 dBm row, but + `deviceSel=0x00` selected SX1262, creating an undocumented hybrid. + Same-capture A/B testing showed about 32 dB less RF than the valid + SX1262 row `(0x02,0x02,0x00,0x01)`. Fix: remove the invalid row + and diagnostic toggle; always use a valid SX1262 row. +5. The rolling-hour duty ledger retained only 384 individual events. At + beacon rates it wrapped after about 2.7 s of airtime and silently + under-counted the hour. Fix: 61 keyed minute buckets retain all airtime + in the overlapping hour; host tests cover high event counts, expiry, + boundary overlap, and invalid timestamps. ## LIKELY @@ -61,20 +74,23 @@ docs/RF_FRONTEND_TESTS.md. - Repeat payloads are byte-exact: 63/63 logged `tx_payload` values equalled mode bytes + the received raw frame. No corruption in the RX->TX path. -- Real repeats are emitted as correct RF: aligning firmware TX timestamps - to bursts in the same stream's IQ confirmed 48 of 57 TX events (84 %), - with alignment error median 1.3 ms (max 3.6 ms) and burst duration within - 0.5 ms (median) of the computed airtime. Firmware repeat delay measured - 200.0 ms min/median/max against a 200 ms setting. +- Real repeats are emitted as correct RF: aligning firmware TX-operation + timestamps to bursts in the same stream's IQ confirmed 48 of 57 TX + events (84 %), with alignment error median 1.3 ms (max 3.6 ms) and + burst duration within 0.5 ms (median) of computed airtime. Firmware + entered the TX routine at 200.0 ms against a 200 ms setting; radio + preparation shifts actual RF onset later. - Firmware TX reliability: 58/58 TX success over 600 s and 72/72 over 1200 s; tx_timeout 0, tx_failures 0, radio recoveries 0, MCU resets 0. - The KCT8103L RX path is essential: holding CSD low in RX costs ~41 dB of chip-referred level and drops the visible meter population from 18 to 1 (600 s per arm, docs/RF_FRONTEND_TESTS.md). No evidence of excessive front-end gain hurting at this site. -- Repeat-delay accuracy: +1 to +8 us against a 200 ms setting across 72 - repeats. RX deaf time per repeat ~225.7 ms median, of which 12.2 ms is - the fixed-255-byte RX tail and 13.4 ms the TX airtime (docs/TIMING.md). +- Repeat-operation scheduling: +1 to +8 us against a 200 ms setting across + 72 repeats. This is accuracy at entry to the TX routine, not RF onset. + The historical 13.4 ms value called “TX airtime” was operation latency + through observed TxDone; computed channel occupancy was 8.8 ms for the + representative 102-byte radio payload (docs/TIMING.md). - The SX1262 GFSK engine cannot express wM-Bus length semantics, so RX uses a fixed 255-byte capture; RxDone therefore fires 9.4-17.1 ms after the real end of frame and the firmware reconstructs the true timestamp. @@ -147,36 +163,38 @@ - Long-run stability at >=100 and >=500 verified repeat events. - Early-stop RX (poll GetRxBufferStatus) to cut the ~20 ms deaf tail. -## Location 3 follow-up (2026-09-01 evening): intermittent TX RF death +## Location 3 follow-up (2026-09-01 evening): PA configuration postmortem -- CONFIRMED: the DUT's radiated TX output is intermittent on minute scales - -- genuinely zero (not weak) for >=10 min windows, then a stable ~+10 dB - at the SDR. Verified with beacon mode against in-capture ambient-meter - controls (observer healthy: meters +26.8 dB median while 25/25 beacons - produced <= +0.5 dB). RX unaffected; chip reports TX_DONE/mode 6/no - errors in both states => fault is post-chip (KCT8103L PA path or TX - chain), invisible to firmware. Hardware follow-up required; strengthens - the RADIO_VDD_SENSE and conducted-RF-test-port devboard requirements. -- UPDATE (same evening): antenna + IPEX pigtail EXCLUDED -- after a full - antenna swap and pigtail reseat the fault recurred (soak segment 13: - 106 TX attempts at median +0.5 dB with observer verified healthy). - The fault is ON-BOARD. Decay signature before death (+8 -> +6 -> dead - over ~10 min under ~50 TX/min load) points at thermal in the FEM PA - path; a 90 s breath-cooling attempt did not revive it (inconclusive -- - needs real thermal provocation or conducted measurement). -- WITHDRAWN: every single-burst "TX aligned to IQ" confirmation (e.g. - "7/7 at +37 dB", "33/41 at +19.3 dB"). verify_repeats_iq's clock fit on - TX events can lock at -repeat_delay and match each repeat to its own - original's burst. Borrow-immune evidence only: structural same-duration - pairs at the configured delay, duplicate content decodes, or - hil/meter_anchored.py (meter-anchored clock, exact TX windows). -- CONFIRMED: observer health must be judged per capture from ambient meter - levels inside the same IQ (meter median >= +15 dB over floor). The - Nooelec went deaf mid-session again; runs without this control are void. -- CORRECTION to yesterday's "placement dominates": placement governs - whether PAIRS are detectable (originals must be SDR-visible), but the - standalone invisibility episodes are better explained by the intermittent - TX death now that it has been directly observed under controls. +- **WITHDRAWN:** the conclusion that this board had an intermittent + post-SX1262 hardware TX death. The canonical “0/25 beacons produced any + RF” capture was judged only by total energy in 1 ms windows. Reanalysis + of the retained IQ finds a repeatable event-correlated FSK signature: + TX median +1.5 dB over floor, nearby controls +0.2 dB, and FSK-selective + excess +7.6 dB across 25/25 windows. It was weak RF, not zero RF. +- Root cause in firmware: the `pa_low` diagnostic combined the SX1261 + +10 dBm PA-duty/hpMax values with SX1262 `deviceSel=0x00`: + `(0x01,0x00,0x00,0x01)`. In a same-stream A/B test, documented SX1262 + `(0x02,0x02,0x00,0x01)` raised the event envelope by about 32 dB. The + invalid toggle and row are now deleted. Configured power + is restricted to -9..+2 dBm; the external KCT8103L still follows. +- The old logs recorded configured SetTxParams power but not which PA row + the diagnostic had selected. Exact row state cannot be reconstructed for + every old soak segment, and the old soak deleted each IQ file after its + scalar verdict. Therefore its claimed +8 -> +6 -> zero thermal decay is + not auditable and is not evidence of a board fault. +- Fixed-firmware RF smoke test at configured -9 dBm: 9/9 scheduled beacons, + median +27.81 dB over floor (range +27.02..+27.90), nearby controls + +0.00 dB, FSK-selective excess +40.73 dB, and no ADC clipping at 0.9 dB + SDR gain. Firmware also reported 9/9 TxDone and exactly 24,480 us of + computed airtime (9 x 2,720 us). +- `hil/meter_anchored.py` now requires event-correlated FSK spectral excess + before calling RF present and reports `strong`, `weak`, or + `not_observed`. The historical weak capture classifies `weak`; a known + live repeat capture classifies `strong`. `hil/beacon_soak.sh` deletes IQ + only for `strong` segments and retains every anomaly. +- Observer health must still be judged per capture from ambient meter + levels. TxDone remains only chip-level evidence. Neither fact supports a + board-fault conclusion without the RF signature and retained raw IQ. ## Field test 2026-09-01 (onsite standalone deployment) @@ -192,12 +210,14 @@ Full narrative with the evidence table: docs/FIELD_TEST_2026-09-01.md. ~20 cm of the free-hanging, wall-powered board, same-duration burst pairs appeared at the 50 ms repeat delay (10-10, 17-17, 16-16 ms; received +14..+24 dB over floor). -- CONFIRMED: the laptop-connected "+37 dB at ~50 cm" reference overstates - standalone radiated field -- the USB cable/host ground contributes - counterpoise (and possibly conducted coupling into the shared-ground - dongle). Radiated performance is dominated by antenna geometry: - orientation, surroundings, and the LiPo pack's position relative to the - antenna. Placement, not power, ruled the day. +- **WITHDRAWN:** the claimed laptop-connected “+37 dB at ~50 cm” + reference and the inferred USB-cable/counterpoise delta. The +37 dB + number came from the repeat-borrowing clock-fit bug, so it cannot support + a cable contribution estimate. +- **NOT ESTABLISHED:** that the stock whip itself explains the weak + standalone results. Antenna geometry unquestionably changes an + over-the-air measurement, but the invalid SX1261-derived PA configuration is sufficient to + explain a roughly 32 dB deficit and was not controlled in those runs. - CONFIRMED: TxDone (and therefore the LED double-blink) proves chip-level packet completion only. It is indistinguishable between a well-matched antenna and a shadowed/mismatched one -- §36's lesson extended to @@ -206,3 +226,34 @@ Full narrative with the evidence table: docs/FIELD_TEST_2026-09-01.md. singletons", 23-24 ms trains at ~30 ms spacing) are a neighbouring meter railing the dongle: rate invariant across firmware versions, reboots and power sources. Excluded from pair detection by fingerprint. + +## T-mode + long-frame support (2026-09-02) + +- CONFIRMED: T1 transmit works end to end. Synthetic T1 frames (3-of-6 + coded in software, 50 kHz TX deviation via lazy SetModulationParams + swap, same 0x543D sync) were decoded by rtl_wmbus as valid `T1;...` + telegrams, including a long L=140 frame (12 decode lines over 6 + airings, `captures/2026-09-02T-t1-verify`). +- CONFIRMED: the T1 receive path decodes standard on-air chips. With no + ambient T1 available (below), the proof is: our own T1 airings -- + independently accepted by rtl_wmbus -- were FSK-demodulated from raw IQ + by a separate host tool, and the post-sync byte stream was fed through + the unmodified firmware decode code (`wmbus_3of6_first_byte` -> + `wmbus_3of6_decode` -> `wmbus_check_crc`): 5/5 CRC-OK with the exact + transmitted content. A C1 burst pushed through the same path is + correctly rejected at the first coded byte. +- CONFIRMED (site): there is NO ambient T-mode traffic at this site -- + every ambient rtl_wmbus decode is C1. **Correction to earlier belief:** + the large `rx_not_c1` bucket (44 % of sync matches overnight) is NOT + T1 meters; it is C1 frames whose two NRZ mode bytes were corrupted by + bit errors, plus false syncs from the deliberately short 8-bit preamble + detector. The Techem meters here are C-mode. +- Long frames: format B >128-byte CRC chunking and format A up to L=200 + are host-tested (sanitizers) and the L=200 format A frame was decoded + CRC-OK over the air by rtl_wmbus. +- T1 length cap: the fixed 255-byte capture holds ceil(total * 1.5) coded + bytes, so T1 receive/repeat covers format A total <= 170 (L <= ~150). + Long C1 covers total <= 253. +- 3-of-6 tables verified against rtl-wmbus t1_c1_packet_decoder.h; codes + are constant weight 3, so every single-chip corruption is detected + (host test flips all 384 chip positions of a 32-byte payload). diff --git a/docs/FUTURE_UI.md b/docs/FUTURE_UI.md index b199e43..6a4ac63 100644 --- a/docs/FUTURE_UI.md +++ b/docs/FUTURE_UI.md @@ -1,10 +1,9 @@ -# Future feature note: OLED + button for immediate local feedback +# OLED + button for immediate local feedback -Status: **design note only. Not implemented.** The task description -explicitly excluded a "production OLED UI" from this experiment, and the -display is untouched by the current firmware. This note records what a -local UI should show and the constraints it must respect, while the -reasoning is still fresh. +Status: **implemented** (firmware/main/ui.c, 2026-09-01). This note holds +the design rationale and constraints; deviations of the implementation +from the original sketch are listed at the end. Pins were verified against +RTNode-HeltecV4 and are now exercised by our firmware (`ui_init ok:true`). ## Why this is worth building @@ -182,3 +181,91 @@ does **not** share a rail or bus with anything RF-critical, plus at least two buttons so navigation and confirmation are separable — a single button forces gesture overloading, which is how dangerous actions end up behind timing tricks. + +## Implementation notes (what was actually built) + +- `firmware/main/ui.c`: esp_lcd SSD1306 panel over the new i2c_master + driver, 5x7 font (`font5x7.h`, host-verified by + `firmware/test/test_font_render.c`), UI task at priority 3 pinned to + core 0 (radio task is priority 10 on core 1). Rendering is a 1024-byte + full-frame draw every 500 ms while awake. +- Data flow: radio task publishes `g_ui` (last RX / last TX / heartbeat + probe results) in `repeater.c`; the UI additionally reads `g_stats`, + `g_cfg`, and `repeater_duty_used_us()`. The UI never touches the SX1262. +- Button: short press = next page / wake; hold ~1.5 s = page action. + Actions emit `{"event":"ui_action","cmd":...,"value":...}` so serial + logs remain authoritative. A press while the display is blanked only + wakes it and fires nothing. +- Page actions: Status = toggle `repeat_enabled`; Health = clear stats; + Config = toggle beacon at a fixed 3000 ms (907 ppm at L=19 — inside the + default 1000 ppm duty budget, so a button-started beacon cannot violate + the guard). Last-telegram page has no action. +- Display blanks after 3 min idle (panel off command; Vext is never + toggled). Init failure is non-fatal: `{"event":"ui_init","ok":false}` + and the task exits. +- Deviations from the sketch: no RESETS counter on the health page (needs + NVS boot counting, still unimplemented); the duty bar shows used/budget + from the minute-bucket ledger; the config page omits `PA LOW` (the + invalid PA row was removed from the firmware entirely) and `fem_rx_csd` + (console-only diagnostic). + +## Wi-Fi dashboard and QR addition (2026-09-09) + +Implemented in `wifi_debug.c`, `debug_history.c`, `wifi_qr.c` and +`firmware/main/web/`. Two pages follow Config: Wi-Fi credentials and Wi-Fi +join QR. A long press on either page toggles the AP; no boot-time gesture +is used. The existing four page actions remain unchanged. + +The radio publishes a separate 256-entry immutable final-outcome history, +including silent/verbose-only rejection and skip paths, with synchronized +per-row copies. Web clients do not read the deliberately racy OLED snapshot +or query the SX1262. Wi-Fi, TCP/IP, HTTP and control processing use core 0; +the radio remains on core 1. No network/JSON allocation was added to the +radio history producer. Final row IDs correlate reception and transmission +directly, not by frame-hash matching. + +The AP is off at boot and stays on until explicitly disabled or rebooted; +display blanking is independent. The password is randomly generated once, +persisted in NVS and shown only on the physical display. The QR encodes +the same credentials and is generated once per boot when its page is used. +Its encoder is the unmodified Nayuki C library v1.8.0 under MIT, with +copyright/license notices retained in `firmware/main/vendor/qrcodegen.*`. +The 64-pixel display height requires small integer-scaled modules; do not +crop the quiet zone or promise camera readability without a physical scan. + +The dashboard is entirely offline. CSV labels, templates, +exports, manufacturer registry lookups and device-type interpretation run +in browser JavaScript. Labels never reach the device or persistent browser +storage. See README and `docs/TESTING.md` for usage, exact success semantics +and the distinction between localhost, USB, phone and RF verification. + +Both surfaces now use German UI text (ASCII transliterations on the OLED). +The phone dashboard puts current events before collapsed diagnosis and +optional CSV naming. A shared, nonmodal **?** help panel explains wM-Bus +terms without hover, stays outside the refreshed rows, and supports outside +click, explicit close and Escape with focus restoration. Its scrollable +body and visible close control are bounded to the viewport. Detail-cell +spans follow the 560 px radio-column breakpoint, including resize, so +expanded rows do not introduce a phantom column on phones. + +## Writable settings addition (2026-09-09) + +`settings.c` is the common coordinator for all ten existing USB settings and +OLED toggles. It serializes writers and two-key NVS persistence, publishes +accepted values under a short spinlock, and lets the radio owner copy a whole +revision at a processing boundary. A pending-only IRQ check drains completed +captures and defers known in-flight captures; required radio reinitialization +can still interrupt reception. The applied revision acknowledges copying, +not RF success. OLED rendering reads the applied snapshot, while settings +responses and serial status report accepted values plus both revision numbers. + +`settings_http.c` adds bounded JSON writes guarded by AP-host/origin checks, +a random per-AP CSRF token and boot/revision comparison. WPA2 access remains +the authorization boundary. The mobile panel has separate drafts/submit +actions, dirty-value preservation, conflict review, persistence labels and +confirmation for dangerous duty limits or AP shutdown. Expert controls remain +collapsed. Wi-Fi shutdown is queued to its control task, not performed inside +the server being stopped. Only delay and duty budget persist, as before. + +The build embeds deterministic gzip assets and serves them directly, avoiding +runtime compression and keeping the existing app/NVS partition layout. diff --git a/docs/OPEN_QUESTIONS.md b/docs/OPEN_QUESTIONS.md index 7846944..0c47a25 100644 --- a/docs/OPEN_QUESTIONS.md +++ b/docs/OPEN_QUESTIONS.md @@ -9,9 +9,10 @@ interferers? -> It helps decisively. CSD low costs ~41 dB and reduces visible meters from 18 to 1; no overload degradation observed (docs/RF_FRONTEND_TESTS.md). -- CLOSED: RX dead time during retransmit -> ~225.7 ms median at a 200 ms - repeat delay, of which 12.2 ms is the fixed-255 RX tail and 13.4 ms the - TX airtime (docs/TIMING.md). Early-stop RX would cut the tail. +- CLOSED: historical RX-unavailable interval during retransmit was + ~225.7 ms median at a 200 ms delay: 12.2 ms fixed-255 RX tail + 200 ms + scheduling delay + 13.4 ms TX operation latency. The representative + packet's computed RF occupancy was 8.8 ms (docs/TIMING.md). - Can the SX1262 re-arm RX fast enough to catch a meter's immediate follow-up telegram after our TX? - Exact production repeater framing/hop requirements (EN 13757-5). @@ -40,12 +41,15 @@ EN 13757-4 text, or a meter that emits one. Until then, a real meter using those lengths is an interoperability risk worth a targeted test. -## Duty ledger ring capacity (noted 2026-09-01, loc3) +## Duty ledger event-capacity bug (closed 2026-09-01, loc3) -DUTY_RING=384 entries saturates at ~2.7 s of history at ~7 ms mean -airtimes -- BELOW the 3.6 s budget, so the guard silently never triggers -at high TX rates. OBSERVED ACTIVE during the 2026-09-01 beacon soak -(duty_used_us plateaued at ~3.46 s while true hourly airtime was ~20 s; -effective duty ~0.6 %). Fix: replace the ring sum with an hourly-bucketed -accumulator (or ring + saturation flag that fails CLOSED, i.e. suppresses -when the ring wraps inside the window). Scheduled for next flash. +The old `DUTY_RING=384` implementation saturated at about 2.7 s of history +at ~7 ms per event, below the 3.6 s hourly budget. It silently forgot old +events while they were still inside the hour; this was observed during the +beacon soak. + +Fixed: 61 keyed minute buckets accumulate every successful transmission. +The rolling-hour query includes every bucket overlapping the window, +including the oldest partial minute in full (safe over-count). Host tests +cover 10,000 events in one bucket, all 61 overlapping minutes, exact expiry, +invalid timestamps, and zero-length records. diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.md index c7b4217..468b572 100644 --- a/docs/RELIABILITY.md +++ b/docs/RELIABILITY.md @@ -1,6 +1,34 @@ # Reliability -## Soak result (final firmware, `captures/2026-09-01T11-34-08-soak`) +## Overnight standalone battery soak (2026-09-01/02): milestones A-D reached + +DUT on LiPo, no USB host, no observer needed -- firmware counters read via +OLED overnight and archived exactly over serial in the morning +(`captures/2026-09-01T22-48-32-standalone-soak/overnight_final_stats.log`). + +``` +uptime : 42 339 s (11 h 45 m), zero resets +RX candidates : 9 617 = 4 905 valid + 1 423 crc_fail + + 3 274 not_c1 + 15 len_invalid (exact closure) +TX attempts / success : 2 822 / 2 822 (100 %) +TX timeouts / failures : 0 / 0 +radio recoveries/reinits : 0 / 0 +duty suppressions : 2 092 (~41 % of valid traffic over budget) +cumulative TX airtime : 23.72 s computed occupancy (8.4 ms avg/frame) +lifetime duty cycle : 560 ppm (guard held the night at 56 % of cap) +``` + +Milestones A (>=10), B (>=100), C (>=500), D (>=1000 repeats, no restart): +**all REACHED** in one unattended run. Independent RF confirmation the +following morning: meter-anchored capture verified live repeats (2 decoded +duplicates, 2 RF pairs at 53-54 ms vs the 50 ms setting). + +The 2 092 suppressions are the first production evidence for the +STANDARDS_GAPS conclusion that repeat-everything exceeds the 0.1 % duty +budget at a real site (~1.7x here); an installation-list filter is +mandatory for a product. + +## Historical soak result (`captures/2026-09-01T11-34-08-soak`) Repeater running unattended, no SDR involved, repeat delay 200 ms, duty-cycle guard at 0.1 %. @@ -18,12 +46,12 @@ TX attempts : 380 TX success : 380 (100.0 %) TX timeouts / failures : 0 / 0 repeat delay : 200.001 ms min, 200.001 median, 200.006 max -TX airtime : 10.2 / 13.4 / 14.9 ms (min/median/max) +TX operation latency : 10.2 / 13.4 / 14.9 ms (min/median/max) radio recoveries : 0 heartbeats not in RX : 0 heartbeats config lost : 0 -duty-cycle suppressions : 8 (guard working as designed) +old-guard suppressions : 8 (based on operation latency, not RF airtime) ``` ## Milestones (firmware-reported repeats, no restart) @@ -35,23 +63,29 @@ duty-cycle suppressions : 8 (guard working as designed) | C | >= 500 | not reached (run stopped early at 380) | | D | >= 1000 | not reached | -Milestone C needs roughly 2 h at the observed rate, or longer once the -duty-cycle guard caps output at ~268 repeats/hour. The run was ended early -by operator request; nothing in the data suggests it would not continue -- -the last heartbeat was as healthy as the first, with zero recoveries and -zero resets across 1.5 h. To continue: +Milestone C needs roughly 2 h at the observed rate. The run was ended +early by operator request; nothing in the data suggests it would not +continue -- the last heartbeat was as healthy as the first, with zero +recoveries and zero resets across 1.5 h. To continue: ``` nix develop --command ./hil/soak.sh 9000 ``` -## Duty-cycle guard behaviour +## Duty-cycle guard correction -At the observed traffic rate the repeater wants ~320 repeats/hour, which at -13.4 ms measured airtime each is ~0.12 % -- above the 0.1 % ceiling for -868.7-869.2 MHz. The guard suppressed 8 repeats to stay inside the budget -(`used_us` 3 594 561 against `budget_us` 3 600 000) and reported a final -`tx_duty_cycle_ppm` of 905, i.e. 0.0905 %. See docs/STANDARDS_GAPS.md. +The old report multiplied ~320 repeats/hour by 13.4 ms TX-operation +latency and called the result ~0.12 % RF duty. For the representative +102-byte radio payload, computed occupancy is 8.8 ms, so the same rate is +about 0.078 %. The eight suppressions at `used_us` 3,594,561 were caused by +the old conservative wall-time metric, not proof that actual channel +occupancy reached 0.1 %. Likewise, the reported 905 ppm was cumulative +operation latency divided by uptime. + +Current firmware budgets computed packet occupancy and stores it in a +61-minute bucket ledger. Historical stability/TxDone counts above remain +valid; historical airtime, suppression, and duty-rate interpretations do +not. See docs/TIMING.md and docs/STANDARDS_GAPS.md. ## What has NOT been demonstrated diff --git a/docs/RF_CONFIG.md b/docs/RF_CONFIG.md index ca100b7..2174a9c 100644 --- a/docs/RF_CONFIG.md +++ b/docs/RF_CONFIG.md @@ -41,8 +41,15 @@ RX gain: boosted (reg 0x08AC = 0x96) TX params: default -9 dBm configured (min), ramp 40 us; PA config (0x02,0x02,0x00,0x01) = +14 dBm-optimal table, deliberately capped -- the external KCT8103L PA adds gain on top + ``` +PA-row invariant: `deviceSel` must remain `0x00` for SX1262, and the other +bytes must come from an SX1262 row. The removed diagnostic +`(0x01,0x00,0x00,0x01)` mixed the SX1261 +10 dBm PA-duty/hpMax values with +SX1262 selection. This undocumented hybrid reduced measured RF by about +32 dB while all chip-side TX diagnostics still passed. + ## Consequences of the fixed-255-byte RX capture RxDone fires ~ (255 - frame_len) * 80 us after the actual end of frame; @@ -54,5 +61,28 @@ time per frame. Acceptable for the experiment; optimization candidate ## TX power Configured SX1262 power is NOT radiated power: the KCT8103L PA follows. -Development default is the SX1262 minimum (-9 dBm). Log lines carry -`configured_tx_dbm` to keep this distinction explicit. +Development default is the SX1262 minimum (-9 dBm), and the console accepts +only -9..+2 dBm. +2 is the level used by known-good Heltec V4.3 firmware, +not a declaration of legal radiated power. Keep -9 dBm until the complete +FEM/antenna path has a conducted or calibrated e.r.p. measurement. + +`wmbus_tx.configured_tx_dbm` records SetTxParams. `airtime_us` is computed +channel occupancy (48-bit preamble + 16-bit sync + payload at 100 kbps); +`tx_operation_us` is the longer firmware-entry-through-TxDone latency. + +## T-mode (T1) support + +T1 shares the 0x543D access code, so ONE radio configuration receives +both modes: C-mode announces itself with two NRZ mode bytes after the +sync; T-mode continues directly with 3-of-6 coded data (decoded in +software from the same fixed 255-byte capture; wM-Bus C/T share the +100 kchip/s rate). Tables verified against rtl-wmbus. + +TX: T1 frames are 3-of-6 encoded in software and sent with 50 kHz FSK +deviation (raw 0x00CCCD) instead of C-mode's 45 kHz; the swap is a lazy +per-TX SetModulationParams (deviation is a TX-only GFSK parameter, RX is +unaffected). Verified: rtl_wmbus decodes our synthetic T1 frames, +including L=140. + +Length caps from the fixed 255-byte capture: C1 total <= 253 bytes; +T1 total <= 170 decoded bytes (coded 1.5x). diff --git a/docs/RF_FRONTEND_TESTS.md b/docs/RF_FRONTEND_TESTS.md index cefab97..cf95e60 100644 --- a/docs/RF_FRONTEND_TESTS.md +++ b/docs/RF_FRONTEND_TESTS.md @@ -127,6 +127,11 @@ negligible. In the *full* state, however, an earlier IQ capture showed our repeat at +44.8 dB over floor with IQ amplitude at the dongle's full scale (rms 88.6 of 127) while the meter original sat at +8.4 dB - a ~36 dB gap. +These TX amplitudes predate PA-row telemetry. They demonstrate the +observer's compression behavior in those captures, but must not be used as +the output level of the fixed firmware: the now-removed `pa_low` diagnostic +could select an undocumented SX1261-values/SX1262-select hybrid. + Consequence: with a fully sensitive dongle, meter originals decode well at high gain but our own transmission is driven into compression and rtl-wmbus receives it with a few byte errors (CRC fails). Observed @@ -134,15 +139,20 @@ directly - the decoded hex differed from the transmitted frame in 1-2 bytes, e.g. transmitted `...78534f4c5458...` received `...78514f4c5448...`, with the rest of a 113-byte frame bit-exact. -Mitigations applied in firmware: SX1262 at its minimum configured power -(-9 dBm) and the lowest documented PA table (paDutyCycle 0x01, hpMax 0x00, -SX1262 datasheet table 13-21). Remaining mitigation is physical (antenna -separation or an inline attenuator) and is not scriptable from this host. +Firmware correction: the former “lowest documented PA table” +`(paDutyCycle=0x01, hpMax=0x00, deviceSel=0x00, paLut=0x01)` mixed the +PA-duty/hpMax values from the SX1261 +10 dBm row with SX1262 selection. +It is not a documented SX1262 row and cost about 32 dB. It has been +deleted. Current firmware uses documented SX1262 +`(0x02,0x02,0x00,0x01)` with SetTxParams -9 dBm. Receiver compression must +be handled at the observer with fixed low gain, physical separation, or an +attenuator—not by misconfiguring the DUT PA. Note: `rtl_sdr -g 0` selects *automatic* gain, not 0 dB. The lowest real -fixed gain on this tuner is 0.9 dB. At 0.9 and 7.7 dB fixed gain the -degraded dongle decoded nothing at all (neither meters nor our TX), so low -gain is not a usable workaround in that state. +fixed gain on this tuner is 0.9 dB. In the historical degraded-observer +test, 0.9 and 7.7 dB decoded neither meters nor the then-configured TX. +With fixed firmware, a 0.9 dB-gain smoke capture measured all 9 beacons at +median +27.81 dB over floor without clipping. ## Repeat-delay dependence of observer decoding @@ -164,6 +174,8 @@ the firmware's own repeat delay measures 200.0 ms min/median/max. 1. Meter originals and Heltec RX match rate: rtl-wmbus decodes at high gain. 2. Heltec TX framing is decodable as C1: proven with synthetic bursts that are not preceded by a weak frame (`hil/verify_tx.py`). -3. Real repeats: proven at RF level from the same stream's IQ by aligning - firmware TX timestamps to observed bursts (`hil/verify_repeats_iq.py`), - which is immune to both compression and observer sensitivity drift. +3. Real repeats: proven at RF level from the same stream's IQ by + `hil/meter_anchored.py` -- clock fitted on strong meter receptions + (never on TX events, which can borrow their original's burst), then + event-correlated FSK spectral excess against non-TX controls. Immune to + compression and observer drift; classifies strong/weak/not_observed. diff --git a/docs/STANDARDS_GAPS.md b/docs/STANDARDS_GAPS.md index 09ccd84..7995e6a 100644 --- a/docs/STANDARDS_GAPS.md +++ b/docs/STANDARDS_GAPS.md @@ -5,36 +5,88 @@ repeater. Known deviations: | # | Experimental behavior | Standard requirement | |---|---|---| -| 1 | Repeats the raw telegram bit-exact, including hop-related fields | EN 13757-5 repeaters must manage the hop counter / C-field semantics so telegrams are not repeated endlessly | -| 2 | Fixed development repeat delay 20-200 ms (default 50 ms) | OMS repeaters use randomized delays in the seconds range to avoid collisions | -| 3 | Repeats every CRC-valid C1 telegram (optional dev whitelist) | Real repeaters filter by installation list | +| 1 | Marks supported OMS uplinks with H=1 and repairs the affected CRC; unsupported layouts are skipped | Full repeater behavior still requires installation handling and timing compliance; see implementation notes below | +| 2 | Fixed repeat delay 20-1000 ms (persisted; 500 ms in current field testing) | OMS repeaters use randomized delays in the seconds range to avoid collisions | +| 3 | Repeats CRC-valid C1/T1 uplinks passing C-field, hop/access-bit and supported-header checks (optional dev whitelist) | Real repeaters filter by installation list | | 4 | Rolling-hour duty-cycle guard, default 0.1 % (1000 ppm), suppresses repeats over budget and counts them (`tx_suppressed_duty`) | A production repeater needs proper duty-cycle management, or LBT+AFA, plus per-sub-band awareness | -| 5 | Duplicate suppression is a 2.5 s hash ring | Standard requires proper duplicate/loop protection semantics | +| 5 | H=1/R=1 frames are suppressed; a 2.5 s hash ring also suppresses duplicate originals | Full installation and duplicate-management policy still needs review | | 6 | No AES, no decryption, payload treated as opaque | OMS security profiles apply to end devices; a transparent repeater may pass ciphertext, but production behavior needs review | -| 7 | T1 telegrams are detected (shared 0x543D sync) but dropped | A combined C/T repeater would need 3-of-6 decode/encode | -| 8 | Format B frames >128 bytes CRC-checked but rare path untested | -- | +| 7 | C1 and T1 both received and repeated in their native mode (3-of-6 codec in software, per-TX deviation swap). S-mode out of scope | A production repeater still needs reviewed mode policy (e.g. never re-air T1 as C1) | +| 8 | Format B >128 bytes and format A up to L=200 host-tested and verified on air (2026-09-02) | -- | None of these shortcuts may be carried into the production Solenos/Gebos gateway without explicit review. +Scope decision (2026-09-02): S-mode support is explicitly OUT OF SCOPE -- +no S-mode traffic observed at either test site. T-mode receive/repeat and +long-frame support were implemented and RF-verified 2026-09-02. Site +finding from that verification: this site has NO ambient T-mode traffic +either (all ambient decodes are C1); the firmware's historical `rx_not_c1` +bucket is mode-byte-corrupted C1 plus false syncs, not T1 meters. + +Persistence exception (2026-09-02): `duty_budget_ppm` and +`repeat_delay_ms` survive power cycles (NVS), because battery-standalone +testing needs settings that outlive reboots. **A persisted +`duty_budget_ppm` of 0 disables the duty guard indefinitely** -- the +device transmits over the ERC/REC 70-03 budget whenever traffic exceeds +it, until `set duty_budget_ppm 1000` is sent again. The OLED config page +shows the effective values (`DUTYCAP`, `DELAY`); boot logs `cfg_restore`. +Re-enable the guard after every guard-off test session. + +## OMS hop marking (2026-09-09) + +`wmbus_mark_repeated()` in `firmware/main/wmbus.c` operates on a CRC-validated +raw format A/B frame, before C1 framing or T1 3-of-6 encoding. It accepts +only C-fields `44` (SND_NR) and `46` (SND_IR), with H=0 and R=0: + +- ELL CI `8C` or `8E`: set bit 4 of the Communication Control byte. + ELL takes precedence over any inner Transport Layer configuration. +- Without ELL: an explicitly recognized short/long uplink TPL header in + security mode 0 or 5 uses bit 0 of the Configuration field. +- Already-repeated/access-marked frames, truncated headers, and unsupported + layouts are left unchanged and not transmitted. Unsupported includes + headerless CI `78`, ELL `8D`/`8F`, and AFL without a supported outer ELL. + There is no bit-exact fallback. + +Only H and its enclosing block's CRC change. The helper maps logical header +offsets across raw CRC boundaries; it neither decrypts nor copies the payload. +Frame length, meter identity, access number and ciphertext remain unchanged. +The original RX hash remains the RX/TX log correlation key; TX `raw` contains +the actual modified physical payload. Verbose mode reports policy rejection +as `tx_suppressed` with reason `hop_policy`. + +H identifies a repeated copy, not a repeater ID, and does not enable multihop. +Synthetic `test_tx`, `test_tx_t1`, beacons and `test_burst` remain unchanged +PHY probes, not examples of received-frame repeater policy. Existing delay, +whitelist and duty-cycle settings are unchanged. + +Reference: [OMS Volume 2, issue 5.0.1](https://oms-group.org/wp-content/uploads/2024/10/OMS-Spec_Vol2_Primary_v501_01.pdf), +sections 5.2.2.2, 5.2.5.2 and 7.2.4. Setting H alone does not establish full +OMS compliance. The new marking path has not yet been verified over RF. + ## Duty cycle — measured, and why it matters 868.95 MHz falls in the 868.7-869.2 MHz sub-band, which ERC/REC 70-03 (annex h1.6) limits to 25 mW e.r.p. with **<= 0.1 % duty cycle**, unless the device implements LBT+AFA. -Measured on this bench: +Absolute e.r.p. from the SX1262 plus KCT8103L and antenna has not been +calibrated. The default -9 dBm SetTxParams value is the minimum chip setting, +not proof of compliance with the 25 mW e.r.p. cap. + +Current computed occupancy for the historical bench traffic: ``` -TX airtime per repeat : 10.4-14.6 ms (median 13.4 ms), frame-length dependent -observed repeat rate : ~5-6 repeats/min when repeating every valid telegram -resulting duty cycle : ~0.11-0.13 % +RF occupancy per repeat : ~5.8-9.8 ms (8.8 ms for 102 radio bytes) +observed repeat rate : ~5-6 repeats/min when repeating every valid telegram +resulting duty cycle : roughly 0.07-0.09 % at the representative length ``` -That is at or slightly above the 0.1 % ceiling — while repeating only the -CRC-valid C1 telegrams audible at a single indoor location. A production -repeater in a denser environment, or one that also handles T-mode, would -exceed the budget substantially. +An older report used 10.4-14.6 ms TX-operation latency (median 13.4 ms) +as if it were RF airtime and consequently reported 0.11-0.13 %. That was a +safe overestimate but not channel occupancy. Even the corrected site rate +is close to 0.1%; traffic variation, longer frames, a denser environment, +or adding T-mode can exceed the budget. Consequences for the production gateway: diff --git a/docs/TESTING.md b/docs/TESTING.md index 7aaeca7..e0dc262 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -13,8 +13,8 @@ capture -> correlation -> report (text + report.json). Exit 0 = PASS. Useful flags: - `--no-flash` reuse the already-flashed firmware - `--duration N` capture window seconds -- `--tx-power N` configured SX1262 dBm (-9..14; FEM adds gain!) -- `--delay MS` repeat delay (20..200) +- `--tx-power N` configured SX1262 dBm (-9..+2; FEM adds gain!) +- `--delay MS` repeat delay (20..1000; note: run.sh sets it on the DUT, which persists) - `--gain G` SDR tuner gain - `--min-repeats N` PASS threshold @@ -33,15 +33,25 @@ setup.log DUT configuration acks ``` ./tools/discover_devices.sh -v # device inventory -./tools/flash.sh # build + flash +./tools/flash.sh # build + flash; no SDR required ./hil/capture_sdr.sh 60 /tmp/sdr.log # SDR-only capture ./result-py/bin/python3 hil/dut_cmd.py PORT --listen 10 status stats ./result-py/bin/python3 hil/iq_energy.py iq.raw # burst energy probe ./result-py/bin/python3 hil/iq_spectrum.py iq.raw # spectrum sanity ./result-py/bin/python3 hil/sdr_health.py # observer sensitivity -./firmware/test/run_tests.sh # frame-layer tests (ASan) +./firmware/test/run_tests.sh # frame + duty-ledger tests (sanitizers) ``` +Hop-marker regression coverage in the host frame suite checks ELL precedence, +short/long TPL headers, H/R and security-mode rejection, truncated frames, +format A/B block boundaries, unchanged payload bytes and T1 encode/decode. +The 2026-09-09 host smoke also marked three captured meter frames and passed +them through `wmbusmeters`: ELL was reported as `relayed`, mode 5 as `hc=1`, +and the unencrypted water telegram still decoded its readings. Encrypted +captures were not decrypted (no meter keys); ciphertext preservation was +checked byte-for-byte. Target firmware build and HIL CLI smoke passed; +the new marking behavior has not been flashed or RF-verified. + ## DUT serial commands All output is single-line JSON; every line the HIL parses starts with `{`. @@ -54,23 +64,199 @@ radio_status chip mode, command status, device errors, RSSI radio_reset queue a full radio re-init probe low-level scripted SPI/IRQ/TX diagnostic with raw dumps test_tx transmit one synthetic C1 frame +test_tx_t1 transmit one synthetic T1 frame (3-of-6, 50 kHz dev) test_burst transmit the same frame twice, repeat_delay_ms apart inject standby fault injection: leave the radio out of RX inject sleep fault injection: destroy the radio configuration reboot -set repeat_delay_ms N 20..200 -set tx_power N -9..14 (configured SX1262 level; FEM adds gain) +set repeat_delay_ms N 20..1000. PERSISTED across reboots (NVS) +set tx_power N -9..+2 (configured SX1262 level; FEM adds gain) set repeat_enabled 0|1 set verbose 0|1 set fem_rx_csd 0|1 KCT8103L CSD level while in RX (LNA experiment) -set pa_low 0|1 1 = lowest documented PA table set preamble_bits N 0|8|16|24|32 preamble detector length -set duty_budget_ppm N rolling-hour TX airtime cap; 1000 = 0.1 %, 0 = off +set duty_budget_ppm N rolling-hour TX airtime cap; 1000 = 0.1 %, 0 = off. + 0..100000. PERSISTED across reboots (NVS), like delay. + Re-enable after guard-off tests! set test_lfield N 19..200, length of synthetic test frames -set whitelist ID[,ID...] | set whitelist clear +set beacon_period_ms N 0 (off) or 250..600000; periodic synthetic frames +set whitelist ID[,ID...] | set whitelist clear (up to 8 unique eight-hex IDs) ``` +The ten `set` commands, OLED toggles and web writes share validation and +revisioned configuration. Integers/booleans must be complete, not valid-looking +prefixes; malformed or duplicate IDs are rejected rather than truncated. +`status` retains its previous fields and adds all missing settings, +`revision`, `applied_revision` and the complete whitelist. Accepted values can +briefly precede application by the radio task. A TX-power or preamble change +requires radio reinitialization; a copied revision is not hardware/RF proof. + +## On-device UI (OLED + PRG button) + +Standalone debug without a host (docs/FUTURE_UI.md for design + pins): + +- Pages (short press cycles, wraps): live status w/ duty bar -> health + counters -> last telegram -> effective config -> Wi-Fi credentials -> + Wi-Fi join QR. +- Hold ~1.5 s for the page action, each logged as + `{"event":"ui_action",...}`: status = toggle repeat, health = clear + stats, config = toggle beacon at 3000 ms (duty-safe at L=19), either + Wi-Fi page = enable/disable the debug access point. +- Display blanks after 3 min; any press wakes it without acting. +- `{"event":"ui_init","ok":...}` at boot reports display bring-up; a + missing/broken display never blocks the repeater. +- Config shown is the applied RAM snapshot. Delay and duty budget are restored + from NVS on boot; other settings reset. Unplugging USB with LiPo attached + does not reboot the device. + +## Wi-Fi dashboard checks + +Enable **WLAN-DIAGNOSE** using the OLED, join the displayed WPA2 network, and +open `http://192.168.4.1`. The following page provides a standard `WIFI:` +join QR (SSID and password, not the dashboard URL). Wi-Fi starts off on +reboot; its password persists in the separate `wifi_debug` NVS namespace. +On enable/disable, serial emits `wifi_debug` states: 0 off, 1 starting, +2 on, 3 stopping, 4 error, plus the ESP-IDF error code. Credentials are +not logged. Never change the test workstation's Wi-Fi without permission; +phone-based testing avoids disturbing its network. + +The history API remains read-only: `GET /api/events?after=N`. Start at zero; +responses carry `boot_id`, cursor, retention/eviction metadata, cached +status and immutable final rows. A stale/ahead cursor causes `reset`. +Clients must fetch from zero on a new boot ID. The browser polls once per +second and keeps at most 256 rows. Pause freezes the event display, not +recording/polling. RX/repeat counters exclude synthetic events and survive +OLED/serial `clear_stats`, but not reboot. + +### Writable settings API + +Open **Einstellungen am Heltec-Gerät ändern**. Each field has its own draft, +accepted device value, persistence indication and explicit submit action. +The ten fields match the serial commands above; T1 synthetic sends still cap +the effective test L-field at 140. The browser warns before duty 0 or >1000 +and before disabling the AP. CSV names remain browser-only. + +- `GET /api/settings` returns `boot_id`, `revision`, `applied_revision`, + `csrf_token` and `settings` containing all ten keys. +- `POST /api/settings` accepts exactly `{boot_id,revision,key,value}`, one + setting at a time. Values are JSON integers, booleans or an array of up to + eight unique eight-hex-digit whitelist strings; an empty list allows all. + The response has the same snapshot shape. +- `POST /api/wifi/disable` accepts exactly `{boot_id}` and returns `{ok:true}` + before requesting an idempotent stop from the separate Wi-Fi control task. + A lost response leaves the client uncertain; it never retries automatically. +- Writes require `Content-Type: application/json` and `X-CSRF-Token` from GET. + Host must be `192.168.4.1` (optional port 80); foreign Origin/Fetch-Site values + are rejected. The random CSRF token rotates when the AP starts. WPA2 AP + access is the authorization boundary; there is no separate per-user login. +- Bodies are bounded to 512 bytes and shallow JSON, with duplicate/unknown + fields and embedded NULs rejected. Invalid inputs return 400, stale boot or + revision 409, origin/token failures 403, and busy/persistence failures 503. + No failed write is reported as saved. +- Pending settings are copied coherently by the radio owner, never written + into a half-processed reception from HTTP/OLED/USB. NVS runs under the writer + mutex outside the radio task; short spinlocks protect only memory copies. + Settings GETs while pending are capped at 1 Hz and 30 attempts, with no + overlap or blind POST retry. Conflicts preserve drafts and require review. + +Static assets are gzip-compressed at build time by `tools/compress_web_asset.py`, +then served directly with `Content-Encoding: gzip`; no runtime compression or +partition-layout change is required. + +Acceptance scenarios: + +- See each reception's CRC/decode result and repeat completion, failure or + explicit skip. A TX timestamp is software-operation entry (including + driver setup), not an independently measured RF-start timestamp. A TX + outcome timestamp can represent an error, not successful completion. +- Load the downloaded CSV template or observed-ID export, preserving + leading zeros. Quoted commas/newlines are supported. A duplicate ID or + malformed CSV must leave all previous mappings unchanged. Refresh clears + mappings. Labels apply only to CRC-valid received identities. +- Check manufacturer-scoped type meanings (e.g. TCH), unknown raw codes, + version bytes and ambiguity warnings. Identity refers to the link-layer + sender, not necessarily a separately addressed transport-layer meter. +- Pause while new events arrive, reconnect after history eviction, and + reboot: no cross-boot or cross-reception joins are allowed. +- Compare radio timing/health with Wi-Fi off, on and actively serving + clients. Verify disable/re-enable and password persistence. Independently + observe RF if claiming unchanged sensitivity or downstream reception. +- Scan the OLED QR with a phone. Its integer-scaled, positive-polarity + modules retain at least a four-module quiet zone. On this small OLED, + camera focus can still be limiting; use text credentials if needed. +- On 320/390-pixel phone viewports, check that the current signals are easy + to find, controls are tappable and there is no horizontal page overflow. + Open **?** explanations by touch, including near the bottom of the page. + Close with the same button, outside tap, explicit close and Escape. + Explanations must remain usable while live rows update. + All authored visible UI/accessibility text is German; technical raw codes, + exact CSV/API headers and registered company names are intentionally kept. + +2026-09-09 integration checks: test/debug builds and existing host suites +passed. A sanitized history smoke exercised 200,000 writes with concurrent +snapshots, overflow, cursor reset and synthetic/counter separation. +The actual browser assets were exercised on a localhost fixture server: +mobile layout, labels/filtering, atomic CSV rejection, downloaded template, +pause, ambiguity, eviction, disconnect/reconnect, reboot and refresh. +An independent ZBar decoder recovered the expected join payload from the +exact 128x64 framebuffer generated by the production QR renderer. + +The German UI was checked at 320/390 px, including popup scrolling, +44 px help/button targets, local German CSV labels, outside/same-button/ +explicit-close/Escape dismissal and focus restoration. A detail popup +remained open while the live history grew from five to six rows. Expanded +rows initially created a phantom fourth column on phones; the corrected +column spans were exercised across 320 -> 760 -> 390 px without reload. +Page/header widths stayed aligned and there was no horizontal overflow. +Browser checks used coordinate mouse input on narrow/mobile-emulated +viewports; native touchscreen automation timed out, so a physical phone +touch/QR check is still required. Evidence is retained locally under +`captures/2026-09-09-wifi-dashboard/`. + +Pre-settings image sizes were test 927,360 bytes and debug 1,041,264 bytes. +The writable-settings build with compressed assets is now 862,672 bytes +(test, 18% app space free) and 978,656 bytes (debug, 7% free). The partition +layout and NVS locations were not changed. + +After flashing the earlier, pre-settings German test build, USB telemetry at 59.74 s showed +six reception candidates, one CRC-valid reception, one CRC rejection, no +TX errors and no recoveries. Heartbeats reported RX mode and intact radio +configuration. This confirms live firmware operation, not phone access. + +The test-profile firmware was flashed to the attached ESP32-S3. In the +pre-QR hardware smoke, the OLED action started the AP successfully; seven +CRC-valid receptions produced five completed repeats with no recoveries or +TX errors. Four repeats while the AP was on had software delays of +50.001–50.005 ms for a 50 ms setting. This small sample is not a controlled +RX-sensitivity comparison, RF delivery proof, or proof of active HTTP load. +Physical phone scanning and end-to-end HTTP checks require a phone report +or an explicitly authorized network connection; do not infer them from the +localhost fixture checks. + +Writable-settings checks: 49 sanitized host scenarios ran the production +`settings.c`, `settings_http.c` and ESP-IDF cJSON parser through small host +HTTP/FreeRTOS/NVS adapters. They exercised all ten accepted settings, +non-mutating validation/security failures, two concurrent revision-CAS writers, +simulated storage failure/reboot and radio-copy acknowledgement. This is not +physical NVS or radio-handoff verification. + +Chromium exercised all ten forms against that coordinator, preserved unrelated +drafts and edits made while a POST was delayed, rejected malformed input, +handled 409/503 without automatic writes, and gated dangerous duty changes and +Wi-Fi-off with native confirm/cancel dialogs. Confirmed Wi-Fi-off stopped +automatic polling while retaining CSV names. Layout/help were inspected at +320/390 px; setting inputs are 16 px and buttons at least 44 px high. All five +built gzip assets decompressed byte-for-byte to their sources and loaded in +Chromium. Existing frame, duty-ledger, long-T1 and font suites passed. + +The settings firmware is built but **not yet flashed**: the Heltec USB device +disappeared before the flash command could open it; both serial discovery and +USB enumeration found it absent. Reconnect USB for the final on-device setting, +radio-handoff and reboot/persistence smoke. Workstation Wi-Fi was not changed. +Physical phone HTTP/QR checks remain separate. Local proof artifacts are in +`captures/2026-09-09-web-settings/`. + ## Known pitfalls - SDR PLL wedge -> gain-independent white noise; `usbreset 0bda:2838`. @@ -96,6 +282,35 @@ Every run records `SDR sensitivity: full | degraded` from runs report the same state -- this dongle loses ~17 dB within minutes of streaming (docs/HARDWARE.md). +## Meter-anchored TX verification + +When serial telemetry and raw IQ were captured together: + +``` +./result-py/bin/python3 hil/meter_anchored.py \ + --iq iq.raw --serial serial.log --json +``` + +The verifier first fits the DUT/IQ clocks only from strong meter receptions, +which prevents a repeat from borrowing its own original burst. It then +compares the median spectrum at exact TX times with nearby non-TX control +windows. `rf_state` is: + +- `strong`: event-correlated FSK is present and median event energy is at + least 6 dB over the capture floor; +- `weak`: event-correlated FSK is present below that energy threshold; +- `not_observed`: no event-correlated FSK signature; +- `no_events`: the serial log contained no in-range TX events. + +`tx_above_6db` is only an energy count; it is not an RF verdict by itself. +`fsk_excess_db` is the in-band (20-80 kHz offset) event/control excess after +subtracting wideband (250-700 kHz offset) event/control change. At least +five analyzable TX/control pairs are required. + +For unattended beacon runs, `hil/beacon_soak.sh` discards raw IQ only when +the result is `strong`. Weak, absent, or unclassified segments retain IQ +for reanalysis. + ## Long reliability soak (DUT only, no SDR) ``` @@ -161,10 +376,13 @@ cat iq.raw | rtl_wmbus -f > decodes.log ./result-py/bin/python3 hil/verify_standalone.py --iq iq.raw --decodes decodes.log ``` -Reports decoded repeats (same payload twice) and RF-level repeat pairs -(burst pairs of matching duration inside the delay window). The latter -matters because our transmission usually overloads the dongle and fails CRC -even though the RF is correct. +Reports decoded C1/T1 repeats (a supported H=0 original followed by its H=1 +counterpart, with every other cooked byte identical) and RF-level repeat +pairs (burst pairs of matching duration inside the delay window). Decoded +pairs use occurrence order because SDR timestamps are quantized; each +original is matched once. Same-H duplicate decodes are not repeat proof. +The RF-level check matters because our transmission usually overloads the +dongle and fails CRC even though the RF is correct. Note: only one process can hold the RTL-SDR. If another tool (e.g. an oms-tui / wmbusmeters pipeline) is running, `rtl_sdr` fails with diff --git a/docs/TIMING.md b/docs/TIMING.md index 8f0c069..9d55ca0 100644 --- a/docs/TIMING.md +++ b/docs/TIMING.md @@ -9,24 +9,24 @@ repeats, 214 RX candidates, repeat delay set to 200 ms). | Quantity | min | median | max | n | |---|---|---|---|---| | RxDone -> reconstructed end of frame ("deaf tail") | 9 360 us | 12 240 us | 17 120 us | 214 | -| TX airtime (SetTx to TxDone) | 10 421 us | 13 422 us | 14 560 us | 72 | -| Programmed repeat delay (frame end -> TX start) | 200 001 us | 200 001 us | 200 008 us | 72 | +| TX operation latency (routine entry -> observed TxDone) | 10 421 us | 13 422 us | 14 560 us | 72 | +| Scheduled frame end -> TX-routine entry | 200 001 us | 200 001 us | 200 008 us | 72 | | Repeated frame length (over-the-air bytes) | 63 | 100 | 113 | 72 | Independent confirmation from the SDR side (raw IQ, same run): observed burst duration differs from the computed airtime by 0.2-1.5 ms -(median 1.2 ms), and firmware TX events align to observed RF within -0-3.1 ms (median 2.4 ms). +(median 1.2 ms), and firmware TX-operation events align to observed RF +within 0-3.1 ms (median 2.4 ms). ## Derived ``` -repeat delay accuracy : +1 us to +8 us against a 200 ms setting - (jitter well under 10 us) -RX deaf time per repeat : ~225.7 ms median +repeat scheduling accuracy : +1 us to +8 us against a 200 ms setting + (at TX-routine entry, not RF onset) +RX deaf time per repeat : ~225.7 ms historical median = 12.2 ms deaf tail + 200.0 ms programmed delay - + 13.4 ms TX airtime + + 13.4 ms TX operation latency ``` The programmed delay dominates by design; it exists to make SDR @@ -59,31 +59,41 @@ a production repeater. artefact, not a design target (see docs/STANDARDS_GAPS.md). - The fixed-255 tail is an SX1262 packet-engine limitation, so it applies to any SX126x-based design. Budget for it, or implement early-stop RX. -- Repeat-delay accuracy of a few microseconds shows the ESP32-S3 side has - ample timing margin; timing precision will not be the constraint. +- TX-routine scheduling jitter is only a few microseconds. Actual RF starts + after packet setup and buffer transfer, so it must be measured at RF when + exact over-the-air delay matters. -## Airtime measurement vs true channel occupancy +## Computed airtime vs TX operation latency -`tx_airtime_us` and the duty-cycle ledger both use wall time from `SetTx` -to the moment `TxDone` is observed. That interval includes SPI round-trips -and up to ~1 ms of polling granularity, so it over-states true channel -occupancy: +Current `airtime_us`, cumulative `stats.tx_airtime_us`, and the duty ledger +all use computed channel occupancy: ``` -computed airtime, 102-byte payload : (102 * 80 us) + 640 us = 8.8 ms -measured SetTx -> TxDone observed : 13.4 ms median +airtime = 48-bit preamble + 16-bit sync + payload at 100 kbps + = (radio_payload_bytes * 80 us) + 640 us + +102-byte radio payload : 8.8 ms +26-byte beacon payload : 2.72 ms ``` -Consequences, both deliberate: +`wmbus_tx.tx_operation_us` separately records the longer interval from +entry into the TX routine through observed TxDone. It includes standby, +packet-parameter and buffer SPI transactions, the actual transmission, and +polling/notification latency. Historical firmware mislabeled this interval +as `airtime_us`; its 102-byte median was 13.4 ms. -- The reported duty cycle is conservative (over-counted by roughly 50 %), - so the 0.1 % guard suppresses slightly earlier than strictly required. - For a compliance guard, erring high is the safe direction. -- Independent confirmation that the real occupancy matches the computed - airtime comes from the SDR side, not the firmware: observed burst - durations in raw IQ differ from the computed airtime by 0.2-1.5 ms - (median 1.2 ms), i.e. the transmission itself is ~9-10 ms for a 100-byte - frame as expected. +The distinction matters: -A production implementation should budget on computed airtime and use the -measured value only for latency diagnostics. +- Computed occupancy is independent of task scheduling and SPI speed, so + the duty guard no longer consumes roughly 50% extra budget. +- Operation latency remains useful for diagnosing stalls and TxDone timing, + but is not a regulatory airtime measurement. +- Raw-IQ burst duration remains the independent check. In the historical + 102-byte run it differed from computed airtime by 0.2-1.5 ms (median + 1.2 ms). + +The rolling-hour ledger uses 61 keyed one-minute buckets rather than a +fixed event ring. It therefore cannot forget high-rate events merely +because an event count was exceeded. The oldest overlapping minute is +counted in full, making the result conservatively high by at most one +minute's included traffic at the window boundary.