From 70762362a4bfa2d12291a979cedf797bffa3a621 Mon Sep 17 00:00:00 2001 From: Lars Nolden Date: Tue, 1 Sep 2026 11:21:04 +0200 Subject: [PATCH] docs: fault-injection evidence for recovery ladder --- docs/FINDINGS.md | 15 +++++++++++++++ docs/TESTING.md | 30 ++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index d71828f..aaa39f8 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -109,6 +109,21 @@ rtl_sdr probes and disables direct-sampling mode); it is not a wedge indicator. The real symptom is railed, gain-independent noise. +## CONFIRMED (failure recovery) + +- The recovery ladder works and escalates correctly, verified by fault + injection rather than assumption: + `inject standby` -> heartbeat sees chip_mode 2 -> level 1 restores RX; + `inject sleep` (configuration destroyed) -> heartbeat sees chip_mode 0 and + `config_intact:false` -> level 1 reports "failed" -> level 2 (full radio + reinit) restores RX, confirmed by chip_mode 5 + config intact + plausible + RSSI on the following heartbeats. +- Checking chip mode alone is NOT sufficient to detect a broken receiver: a + radio that lost its configuration can sit in RX looking healthy while + deaf to C1. The firmware therefore reads the sync-word registers back + every heartbeat (`config_intact`), and recovery levels only report success + when both the mode and the configuration verify. + ## NEEDS TESTING - Narrower RX bandwidth (156 kHz) vs current 234.3 kHz on weak meters. diff --git a/docs/TESTING.md b/docs/TESTING.md index 8f56f6c..7a324a9 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -72,3 +72,33 @@ Every run records `SDR sensitivity: full | degraded` from `hil/sdr_health.py`. Treat cross-run comparisons as invalid unless both runs report the same state -- this dongle loses ~17 dB within minutes of streaming (docs/HARDWARE.md). + +## Long reliability soak (DUT only, no SDR) + +``` +nix develop --command ./hil/soak.sh 9000 # seconds +``` + +Measures firmware-side reliability over hours: repeat success rate, timing +stability, radio recoveries, config-integrity failures and MCU resets, and +reports progress against the milestone targets. Deliberately does not use +the SDR, which cannot stream for hours without faulting. + +## Fault injection (verifying the recovery ladder) + +``` +./result-py/bin/python3 hil/dut_cmd.py --listen 26 "inject standby" +./result-py/bin/python3 hil/dut_cmd.py --listen 26 "inject sleep" +``` + +`standby` leaves the radio out of RX with its configuration intact; +`sleep` additionally destroys the configuration. The heartbeat probe must +detect both and the recovery ladder must restore RX. Verified behaviour: + +``` +inject standby -> heartbeat chip_mode=2, config_intact=true + -> radio_recovery level 1 -> rx_restored +inject sleep -> heartbeat chip_mode=0, config_intact=false + -> radio_recovery level 1 -> failed (correctly refuses) + -> radio_recovery level 2 -> rx_restored +```