docs: fault-injection evidence for recovery ladder
This commit is contained in:
@@ -109,6 +109,21 @@
|
|||||||
rtl_sdr probes and disables direct-sampling mode); it is not a wedge
|
rtl_sdr probes and disables direct-sampling mode); it is not a wedge
|
||||||
indicator. The real symptom is railed, gain-independent noise.
|
indicator. The real symptom is railed, gain-independent noise.
|
||||||
|
|
||||||
|
## CONFIRMED (failure recovery)
|
||||||
|
|
||||||
|
- The recovery ladder works and escalates correctly, verified by fault
|
||||||
|
injection rather than assumption:
|
||||||
|
`inject standby` -> heartbeat sees chip_mode 2 -> level 1 restores RX;
|
||||||
|
`inject sleep` (configuration destroyed) -> heartbeat sees chip_mode 0 and
|
||||||
|
`config_intact:false` -> level 1 reports "failed" -> level 2 (full radio
|
||||||
|
reinit) restores RX, confirmed by chip_mode 5 + config intact + plausible
|
||||||
|
RSSI on the following heartbeats.
|
||||||
|
- Checking chip mode alone is NOT sufficient to detect a broken receiver: a
|
||||||
|
radio that lost its configuration can sit in RX looking healthy while
|
||||||
|
deaf to C1. The firmware therefore reads the sync-word registers back
|
||||||
|
every heartbeat (`config_intact`), and recovery levels only report success
|
||||||
|
when both the mode and the configuration verify.
|
||||||
|
|
||||||
## NEEDS TESTING
|
## NEEDS TESTING
|
||||||
|
|
||||||
- Narrower RX bandwidth (156 kHz) vs current 234.3 kHz on weak meters.
|
- Narrower RX bandwidth (156 kHz) vs current 234.3 kHz on weak meters.
|
||||||
|
|||||||
@@ -72,3 +72,33 @@ Every run records `SDR sensitivity: full | degraded` from
|
|||||||
`hil/sdr_health.py`. Treat cross-run comparisons as invalid unless both
|
`hil/sdr_health.py`. Treat cross-run comparisons as invalid unless both
|
||||||
runs report the same state -- this dongle loses ~17 dB within minutes of
|
runs report the same state -- this dongle loses ~17 dB within minutes of
|
||||||
streaming (docs/HARDWARE.md).
|
streaming (docs/HARDWARE.md).
|
||||||
|
|
||||||
|
## Long reliability soak (DUT only, no SDR)
|
||||||
|
|
||||||
|
```
|
||||||
|
nix develop --command ./hil/soak.sh 9000 # seconds
|
||||||
|
```
|
||||||
|
|
||||||
|
Measures firmware-side reliability over hours: repeat success rate, timing
|
||||||
|
stability, radio recoveries, config-integrity failures and MCU resets, and
|
||||||
|
reports progress against the milestone targets. Deliberately does not use
|
||||||
|
the SDR, which cannot stream for hours without faulting.
|
||||||
|
|
||||||
|
## Fault injection (verifying the recovery ladder)
|
||||||
|
|
||||||
|
```
|
||||||
|
./result-py/bin/python3 hil/dut_cmd.py <port> --listen 26 "inject standby"
|
||||||
|
./result-py/bin/python3 hil/dut_cmd.py <port> --listen 26 "inject sleep"
|
||||||
|
```
|
||||||
|
|
||||||
|
`standby` leaves the radio out of RX with its configuration intact;
|
||||||
|
`sleep` additionally destroys the configuration. The heartbeat probe must
|
||||||
|
detect both and the recovery ladder must restore RX. Verified behaviour:
|
||||||
|
|
||||||
|
```
|
||||||
|
inject standby -> heartbeat chip_mode=2, config_intact=true
|
||||||
|
-> radio_recovery level 1 -> rx_restored
|
||||||
|
inject sleep -> heartbeat chip_mode=0, config_intact=false
|
||||||
|
-> radio_recovery level 1 -> failed (correctly refuses)
|
||||||
|
-> radio_recovery level 2 -> rx_restored
|
||||||
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user