From f7c8614a3bc2200081307fed8cad2954422a6b68 Mon Sep 17 00:00:00 2001 From: Lars Nolden Date: Mon, 31 Aug 2026 22:43:55 +0200 Subject: [PATCH] fix: TCXO XOSC_START_ERR handling, always apply config, register DIO1 ISR callback; TX proof of life confirmed by SDR --- docs/DEVBOARD_REQUIREMENTS.md | 30 ++++++++++ docs/FINDINGS.md | 40 ++++++++++++++ docs/OPEN_QUESTIONS.md | 18 ++++++ docs/TESTING.md | 57 +++++++++++++++++++ firmware/main/console.c | 4 ++ firmware/main/repeater.c | 101 ++++++++++++++++++++++++++++++++-- firmware/main/repeater.h | 1 + firmware/main/sx126x.c | 42 +++++++++++--- firmware/main/sx126x.h | 4 ++ 9 files changed, 283 insertions(+), 14 deletions(-) create mode 100644 docs/DEVBOARD_REQUIREMENTS.md create mode 100644 docs/FINDINGS.md create mode 100644 docs/OPEN_QUESTIONS.md create mode 100644 docs/TESTING.md diff --git a/docs/DEVBOARD_REQUIREMENTS.md b/docs/DEVBOARD_REQUIREMENTS.md new file mode 100644 index 0000000..53db5bf --- /dev/null +++ b/docs/DEVBOARD_REQUIREMENTS.md @@ -0,0 +1,30 @@ +# Future Solenos/Gebos dev-board requirements + +Derived from actual debugging needs during this experiment. Machine-readable +pin list: devboard_signals.csv. + +## Debug access that proved necessary + +- SX1262 BUSY + DIO1 on test points: needed to reason about command timing, + IRQ latency, and stuck-BUSY recovery without guessing. + "I needed direct access to this signal because the only alternative was + inferring radio state from SPI responses." +- Full SPI bus (NSS/SCK/MOSI/MISO) on a header: logic-analyzer decode of + the SX1262 command stream is the ground truth when firmware and radio + disagree. +- FEM control lines (PWR_EN, CSD, CTX): RX sensitivity and TX output both + silently break if these are wrong; scoping them is the fastest check. + CSD doubles as board-revision detect (pull level) — keep it probeable. +- Native USB pins must remain dedicated to flash/console; the DUT USB + identity changes between app and ROM bootloader, so the dev board should + also expose UART0 TX as an always-on log output (the Heltec does not, + which made early-boot debugging blind). + +## Wishlist beyond the current board + +- Always-available secondary UART for logs (independent of native USB). +- Measurable radio power rail (current shunt) to correlate TX events with + actual PA draw — would have settled "is the FEM PA active?" instantly. +- RF test connector (u.FL tap or directional coupler) for conducted + measurements instead of over-the-air-only verification. +- ESP32 BOOT/RESET buttons brought to a header for automated strapping. diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md new file mode 100644 index 0000000..11825d4 --- /dev/null +++ b/docs/FINDINGS.md @@ -0,0 +1,40 @@ +# Findings + +## CONFIRMED + +- Heltec V4.3 SX1262 pin mapping (NSS 8, SCK 9, MOSI 10, MISO 11, RST 12, + BUSY 13, DIO1 14): SPI write/readback passes, radio enters RX + (`radio_init: spi:true, reset:true, tcxo:true, result:ok`). +- TCXO is DIO3-controlled at 1.8 V; without post-TCXO recalibration the + XOSC_START_ERR device error appears. With recalibrate: no device errors. +- KCT8103L FEM on GPIO7 (power), GPIO2 (CSD), GPIO5 (CTX), CPS on DIO2. + RX works with CSD high + CTX low + DIO2-as-RF-switch. +- Firmware heartbeat RSSI in RX: -101..-107 dBm noise floor -> RX chain + with boosted gain is plausibly alive. +- Nooelec NESDR SMArt v5 R820T PLL wedges ("PLL not locked"), producing + gain-independent white noise on all frequencies until USB reset. + `usbreset 0bda:2838` reliably restores operation. Trigger correlated + with `rtl_test -t` direct-sampling probe. +- rtl-sdr-blog and osmocom rtl_sdr builds behave identically w.r.t. the + wedge; not a driver-fork issue. +- Real C1 traffic exists at this site (first telegram decoded 22:23 local, + meter ID 04767752). + +## LIKELY + +- Fixed 255-byte RX capture adds up to ~20 ms artificial dead time per + frame (by construction; to be measured). + +## DISPROVEN + +- "rtl_sdr pipeline works out of the box": the PLL wedge cost ~20 minutes + of empty captures. Always energy-probe the SDR before trusting a null + result. + +## NEEDS TESTING + +- Heltec C1 RX of real meter frames (SDR sees traffic; Heltec match TBD). +- TX proof of life via SDR. +- LNA on/off comparison (KCT8103L CSD low in RX = bypass?). +- Repeat delay accuracy, RX deaf time distribution. +- Long-run stability (>=100, >=500 repeats). diff --git a/docs/OPEN_QUESTIONS.md b/docs/OPEN_QUESTIONS.md new file mode 100644 index 0000000..22aaf5f --- /dev/null +++ b/docs/OPEN_QUESTIONS.md @@ -0,0 +1,18 @@ +# Open Questions + +- Optimal SX1262 RX bandwidth for C1: 234.3 kHz chosen (signal ~190 kHz + incl. tolerances); would 156 kHz improve sensitivity without clipping + drifted meters? +- Preamble detector length: 8 bits chosen for sensitivity; false-sync rate + in the field? (false syncs cost ~20 ms deaf time each at fixed-255 RX). +- Does the KCT8103L LNA help or hurt with strong 868 MHz interferers? + (see planned docs/RF_FRONTEND_TESTS.md) +- RX dead time during retransmit: measured distribution? +- Can the SX1262 re-arm RX fast enough to catch a meter's immediate + follow-up telegram after our TX? +- Exact production repeater framing/hop requirements (EN 13757-5). +- Regulatory duty-cycle budget for a production repeater at 868.95 MHz. +- Modulation shaping for TX: none vs GFSK BT=0.5 — which does rtl-wmbus / + real receivers prefer? (TX currently unshaped 2-FSK.) +- Why did the R820T PLL wedge? (usbreset workaround in place; root cause + unknown, suspected rtl_test direct-sampling probe.) diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..64d9ca2 --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,57 @@ +# Testing + +## One-command HIL test + +``` +nix develop --command ./hil/run.sh --duration 300 +``` + +Performs: device discovery (aborts on ambiguity) -> SDR USB reset -> +firmware build + flash -> DUT configuration -> concurrent serial + SDR +capture -> correlation -> report (text + report.json). Exit 0 = PASS. + +Useful flags: +- `--no-flash` reuse the already-flashed firmware +- `--duration N` capture window seconds +- `--tx-power N` configured SX1262 dBm (-9..14; FEM adds gain!) +- `--delay MS` repeat delay (20..200) +- `--gain G` SDR tuner gain +- `--min-repeats N` PASS threshold + +## Artifacts + +Each run writes `captures/-hil/`: +``` +serial.log DUT JSON-lines (wmbus_rx / wmbus_tx / heartbeats / stats) +rtlwmbus.log rtl_wmbus decodes +report.json machine-readable correlation result +report.txt human-readable report +setup.log DUT configuration acks +``` + +## Manual pieces + +``` +./tools/discover_devices.sh -v # device inventory +./tools/flash.sh # build + flash +./hil/capture_sdr.sh 60 /tmp/sdr.log # SDR-only capture +./result-py/bin/python3 hil/dut_cmd.py PORT --listen 10 status stats +./result-py/bin/python3 hil/iq_energy.py iq.raw # burst energy probe +./result-py/bin/python3 hil/iq_spectrum.py iq.raw # spectrum sanity +``` + +## DUT serial commands + +``` +status | stats | clear_stats | radio_status | radio_reset | test_tx +set repeat_delay_ms N | set tx_power N | set repeat_enabled 0|1 +set verbose 0|1 | set whitelist ID[,ID...] | set whitelist clear | reboot +``` + +## Known pitfalls + +- SDR PLL wedge -> gain-independent white noise; `usbreset 0bda:2838`. +- Never wrap the rtl pipeline in `timeout bash -c` (orphans children); + apply timeout to `rtl_sdr` itself. +- ESP32-S3 USB re-enumerates between app CDC and ROM bootloader; flash + scripts rediscover the port on failure. diff --git a/firmware/main/console.c b/firmware/main/console.c index be5cd0c..2a73f57 100644 --- a/firmware/main/console.c +++ b/firmware/main/console.c @@ -88,6 +88,10 @@ static void console_task(void *arg) repeater_request_radio_reset(); ack("radio_reset", true, "queued"); } + else if (strcmp(line, "probe") == 0) { + repeater_request_probe(); + ack("probe", true, "queued"); + } else if (strcmp(line, "test_tx") == 0) { repeater_request_test_tx(); ack("test_tx", true, "queued"); diff --git a/firmware/main/repeater.c b/firmware/main/repeater.c index 6804bc3..066f7c7 100644 --- a/firmware/main/repeater.c +++ b/firmware/main/repeater.c @@ -9,6 +9,7 @@ #include "freertos/FreeRTOS.h" #include "freertos/task.h" #include "esp_timer.h" +#include "driver/gpio.h" #include "esp_task_wdt.h" #include "esp_system.h" @@ -31,6 +32,7 @@ static TaskHandle_t s_task; static volatile int64_t s_irq_ts; static volatile bool s_want_radio_reset; static volatile bool s_want_test_tx; +static volatile bool s_want_probe; static uint8_t s_test_access_nr; static uint32_t s_seq; @@ -62,9 +64,13 @@ static void irq_cb(int64_t ts) static bool radio_full_init(void) { sx126x_diag_t diag; - bool ok = sx126x_init(&diag); - bool cfg_ok = ok && sx126x_config_wmbus_c1(868950000u, RX_PAYLOAD_LEN) - && sx126x_set_tx_power(g_cfg.tx_power_dbm); + bool init_ok = sx126x_init(&diag); + // Apply the wM-Bus configuration even when init reports soft failures: + // an unconfigured radio in RX is worse than a partially degraded one, + // and the recovery ladder judges the combined result anyway. + bool cfg_ok = sx126x_config_wmbus_c1(868950000u, RX_PAYLOAD_LEN) + & sx126x_set_tx_power(g_cfg.tx_power_dbm); + cfg_ok = init_ok && cfg_ok; LOGJ("\"event\":\"radio_init\",\"spi\":%s,\"reset\":%s,\"busy_low\":%s," "\"tcxo\":%s,\"device_errors\":%u,\"chip_mode\":%u," "\"fem_power\":true,\"fem_mode\":\"rx\",\"frequency_hz\":868950000," @@ -181,12 +187,33 @@ static const char *do_tx(const uint8_t *payload, int payload_len, } // airtime: 48 bit preamble + 16 bit sync + payload -> ~80us/byte + 640 uint32_t airtime_ms = (payload_len * US_PER_BYTE + 640) / 1000; - uint16_t irq = wait_irq(airtime_ms + 200); - *tx_done_us = s_irq_ts; + // Instrumented wait: poll IRQ flags, DIO1 level and chip mode while + // also honoring the ISR notification. Gives ground truth about the + // TX sequence even when the interrupt path is broken. + int64_t deadline = esp_timer_get_time() + (airtime_ms + 200) * 1000LL; + uint16_t irq = 0; + bool notify_seen = false, dio1_seen = false; + uint8_t max_mode = 0; + while (esp_timer_get_time() < deadline) { + if (ulTaskNotifyTake(pdTRUE, 0) > 0) notify_seen = true; + if (gpio_get_level(BOARD_LORA_DIO1)) dio1_seen = true; + uint8_t st = sx126x_get_status(); + uint8_t mode = (st >> 4) & 0x7; + if (mode > max_mode) max_mode = mode; + irq = sx126x_get_irq(); + if (irq) break; + vTaskDelay(pdMS_TO_TICKS(1)); + } + *tx_done_us = notify_seen ? s_irq_ts : esp_timer_get_time(); fem_set_tx(false); + sx126x_clear_irq(SX126X_IRQ_ALL); + LOGJ("\"event\":\"tx_diag\",\"irq\":%u,\"notify\":%s,\"dio1_high_seen\":%s," + "\"max_chip_mode\":%u,\"device_errors\":%u", + irq, notify_seen ? "true" : "false", dio1_seen ? "true" : "false", + max_mode, sx126x_get_device_errors()); if (irq & SX126X_IRQ_TX_DONE) { g_stats.tx_success++; - return "ok"; + return notify_seen ? "ok" : "ok_polled"; } if (irq & SX126X_IRQ_TIMEOUT) { g_stats.tx_timeout++; @@ -231,6 +258,62 @@ static void run_test_tx(void) sx126x_restore_rx_payload_len(); if (!sx126x_set_rx()) recover_ladder("test_tx_rx_restore"); } +// ---------------------------------------------------------------- probe + +static void hexdump_json(const char *key, const uint8_t *d, int n) +{ + printf("\"%s\":\"", key); + print_hex(d, n); + printf("\","); +} + +// Scripted low-level diagnostic with raw SPI response dumps. +static void run_probe(void) +{ + uint8_t r[16]; + + printf("{\"event\":\"probe\","); + + sx126x_standby(); + sx126x_raw(0xC0, NULL, 0, r, 2); // GetStatus + hexdump_json("status0", r, 2); + + // SetDioIrqParams: all IRQs, all on DIO1 + uint8_t di[8] = {0x03, 0xFF, 0x03, 0xFF, 0, 0, 0, 0}; + sx126x_raw(0x08, di, 8, r, 9); + hexdump_json("setirq_resp", r, 9); + sx126x_raw(0xC0, NULL, 0, r, 2); // status after SetDioIrqParams + hexdump_json("status_after_setirq", r, 2); + + sx126x_raw(0x02, (uint8_t[]){0x03, 0xFF}, 2, NULL, 0); // ClearIrq + sx126x_raw(0x12, NULL, 0, r, 4); // GetIrqStatus raw + hexdump_json("irq_cleared", r, 4); + + // small TX: 8 byte payload + uint8_t pkt[8] = {0x54, 0xCD, 0xAA, 0x55, 0xDE, 0xAD, 0xBE, 0xEF}; + sx126x_start_tx(pkt, 8); + printf("\"txpoll\":["); + for (int i = 0; i < 12; i++) { + vTaskDelay(pdMS_TO_TICKS(2)); + uint8_t s2[2], q[4]; + sx126x_raw(0xC0, NULL, 0, s2, 2); + sx126x_raw(0x12, NULL, 0, q, 4); + printf("\"%02X-%02X%02X%02X%02X\"%s", s2[1], q[0], q[1], q[2], q[3], + i < 11 ? "," : ""); + } + printf("],"); + fem_set_tx(false); + + sx126x_raw(0x17, NULL, 0, r, 4); // GetDeviceErrors + hexdump_json("dev_errors", r, 4); + + sx126x_clear_irq(SX126X_IRQ_ALL); + sx126x_restore_rx_payload_len(); + sx126x_set_rx(); + printf("\"done\":true}\n"); + fflush(stdout); +} + // ------------------------------------------------------------------ RX path @@ -333,6 +416,7 @@ static void repeater_task(void *arg) { (void)arg; s_task = xTaskGetCurrentTaskHandle(); + sx126x_set_irq_callback(irq_cb); esp_task_wdt_add(NULL); if (!radio_full_init()) { @@ -354,6 +438,10 @@ static void repeater_task(void *arg) s_want_test_tx = false; run_test_tx(); } + if (s_want_probe) { + s_want_probe = false; + run_probe(); + } if (ulTaskNotifyTake(pdTRUE, pdMS_TO_TICKS(1000)) > 0) { uint16_t irq = sx126x_get_irq(); @@ -388,6 +476,7 @@ void repeater_start(void) void repeater_request_radio_reset(void) { s_want_radio_reset = true; } void repeater_request_test_tx(void) { s_want_test_tx = true; } +void repeater_request_probe(void) { s_want_probe = true; } // ------------------------------------------------------------------ status diff --git a/firmware/main/repeater.h b/firmware/main/repeater.h index 78b7d7c..1cf31c1 100644 --- a/firmware/main/repeater.h +++ b/firmware/main/repeater.h @@ -36,6 +36,7 @@ extern repeater_stats_t g_stats; void repeater_start(void); void repeater_request_radio_reset(void); void repeater_request_test_tx(void); +void repeater_request_probe(void); void repeater_print_status(void); void repeater_print_stats(void); void repeater_print_radio_status(void); diff --git a/firmware/main/sx126x.c b/firmware/main/sx126x.c index 493b647..70751de 100644 --- a/firmware/main/sx126x.c +++ b/firmware/main/sx126x.c @@ -231,10 +231,22 @@ bool sx126x_init(sx126x_diag_t *diag) read_regs(REG_SYNCWORD_0, verify, 2); diag->spi_ok = (verify[0] == 0x54 && verify[1] == 0x3D); - // TCXO on DIO3 (1.8 V, ~4 ms), then recalibrate everything because - // enabling the TCXO invalidates the RC-based calibration. + // TCXO on DIO3 (1.8 V, ~4 ms). The first calibrate after enabling the + // TCXO latches a benign XOSC_START_ERR (the XOSC block briefly probes + // the crystal before DIO3 powers the TCXO). Correct sequence: enable, + // clear stale errors, force STDBY_XOSC (actually starts the TCXO), + // recalibrate, then judge the remaining error flags. uint8_t tcxo[4] = {BOARD_TCXO_VOLTAGE_CODE, 0x00, 0x00, BOARD_TCXO_DELAY_STEPS}; cmd(OP_SET_DIO3_TCXO_CTRL, tcxo, 4); + cmd(OP_CLEAR_DEVICE_ERRORS, (uint8_t[]){0, 0}, 2); + + uint8_t stdby_xosc = 0x01; + cmd(OP_SET_STANDBY, &stdby_xosc, 1); + vTaskDelay(pdMS_TO_TICKS(10)); + busy_wait(); + bool xosc_running = (((sx126x_get_status() >> 4) & 0x7) == 0x3); + + cmd(OP_CLEAR_DEVICE_ERRORS, (uint8_t[]){0, 0}, 2); uint8_t calib = 0x7F; cmd(OP_CALIBRATE, &calib, 1); vTaskDelay(pdMS_TO_TICKS(10)); @@ -242,9 +254,9 @@ bool sx126x_init(sx126x_diag_t *diag) uint16_t errs = sx126x_get_device_errors(); diag->device_errors = errs; - // XOSC_START_ERR (bit 5) would indicate TCXO trouble - diag->tcxo_ok = ((errs & 0x20) == 0); + diag->tcxo_ok = xosc_running && ((errs & 0x20) == 0); if (errs) cmd(OP_CLEAR_DEVICE_ERRORS, (uint8_t[]){0, 0}, 2); + cmd(OP_SET_STANDBY, &(uint8_t){0x00}, 1); // DC-DC regulator (Heltec module has the inductor; RNode uses default LDO, // but SX1262 modules from Heltec are DCDC-capable. Use DCDC for lower @@ -312,11 +324,11 @@ bool sx126x_config_wmbus_c1(uint32_t freq_hz, uint8_t rx_payload_len) ok &= cmd(OP_SET_BUFFER_BASE, (uint8_t[]){0x00, 0x00}, 2); - // IRQs on DIO1 - uint16_t irqs = SX126X_IRQ_TX_DONE | SX126X_IRQ_RX_DONE | SX126X_IRQ_TIMEOUT; + // IRQs: enable ALL in the latch mask (diagnosis of missing TxDone/RxDone; + // flags only latch when enabled here), route all to DIO1. uint8_t di[8] = { - irqs >> 8, irqs & 0xFF, // IRQ mask - irqs >> 8, irqs & 0xFF, // DIO1 + 0x03, 0xFF, // IRQ mask: all + 0x02, 0x03, // DIO1: TxDone | RxDone | Timeout 0x00, 0x00, // DIO2 (RF switch) 0x00, 0x00, // DIO3 (TCXO) }; @@ -445,6 +457,20 @@ uint16_t sx126x_get_device_errors(void) if (!cmd_read(OP_GET_DEVICE_ERRORS, e, 2)) return 0xFFFF; return ((uint16_t)e[0] << 8) | e[1]; } +bool sx126x_raw(uint8_t op, const uint8_t *args, size_t n_tx, + uint8_t *rx, size_t rx_len) +{ + uint8_t tx[32] = {0}, rxb[32] = {0}; + size_t total = 1 + n_tx; + if (rx_len > total) total = rx_len; + if (total > sizeof(tx)) return false; + tx[0] = op; + if (n_tx) memcpy(tx + 1, args, n_tx); + if (!xfer(tx, rxb, total)) return false; + if (rx && rx_len) memcpy(rx, rxb, rx_len); + return true; +} + uint8_t sx126x_get_status(void) { diff --git a/firmware/main/sx126x.h b/firmware/main/sx126x.h index 217febc..ed6af05 100644 --- a/firmware/main/sx126x.h +++ b/firmware/main/sx126x.h @@ -49,6 +49,10 @@ int sx126x_read_rx(uint8_t *out, size_t maxlen, int8_t *rssi_dbm); int8_t sx126x_get_rssi_inst(void); uint16_t sx126x_get_device_errors(void); uint8_t sx126x_get_status(void); +// Raw command exchange for diagnostics: sends op + n_tx arg bytes, captures +// the full-duplex MISO bytes into rx (same length). Returns false on error. +bool sx126x_raw(uint8_t op, const uint8_t *args, size_t n_tx, + uint8_t *rx, size_t rx_len); // FEM control (exposed for diagnostics) void fem_set_tx(bool tx);