diff --git a/prototyping-session.md b/prototyping-session.md index f561ca9..8fefaab 100644 --- a/prototyping-session.md +++ b/prototyping-session.md @@ -123,4 +123,194 @@ Note: `react-native-beacons-manager` uses iOS's native CLBeaconRegion ranging (v --- ## Integration Session (Future, Not This Round) -Once Session 1 and Session 2 are each independently validated, a follow-up session should feed Session 2's live position estimate as Session 1's position vector, and test the two together in the same real space. Keeping them separate now avoids debugging two unproven systems at once. \ No newline at end of file +Once Session 1 and Session 2 are each independently validated, a follow-up session should feed Session 2's live position estimate as Session 1's position vector, and test the two together in the same real space. Keeping them separate now avoids debugging two unproven systems at once. + + + +## Fables more detailed response +# Prototyping Sessions: React Native Ship Navigation Prototype + +## Context +Part of the cruise ship AR indoor navigation project. Target platform is **React Native** — chiefly because BLE beacon positioning needs native Bluetooth access, which Safari/iOS doesn't expose (no Web Bluetooth on iOS). Splat rendering works well with web tech, so the plan reuses a web-based renderer inside a WebView rather than rebuilding it natively. + +Two sessions, deliberately decoupled: +1. **Pose → Splat Rendering** (RN + WebView) — validates the rendering half +2. **Native BLE Beacon Positioning** — validates the positioning half + +Session 1 assumes pose is given manually; Session 2 assumes rendering isn't built yet. They meet at a shared **pose contract** (below) and integrate in a later session once both are proven independently. This avoids debugging two unproven systems at once. + +--- + +## Shared Pose Contract (define before either session) +Both sessions build against this interface, so integration later is a plumbing task, not a redesign: + +```json +{ + "timestamp": 1722672000123, + "position": { "x": 12.40, "y": 3.10, "z": 0.00 }, + "heading": { "yaw": 87.5, "pitch": -4.0, "roll": 0.0 }, + "confidence": { "positionM": 1.8, "headingDeg": 10 } +} +``` + +Conventions to lock in now (cheap to decide, expensive to change): +- **Units**: meters, degrees, milliseconds since epoch +- **Frame**: right-handed, x = ship forward (bow), y = starboard, z = up; origin at a fixed, documented ship reference point +- **z / deck**: z is continuous meters, but deck number is the *useful* unit for navigation — carry both once decks matter (not in these sessions) +- **Heading**: yaw relative to ship frame (not magnetic north — see risk on magnetometers below) +- **Confidence**: every producer of pose reports its own uncertainty; the renderer can degrade gracefully (e.g. widen the "you are here" indicator) instead of pretending to be precise + +--- + +## Pre-Session Prep (do 1–2 weeks ahead — sessions die when setup eats the timebox) + +**Hardware / captures** +- [ ] Order 4–6 configurable BLE beacons (e.g. Kontakt.io, Minew, generic iBeacon) — lead time is the long pole; order first +- [ ] Configure beacons before the session: advertising interval to ~100 ms (default 700–1000 ms is far too slow for live positioning), max TX power, unique major/minor IDs +- [ ] Capture the test-space splat ahead of time (Polycam / Luma / Postshot from phone video) and confirm it loads in the chosen viewer — capture quality issues shouldn't burn session time +- [ ] One iOS + one Android test device, developer mode enabled, provisioning sorted + +**Software scaffolding** +- [ ] RN app scaffolded (Expo dev build — `expo-sensors` etc. work, but BLE needs a dev client, not Expo Go) +- [ ] `react-native-webview` embedded and loading a stock splat viewer with a sample splat +- [ ] BLE library (`react-native-ble-plx` and/or `react-native-beacons-manager`) installed with permissions declared (iOS: Bluetooth + Location usage strings; Android: Location + Nearby Devices for API 31+) and the permission prompts verified once on both platforms + +**Space & measurement** +- [ ] Book/claim the physical test space (one room or corridor) for both sessions +- [ ] Laser distance meter or tape measure; masking tape to mark measured reference points on the floor +- [ ] Measure and write down 4–6 reference points in the shared coordinate frame before the session + +**Roles (if 2–3 people)** +- Driver (writes code), Measurer (physical setup, ground-truth measurements, photos), Scribe (logs results, screenshots, decisions). One person can do it all, but it stretches the timeboxes. + +--- + +## Session 1: Pose → Splat Rendering (React Native) + +### Core Hypothesis +Given a pose (per the contract), we can render the matching camera view inside a pre-captured splat, live, inside an RN app, at a frame rate and latency that feels like AR navigation rather than a slideshow. + +### Architecture +- **A. WebView-hosted renderer (this session)**: three.js/gsplat viewer inside `react-native-webview`; RN pushes pose via `injectJavaScript`, WebView reports back via `postMessage`. +- **B. Native renderer (only if A fails the perf gate)**: Metal/OpenGL splat renderer behind a native module. Don't build until the data says you must. + +**Renderer choice inside the WebView: use a WebGL2-based viewer (e.g. `@mkkellogg/GaussianSplats3D`, gsplat.js), not a WebGPU-only one.** The app runs in WKWebView on iOS, and WebGPU availability inside embedded WebViews lags behind Safari proper and varies by OS version — verify before relying on it. WebGL2 works everywhere RN's WebView does and is plenty for a room-scale splat. Note WebGPU as the later upgrade path. + +### In Scope +- Pose contract → camera transform inside the viewer +- One room/corridor splat, aligned to the measured reference points +- RN ↔ WebView pose channel, with latency measured (not just eyeballed) +- Device orientation from `expo-sensors` (`DeviceMotion`) feeding yaw/pitch +- Rendered-vs-real validation photos +- Animated waypoint path ("fake navigation") + +### Out of Scope +- BLE-derived position (Session 2), native renderer, ship-scale splat, routing + +### Agenda (~4.5 hrs incl. buffer) + +**1. Baseline & instrumentation (30 min)** +Splat loads in the WebView on-device. Add an FPS counter and a latency probe (RN timestamps the pose send; WebView echoes it back on render; log the delta). Measuring from minute one turns "feels okay?" into numbers. + +**2. Coordinate alignment (45 min)** +Pick 3+ of the pre-measured reference points, find them in the splat, and solve the similarity transform (rotation + translation + uniform scale — Umeyama/Procrustes; a small JS function, no library needed). Verify with a 4th held-out point: predicted vs. actual splat location. Record the residual error in cm — this number matters for the ship-scale question later. +*Watch out:* splats often come with arbitrary up-axis and scale; gravity-align first if the tool supports it. + +**3. Pose bridge (45 min)** +`renderPose(pose)` end to end: RN state → inject → camera set → confirmation back. Throttle to display rate; batch to the latest pose (never queue stale poses). Target: **pose-to-photon under ~100 ms** at the bridge level. + +**4. Rendered vs. real validation (45 min)** +Stand at 4 reference points, face a documented direction (align phone edge to a taped line rather than guessing), photograph reality, screenshot the render at the same pose. Side-by-side comparison; note where and why they diverge (alignment residual vs. splat artifacts vs. heading error — different fixes for each). + +**5. Fake live navigation (30 min)** +Waypoint path through the space; lerp position, slerp heading; run at walking speed (~1.2 m/s). Judge: does it read as *moving through* the space? + +**6. Gyro-driven heading (30 min)** +`DeviceMotion` yaw/pitch → pose heading, position fixed. Physically turn in place; the splat view should track. +*Important:* device orientation gives *relative* rotation plus a magnetometer-derived absolute heading — and **magnetometers are unreliable near steel** (a ship is a giant magnet). Design for relative gyro heading + an explicit calibration step (e.g. user aligns phone with a known direction, or later: visual alignment), and note absolute-heading acquisition as an open problem for the real product. + +**7. Wrap-up (30 min)** +Debrief against the gates below; capture demo video; log FPS/latency/alignment numbers. + +### Success Gates (quantified) +| Metric | Pass | Investigate | Fail → fallback | +|---|---|---|---| +| Render frame rate (mid-tier phone) | ≥ 30 fps | 20–30 fps | < 20 fps → try smaller/compressed splat (.ksplat/SOG), reduced resolution; if still failing, schedule native-renderer spike | +| Pose-to-photon latency | < 100 ms | 100–200 ms | > 200 ms → profile bridge vs. render; consider moving pose loop into WebView | +| Alignment residual (held-out point) | < 15 cm | 15–50 cm | > 50 cm → alignment method is the problem, fix before anything else | +| Human check: rendered ≈ real | clearly same view | recognizably same, off in detail | not matching → stop, diagnose before proceeding | + +### Deliverables +- Demo video: real photo vs. render at 4 poses + the animated walk-through +- Numbers: FPS, latency, alignment residual +- One-pager: alignment method + a go/no-go read on WebView rendering + +--- + +## Session 2: Native BLE Beacon Positioning (React Native) + +### Core Hypothesis +Native BLE scanning in RN can continuously range multiple beacons and produce a position estimate with a characterized error — target **≤ 3 m median** for this naive first pass (the ~1 m product goal will need fingerprinting/fusion; this session establishes the baseline and the gap). + +### Why Native (Not Web) +No Web Bluetooth on iOS at all, and even on Android the `requestDevice()` + GATT model fits "pair one device," not "continuously read advertisements from many beacons." Native CoreBluetooth / `BluetoothLeScanner` (or CoreLocation beacon ranging) have neither limitation. + +### Approach: run two tracks in parallel and compare +- **Track A — OS-level beacon ranging** (`react-native-beacons-manager` → CLBeaconRegion on iOS): least code, OS-smoothed proximity estimates. Try first. +- **Track B — raw scan RSSI** (`react-native-ble-plx`): more control, needed if A's update rate or accuracy disappoints. Apply a **sliding median filter (~1–2 s window)** to RSSI before any distance math — raw RSSI is far too noisy to use directly, and the filter is ~10 lines. + +### In Scope +- Continuous multi-beacon detection on iOS and Android (foreground only) +- RSSI/distance characterization in a real space, including obstruction effects +- Naive position estimate (nearest-beacon zone, then weighted trilateration from 3+ beacons) emitting the **shared pose contract** (position + confidence; heading null) +- An honest accuracy number + +### Out of Scope +- Fingerprinting maps, IMU fusion, Kalman/particle filters, background scanning, rendering integration + +### Agenda (~4.5 hrs incl. buffer) + +**1. Detection baseline (30 min)** +3–4 beacons placed high (chest height or above, not floor — bodies attenuate 2.4 GHz badly), spread around the space at measured positions. Confirm both platforms see all beacons; log update rate per beacon (target ≥ 1 Hz per beacon — if lower, check beacon advertising interval and Android scan mode `SCAN_MODE_LOW_LATENCY`). + +**2. Calibrate TX power reference (30 min)** +Measure filtered RSSI at exactly 1 m from each beacon — this is the reference for any path-loss distance model, and per-beacon variance here is itself a finding. Don't trust the manufacturer's advertised value. + +**3. RSSI vs. distance characterization (45 min)** +Filtered RSSI at 1/2/4/8 m per beacon; repeat with a person standing in the line of sight and (if possible) near a large metal surface. Plot. The spread you see here bounds what any trilateration can achieve — capture it honestly. + +**4. Position estimate (60 min)** +Ship both: (a) nearest-beacon zone (robust, coarse — may honestly be enough for "you're near the aft elevator lobby" navigation), and (b) weighted trilateration from the 3 strongest filtered RSSIs. At 6+ marked ground-truth points, log estimated vs. true position, walking and standing. Report **median and p95 error** — a single "accuracy" number hides the tail, and the tail is what makes navigation feel broken. + +**5. Cross-platform + motion check (45 min)** +Repeat a subset on the other platform; note scan-rate and permission quirks. Then walk a loop at normal pace: how stale/jumpy is the estimate in motion? (Latency and jitter matter as much as static accuracy for live navigation.) + +**6. Wrap-up (30 min)** +Debrief against gates; decide the next positioning step with data. + +### Success Gates +| Metric | Pass | Investigate | Fail → fallback | +|---|---|---|---| +| Update rate per beacon | ≥ 1 Hz | 0.3–1 Hz | < 0.3 Hz → beacon config / scan mode issue; fix before judging accuracy | +| Static median error (trilateration) | ≤ 3 m | 3–5 m | > 5 m → RSSI-only trilateration insufficient here; pivot to zone-based UX + plan fingerprinting/fusion spike | +| Zone (nearest beacon) correctness | ≥ 90 % | 70–90 % | < 70 % → beacon placement/density problem | +| Estimate behavior while walking | tracks within ~1 beacon spacing | lags noticeably | jumps wildly → filtering window / update-rate work needed | + +### Deliverables +- RSSI-vs-distance plots (clear vs. obstructed), per-beacon 1 m calibration table +- Error table: median + p95, static + walking, both platforms, both estimators +- Recommendation memo: is naive BLE enough for zone-level nav, and what (fingerprinting, IMU fusion, more density) is the cheapest route toward ~1 m? + +### Key Risks / Open Questions +- **Steel environment**: this session's test space is not a ship. Multipath and attenuation on board will be worse; treat these results as an upper bound on BLE performance and plan an on-ship measurement day before committing to beacon density/cost figures. +- **Vertical/deck ambiguity**: BLE bleeds through decks; z-estimation may need per-deck beacon ID schemes rather than geometry. Not this session, but it shapes the beacon deployment plan. +- **iOS background scanning limits** — constrains "always on" positioning; may be fine if navigation is a foreground activity. +- **Magnetometer unreliability on steel** (shared with Session 1): heading will have to come from gyro + calibration or visual methods, not compass. + +--- + +## Integration Session (Future) +When both sessions pass their gates: feed Session 2's live pose (with its real confidence values) into Session 1's renderer in the same physical space. Success looks like walking the room with the phone and watching the splat view track you — jitter and all. The confidence field earns its keep here: render uncertainty honestly rather than snapping confidently to wrong positions. + +## Sequencing Note +Order beacons **now** (longest lead time). Session 1 has no hardware dependency and can run first. Prep checklist items are parallelizable across the team. \ No newline at end of file