splat-indoor-navigation/prototyping-session.md

# Prototyping Sessions: React Native Navigation Prototype

## Context
Part of the cruise ship AR indoor navigation project. Target platform is **React Native** (not a pure web app) — chiefly because BLE beacon positioning needs native Bluetooth access, which Safari/iOS doesn't expose (no Web Bluetooth on iOS). Splat rendering itself is fine on the web, so the plan reuses a web-based renderer inside a WebView rather than rebuilding it natively.

Two sessions, can run back to back or independently:
1. **Pose → Splat Rendering** (RN + WebView) — validates the rendering half
2. **Native BLE Beacon Positioning** — validates the positioning half

They're deliberately decoupled: Session 1 assumes pose is given manually; Session 2 assumes rendering isn't built yet. They meet in a later integration session once both are proven.

---

## Session 1: Pose → Splat Rendering (React Native)

### Core Hypothesis
Given a position vector and direction vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, inside a React Native app, at a fidelity/frame rate good enough to feel like AR navigation.

### Architecture Decision
Two options — pick the first for this session, note the second as a later optimization:

- **A. WebView-hosted renderer (recommended for prototyping)**: run a three.js/gsplat.js splat viewer inside `react-native-webview`, drive the camera pose from RN JS via `injectJavaScript` / `postMessage` bridge. Fastest to build, reuses any web prototyping already done, easy to iterate on shaders/rendering.
- **B. Native renderer**: a native module wrapping a Metal (iOS) / Vulkan or OpenGL (Android) Gaussian splat renderer, exposed to RN via a native bridge. Better performance ceiling, no WebView overhead — but much more setup work. Not worth it until Session 1(A) proves the concept.

### In Scope
- Pose representation: `(x, y, z)` position + normalized direction vector (or yaw/pitch), passed from RN state into the WebView
- One small real-world splat (room/corridor), coordinate-aligned to a real-world metric frame
- RN ↔ WebView bridge for feeding pose data every frame
- Native device orientation (gyroscope) feeding the direction vector, via `expo-sensors` or `react-native-sensors` — not the browser's `DeviceOrientationEvent`
- Validation of rendered view vs. real photo (via `expo-camera` / `react-native-vision-camera`)
- Simulated "live navigation" by animating pose along a path

### Out of Scope
- Actual BLE-derived position (see Session 2)
- Native (non-WebView) renderer
- Full ship-scale splat
- Route computation

### Tooling
| Need | Library |
|---|---|
| WebView host | `react-native-webview` |
| Splat rendering (inside WebView) | three.js + a gsplat viewer (e.g. `@mkkellogg/GaussianSplats3D`), or `gsplat.js` |
| Device orientation | `expo-sensors` (`DeviceMotion`) or `react-native-sensors` |
| Camera (for validation photos) | `expo-camera` or `react-native-vision-camera` |
| RN ↔ WebView messaging | `webviewRef.injectJavaScript()` out, `window.ReactNativeWebView.postMessage()` back |

### Session Agenda (~4 hrs)
**1. Setup (30 min)** — scaffold RN app (Expo recommended for speed), embed WebView with a working splat viewer loading your test splat.

**2. Coordinate alignment (45 min)** — same as before: measure 2–3 real-world reference points, compute the transform between splat space and real-world metric space.

**3. Pose bridge (45 min)** — build the RN → WebView pose channel: RN holds `{position, direction}` in state, pushes it into the WebView on each update via `injectJavaScript`, WebView-side JS sets the three.js camera accordingly. Confirm round-trip latency is low enough to feel live.

**4. Validation (45 min)** — manually set poses matching physical spots you can stand in; compare rendered screenshots to real photos taken with `expo-camera` at the same spot/orientation.

**5. Fake "live" navigation (45 min)** — animate through a sequence of waypoints (interpolated position, slerped direction) to simulate walking a route.

**6. Gyroscope stretch (remaining time)** — wire `expo-sensors` device orientation into the direction vector so looking around live with the phone updates the splat view.

**7. Wrap-up (15–30 min)** — debrief, capture demo video, note WebView performance ceiling (frame drops, memory) as a data point for whether Option B (native renderer) becomes necessary later.

### Success Criteria
- Rendered view at a given pose visually matches the real view at that spot/orientation
- Pose updates feel responsive enough (no visible lag) driving the camera through the WebView bridge
- Clear read on whether WebView performance is sufficient or whether a native renderer will be needed before ship-wide rollout

---

## Session 2: Native BLE Beacon Positioning (React Native)

### Core Hypothesis
Native BLE scanning in a React Native app can detect nearby beacons and estimate a passenger's position to a useful accuracy (~1–3m as a first-pass target; ~1m is the eventual goal but likely needs fingerprinting/fusion beyond this session).

### Why Native (Not Web)
Web Bluetooth isn't available on Safari/iOS, and even on Android its `requestDevice()`-plus-GATT model isn't well suited to continuously scanning many beacons' advertisement packets at once. Native BLE modules (backed by CoreBluetooth on iOS, `BluetoothLeScanner` on Android) don't have either limitation.

### In Scope
- Scanning for BLE beacon advertisements (iBeacon or Eddystone format) continuously on both iOS and Android
- Reading RSSI per beacon
- A first-pass position estimate from RSSI (simple trilateration or nearest-beacon heuristic — not full fingerprinting yet)
- Testing signal behavior in a real indoor space with obstructions (walls, people) as a proxy for ship conditions

### Out of Scope
- Full RSSI fingerprinting map of a space
- Sensor fusion with IMU/dead-reckoning
- Production-grade filtering (Kalman/particle filters) — note as future work
- Splat rendering integration (that's Session 1 + a later integration session)

### Tooling
| Need | Library |
|---|---|
| BLE scanning | `react-native-ble-plx` (mature, cross-platform) or `react-native-ble-manager` |
| Beacon-specific (iBeacon/Eddystone parsing) | `react-native-beacons-manager` (wraps CoreLocation/AltBeacon) if you want OS-level beacon ranging instead of raw BLE scanning |
| Test beacons | A handful of cheap iBeacon/Eddystone hardware beacons (e.g. Estimote, Kontakt.io, or generic ones) |

Note: `react-native-beacons-manager` uses iOS's native CLBeaconRegion ranging (via CoreLocation) rather than raw BLE scans — often more stable for iBeacon-format positioning than manual RSSI parsing, worth trying first since it's less code.

### Session Agenda (~4 hrs)
**1. Setup (30 min)** — scaffold BLE scanning in the RN app, confirm permissions flow works on both iOS (Bluetooth + Location permission prompts) and Android (Location permission, and Nearby Devices permission on Android 12+).

**2. Basic detection (45 min)** — place 3–4 beacons around a real room/corridor, confirm the app reliably detects all of them and logs RSSI over time.

**3. RSSI behavior characterization (45 min)** — walk around the space, log RSSI vs. known distance at several points. Note noise, variance, and how much it degrades with obstructions (a person standing between phone and beacon, walls).

**4. First-pass position estimate (60 min)** — implement a simple estimate (nearest-beacon-wins, or basic trilateration from 3+ beacons) and compare estimated position to actual measured position at several test points.

**5. Cross-platform check (30 min)** — repeat a subset of the above on both an iOS and an Android device; note any platform-specific quirks (scan intervals, background restrictions, permission differences).

**6. Wrap-up (15–30 min)** — debrief: current accuracy vs. the ~1m target, what's driving the gap (beacon density? RSSI noise? algorithm?), and whether fingerprinting or sensor fusion is the likely next step.

### Success Criteria
- Reliable beacon detection on both iOS and Android with no missed beacons in range
- A documented RSSI-to-distance relationship (even if noisy) for this beacon hardware/environment
- A first accuracy number for the naive position estimate, to set a baseline before investing in fingerprinting or fusion

### Key Risks / Open Questions
- iOS background BLE scanning restrictions — may limit how "always on" positioning can be
- Beacon density needed for ~1m accuracy at ship scale (cost/logistics implication)
- Metal ship structure's effect on BLE signal propagation (multipath, attenuation) vs. this session's test environment
- Whether iBeacon ranging (CoreLocation-based) or raw RSSI scanning gives more stable results

---

## Integration Session (Future, Not This Round)
Once Session 1 and Session 2 are each independently validated, a follow-up session should feed Session 2's live position estimate as Session 1's position vector, and test the two together in the same real space. Keeping them separate now avoids debugging two unproven systems at once.