From 2a473e1b986ec7a3dbe86294b3913f92070806a8 Mon Sep 17 00:00:00 2001 From: lars Date: Mon, 3 Aug 2026 17:24:51 +0000 Subject: [PATCH] Update prototyping-session.md --- prototyping-session.md | 180 ++++++++++++++++++++++++----------------- 1 file changed, 107 insertions(+), 73 deletions(-) diff --git a/prototyping-session.md b/prototyping-session.md index 7c36dc3..f561ca9 100644 --- a/prototyping-session.md +++ b/prototyping-session.md @@ -1,92 +1,126 @@ -# Prototyping Session: Pose → Splat Rendering +# Prototyping Sessions: React Native Navigation Prototype ## Context -Part of the cruise ship AR indoor navigation project. This session isolates and de-risks the **rendering half** of the pipeline (position + direction vector → correct camera view in a Gaussian Splat), independent of the indoor positioning problem. Indoor positioning accuracy is a separate, harder problem — here we assume we *have* a pose and just need to prove we can turn it into a convincing live AR view. +Part of the cruise ship AR indoor navigation project. Target platform is **React Native** (not a pure web app) — chiefly because BLE beacon positioning needs native Bluetooth access, which Safari/iOS doesn't expose (no Web Bluetooth on iOS). Splat rendering itself is fine on the web, so the plan reuses a web-based renderer inside a WebView rather than rebuilding it natively. -## Core Hypothesis -Given a 3D position vector and a direction (viewing) vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, at a frame rate and visual fidelity good enough to feel like "AR navigation" rather than a static 3D viewer. +Two sessions, can run back to back or independently: +1. **Pose → Splat Rendering** (RN + WebView) — validates the rendering half +2. **Native BLE Beacon Positioning** — validates the positioning half -## Goal for the Session -By the end, have a working demo: move a virtual camera through a splat of a real room/corridor using manually or programmatically fed (position, direction) pairs, and confirm the rendered view matches what a person standing there and looking that way would actually see. +They're deliberately decoupled: Session 1 assumes pose is given manually; Session 2 assumes rendering isn't built yet. They meet in a later integration session once both are proven. -## In Scope -- Defining the pose representation (position vector + direction vector, or quaternion) -- Capturing or reusing one small real-world splat (a room, hallway, or stairwell — not the whole ship) -- Aligning the splat's internal coordinate system to a real-world/metric coordinate system -- Feeding pose data into a splat viewer/renderer and rendering the correct view -- Validating rendered view vs. reality at a few reference points -- Simulating a "live navigation" feel by animating/interpolating pose along a path +--- -## Out of Scope (for this session) -- Actual indoor positioning tech (assume pose is given/manually set) -- Full ship-scale splat capture -- Route computation / pathfinding -- Production mobile app integration -- Gyroscope fusion (optional stretch only, see below) +## Session 1: Pose → Splat Rendering (React Native) -## Pose Representation -Decide and document early — this is the contract between "positioning" and "rendering": -- **Position vector**: `(x, y, z)` in a fixed world/ship coordinate frame, metric units -- **Direction vector**: either - - a normalized forward vector `(dx, dy, dz)`, or - - yaw/pitch (+ fixed roll) if you want to lean on the phone's gyroscope later, or - - a full quaternion if you need roll too -- Decide now whether "up" is world-up or splat-native up (splats often come with an arbitrary up axis — needs alignment) +### Core Hypothesis +Given a position vector and direction vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, inside a React Native app, at a fidelity/frame rate good enough to feel like AR navigation. -## Tooling Options to Test -- **Splat capture**: Polycam, Luma AI, or a quick Postshot/nerfstudio run from a phone video of a real room -- **Splat rendering/viewer**: - - Web: three.js gsplat viewers (e.g. antimatter15 splat viewer, `@mkkellogg/GaussianSplats3D`, or Luma's web SDK) — easiest to script custom camera poses into - - Engine: Unity/Unreal with a Gaussian Splatting plugin, if you want mobile deployment sooner -- Pick web-based first for this session — fastest to iterate on "set camera pose programmatically." +### Architecture Decision +Two options — pick the first for this session, note the second as a later optimization: -## Session Agenda (half-day, ~4 hrs) +- **A. WebView-hosted renderer (recommended for prototyping)**: run a three.js/gsplat.js splat viewer inside `react-native-webview`, drive the camera pose from RN JS via `injectJavaScript` / `postMessage` bridge. Fastest to build, reuses any web prototyping already done, easy to iterate on shaders/rendering. +- **B. Native renderer**: a native module wrapping a Metal (iOS) / Vulkan or OpenGL (Android) Gaussian splat renderer, exposed to RN via a native bridge. Better performance ceiling, no WebView overhead — but much more setup work. Not worth it until Session 1(A) proves the concept. -**1. Setup (30 min)** -- Pick/capture one small real space (a room or corridor you can physically walk into) -- Get it into a splat viewer that accepts programmatic camera pose input +### In Scope +- Pose representation: `(x, y, z)` position + normalized direction vector (or yaw/pitch), passed from RN state into the WebView +- One small real-world splat (room/corridor), coordinate-aligned to a real-world metric frame +- RN ↔ WebView bridge for feeding pose data every frame +- Native device orientation (gyroscope) feeding the direction vector, via `expo-sensors` or `react-native-sensors` — not the browser's `DeviceOrientationEvent` +- Validation of rendered view vs. real photo (via `expo-camera` / `react-native-vision-camera`) +- Simulated "live navigation" by animating pose along a path -**2. Coordinate alignment (45 min)** -- Establish 2–3 known reference points in the real space (measured with tape measure/laser) and their pixel/splat coordinates -- Compute the transform (scale, rotation, offset) between splat space and real-world metric space -- Sanity check: does a known real position map to the expected point in the splat? +### Out of Scope +- Actual BLE-derived position (see Session 2) +- Native (non-WebView) renderer +- Full ship-scale splat +- Route computation -**3. Pose → render pipeline (45 min)** -- Write the small script/function: `renderView(position, direction) → camera frame` -- Manually feed in a handful of (position, direction) pairs corresponding to places you can physically stand -- Render and screenshot each +### Tooling +| Need | Library | +|---|---| +| WebView host | `react-native-webview` | +| Splat rendering (inside WebView) | three.js + a gsplat viewer (e.g. `@mkkellogg/GaussianSplats3D`), or `gsplat.js` | +| Device orientation | `expo-sensors` (`DeviceMotion`) or `react-native-sensors` | +| Camera (for validation photos) | `expo-camera` or `react-native-vision-camera` | +| RN ↔ WebView messaging | `webviewRef.injectJavaScript()` out, `window.ReactNativeWebView.postMessage()` back | -**4. Validation (45 min)** -- Stand in the real space at those same points, facing the same direction, take a phone photo -- Compare rendered splat view vs. real photo side by side -- Note: does it match well enough? Where does it break (occlusion, splat artifacts, alignment drift)? +### Session Agenda (~4 hrs) +**1. Setup (30 min)** — scaffold RN app (Expo recommended for speed), embed WebView with a working splat viewer loading your test splat. -**5. Fake "live" navigation (45 min)** -- Define a short path as a sequence of (position, direction) waypoints -- Interpolate between them (linear position, slerp direction) and animate the camera through the splat -- Does it feel smooth and directionally correct, like walking through the space? +**2. Coordinate alignment (45 min)** — same as before: measure 2–3 real-world reference points, compute the transform between splat space and real-world metric space. -**6. Stretch goal (remaining time, optional)** -- Pipe live phone gyroscope data into the direction vector while position stays fixed/manual -- Test whether looking around live in the splat feels responsive enough +**3. Pose bridge (45 min)** — build the RN → WebView pose channel: RN holds `{position, direction}` in state, pushes it into the WebView on each update via `injectJavaScript`, WebView-side JS sets the three.js camera accordingly. Confirm round-trip latency is low enough to feel live. -**7. Wrap-up (15–30 min)** -- Debrief: what worked, what didn't, what's the biggest open risk -- Capture screenshots/video of best result as a demo artifact +**4. Validation (45 min)** — manually set poses matching physical spots you can stand in; compare rendered screenshots to real photos taken with `expo-camera` at the same spot/orientation. -## Success Criteria -- Rendered splat view at a given pose is recognizably the same as the real view (same landmarks, correct orientation) to a human comparing them -- Camera can be moved through a sequence of poses smoothly enough to read as "navigation," not a slideshow -- Coordinate alignment error is small enough to be usable at ~1m positioning accuracy (i.e. rendering error doesn't dominate over positioning error) +**5. Fake "live" navigation (45 min)** — animate through a sequence of waypoints (interpolated position, slerped direction) to simulate walking a route. -## Key Risks / Open Questions to Surface -- How much manual calibration does each splat capture need before poses map correctly? (Ship-wide, this needs to scale — can't hand-align every room.) -- Splat rendering performance on-device (mobile GPU) at usable frame rates -- How splat quality degrades in low-light, mirrored/repetitive ship corridors (relevant to capture quality later, not this session) -- Whether direction vector alone is enough, or roll/tilt from gyroscope is needed for a convincing AR feel +**6. Gyroscope stretch (remaining time)** — wire `expo-sensors` device orientation into the direction vector so looking around live with the phone updates the splat view. -## Deliverables -- Short demo video: real photo vs. rendered splat view at 3–4 poses -- Video of interpolated "walk" through the splat along a path -- One-page write-up of the coordinate alignment method and error observed -- List of blockers/questions for the next session (likely: scaling capture+alignment to full ship, and integrating with actual positioning data) \ No newline at end of file +**7. Wrap-up (15–30 min)** — debrief, capture demo video, note WebView performance ceiling (frame drops, memory) as a data point for whether Option B (native renderer) becomes necessary later. + +### Success Criteria +- Rendered view at a given pose visually matches the real view at that spot/orientation +- Pose updates feel responsive enough (no visible lag) driving the camera through the WebView bridge +- Clear read on whether WebView performance is sufficient or whether a native renderer will be needed before ship-wide rollout + +--- + +## Session 2: Native BLE Beacon Positioning (React Native) + +### Core Hypothesis +Native BLE scanning in a React Native app can detect nearby beacons and estimate a passenger's position to a useful accuracy (~1–3m as a first-pass target; ~1m is the eventual goal but likely needs fingerprinting/fusion beyond this session). + +### Why Native (Not Web) +Web Bluetooth isn't available on Safari/iOS, and even on Android its `requestDevice()`-plus-GATT model isn't well suited to continuously scanning many beacons' advertisement packets at once. Native BLE modules (backed by CoreBluetooth on iOS, `BluetoothLeScanner` on Android) don't have either limitation. + +### In Scope +- Scanning for BLE beacon advertisements (iBeacon or Eddystone format) continuously on both iOS and Android +- Reading RSSI per beacon +- A first-pass position estimate from RSSI (simple trilateration or nearest-beacon heuristic — not full fingerprinting yet) +- Testing signal behavior in a real indoor space with obstructions (walls, people) as a proxy for ship conditions + +### Out of Scope +- Full RSSI fingerprinting map of a space +- Sensor fusion with IMU/dead-reckoning +- Production-grade filtering (Kalman/particle filters) — note as future work +- Splat rendering integration (that's Session 1 + a later integration session) + +### Tooling +| Need | Library | +|---|---| +| BLE scanning | `react-native-ble-plx` (mature, cross-platform) or `react-native-ble-manager` | +| Beacon-specific (iBeacon/Eddystone parsing) | `react-native-beacons-manager` (wraps CoreLocation/AltBeacon) if you want OS-level beacon ranging instead of raw BLE scanning | +| Test beacons | A handful of cheap iBeacon/Eddystone hardware beacons (e.g. Estimote, Kontakt.io, or generic ones) | + +Note: `react-native-beacons-manager` uses iOS's native CLBeaconRegion ranging (via CoreLocation) rather than raw BLE scans — often more stable for iBeacon-format positioning than manual RSSI parsing, worth trying first since it's less code. + +### Session Agenda (~4 hrs) +**1. Setup (30 min)** — scaffold BLE scanning in the RN app, confirm permissions flow works on both iOS (Bluetooth + Location permission prompts) and Android (Location permission, and Nearby Devices permission on Android 12+). + +**2. Basic detection (45 min)** — place 3–4 beacons around a real room/corridor, confirm the app reliably detects all of them and logs RSSI over time. + +**3. RSSI behavior characterization (45 min)** — walk around the space, log RSSI vs. known distance at several points. Note noise, variance, and how much it degrades with obstructions (a person standing between phone and beacon, walls). + +**4. First-pass position estimate (60 min)** — implement a simple estimate (nearest-beacon-wins, or basic trilateration from 3+ beacons) and compare estimated position to actual measured position at several test points. + +**5. Cross-platform check (30 min)** — repeat a subset of the above on both an iOS and an Android device; note any platform-specific quirks (scan intervals, background restrictions, permission differences). + +**6. Wrap-up (15–30 min)** — debrief: current accuracy vs. the ~1m target, what's driving the gap (beacon density? RSSI noise? algorithm?), and whether fingerprinting or sensor fusion is the likely next step. + +### Success Criteria +- Reliable beacon detection on both iOS and Android with no missed beacons in range +- A documented RSSI-to-distance relationship (even if noisy) for this beacon hardware/environment +- A first accuracy number for the naive position estimate, to set a baseline before investing in fingerprinting or fusion + +### Key Risks / Open Questions +- iOS background BLE scanning restrictions — may limit how "always on" positioning can be +- Beacon density needed for ~1m accuracy at ship scale (cost/logistics implication) +- Metal ship structure's effect on BLE signal propagation (multipath, attenuation) vs. this session's test environment +- Whether iBeacon ranging (CoreLocation-based) or raw RSSI scanning gives more stable results + +--- + +## Integration Session (Future, Not This Round) +Once Session 1 and Session 2 are each independently validated, a follow-up session should feed Session 2's live position estimate as Session 1's position vector, and test the two together in the same real space. Keeping them separate now avoids debugging two unproven systems at once. \ No newline at end of file