Update prototyping-session.md
This commit is contained in:
+107
-73
@@ -1,92 +1,126 @@
|
|||||||
# Prototyping Session: Pose → Splat Rendering
|
# Prototyping Sessions: React Native Navigation Prototype
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
Part of the cruise ship AR indoor navigation project. This session isolates and de-risks the **rendering half** of the pipeline (position + direction vector → correct camera view in a Gaussian Splat), independent of the indoor positioning problem. Indoor positioning accuracy is a separate, harder problem — here we assume we *have* a pose and just need to prove we can turn it into a convincing live AR view.
|
Part of the cruise ship AR indoor navigation project. Target platform is **React Native** (not a pure web app) — chiefly because BLE beacon positioning needs native Bluetooth access, which Safari/iOS doesn't expose (no Web Bluetooth on iOS). Splat rendering itself is fine on the web, so the plan reuses a web-based renderer inside a WebView rather than rebuilding it natively.
|
||||||
|
|
||||||
## Core Hypothesis
|
Two sessions, can run back to back or independently:
|
||||||
Given a 3D position vector and a direction (viewing) vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, at a frame rate and visual fidelity good enough to feel like "AR navigation" rather than a static 3D viewer.
|
1. **Pose → Splat Rendering** (RN + WebView) — validates the rendering half
|
||||||
|
2. **Native BLE Beacon Positioning** — validates the positioning half
|
||||||
|
|
||||||
## Goal for the Session
|
They're deliberately decoupled: Session 1 assumes pose is given manually; Session 2 assumes rendering isn't built yet. They meet in a later integration session once both are proven.
|
||||||
By the end, have a working demo: move a virtual camera through a splat of a real room/corridor using manually or programmatically fed (position, direction) pairs, and confirm the rendered view matches what a person standing there and looking that way would actually see.
|
|
||||||
|
|
||||||
## In Scope
|
---
|
||||||
- Defining the pose representation (position vector + direction vector, or quaternion)
|
|
||||||
- Capturing or reusing one small real-world splat (a room, hallway, or stairwell — not the whole ship)
|
|
||||||
- Aligning the splat's internal coordinate system to a real-world/metric coordinate system
|
|
||||||
- Feeding pose data into a splat viewer/renderer and rendering the correct view
|
|
||||||
- Validating rendered view vs. reality at a few reference points
|
|
||||||
- Simulating a "live navigation" feel by animating/interpolating pose along a path
|
|
||||||
|
|
||||||
## Out of Scope (for this session)
|
## Session 1: Pose → Splat Rendering (React Native)
|
||||||
- Actual indoor positioning tech (assume pose is given/manually set)
|
|
||||||
- Full ship-scale splat capture
|
|
||||||
- Route computation / pathfinding
|
|
||||||
- Production mobile app integration
|
|
||||||
- Gyroscope fusion (optional stretch only, see below)
|
|
||||||
|
|
||||||
## Pose Representation
|
### Core Hypothesis
|
||||||
Decide and document early — this is the contract between "positioning" and "rendering":
|
Given a position vector and direction vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, inside a React Native app, at a fidelity/frame rate good enough to feel like AR navigation.
|
||||||
- **Position vector**: `(x, y, z)` in a fixed world/ship coordinate frame, metric units
|
|
||||||
- **Direction vector**: either
|
|
||||||
- a normalized forward vector `(dx, dy, dz)`, or
|
|
||||||
- yaw/pitch (+ fixed roll) if you want to lean on the phone's gyroscope later, or
|
|
||||||
- a full quaternion if you need roll too
|
|
||||||
- Decide now whether "up" is world-up or splat-native up (splats often come with an arbitrary up axis — needs alignment)
|
|
||||||
|
|
||||||
## Tooling Options to Test
|
### Architecture Decision
|
||||||
- **Splat capture**: Polycam, Luma AI, or a quick Postshot/nerfstudio run from a phone video of a real room
|
Two options — pick the first for this session, note the second as a later optimization:
|
||||||
- **Splat rendering/viewer**:
|
|
||||||
- Web: three.js gsplat viewers (e.g. antimatter15 splat viewer, `@mkkellogg/GaussianSplats3D`, or Luma's web SDK) — easiest to script custom camera poses into
|
|
||||||
- Engine: Unity/Unreal with a Gaussian Splatting plugin, if you want mobile deployment sooner
|
|
||||||
- Pick web-based first for this session — fastest to iterate on "set camera pose programmatically."
|
|
||||||
|
|
||||||
## Session Agenda (half-day, ~4 hrs)
|
- **A. WebView-hosted renderer (recommended for prototyping)**: run a three.js/gsplat.js splat viewer inside `react-native-webview`, drive the camera pose from RN JS via `injectJavaScript` / `postMessage` bridge. Fastest to build, reuses any web prototyping already done, easy to iterate on shaders/rendering.
|
||||||
|
- **B. Native renderer**: a native module wrapping a Metal (iOS) / Vulkan or OpenGL (Android) Gaussian splat renderer, exposed to RN via a native bridge. Better performance ceiling, no WebView overhead — but much more setup work. Not worth it until Session 1(A) proves the concept.
|
||||||
|
|
||||||
**1. Setup (30 min)**
|
### In Scope
|
||||||
- Pick/capture one small real space (a room or corridor you can physically walk into)
|
- Pose representation: `(x, y, z)` position + normalized direction vector (or yaw/pitch), passed from RN state into the WebView
|
||||||
- Get it into a splat viewer that accepts programmatic camera pose input
|
- One small real-world splat (room/corridor), coordinate-aligned to a real-world metric frame
|
||||||
|
- RN ↔ WebView bridge for feeding pose data every frame
|
||||||
|
- Native device orientation (gyroscope) feeding the direction vector, via `expo-sensors` or `react-native-sensors` — not the browser's `DeviceOrientationEvent`
|
||||||
|
- Validation of rendered view vs. real photo (via `expo-camera` / `react-native-vision-camera`)
|
||||||
|
- Simulated "live navigation" by animating pose along a path
|
||||||
|
|
||||||
**2. Coordinate alignment (45 min)**
|
### Out of Scope
|
||||||
- Establish 2–3 known reference points in the real space (measured with tape measure/laser) and their pixel/splat coordinates
|
- Actual BLE-derived position (see Session 2)
|
||||||
- Compute the transform (scale, rotation, offset) between splat space and real-world metric space
|
- Native (non-WebView) renderer
|
||||||
- Sanity check: does a known real position map to the expected point in the splat?
|
- Full ship-scale splat
|
||||||
|
- Route computation
|
||||||
|
|
||||||
**3. Pose → render pipeline (45 min)**
|
### Tooling
|
||||||
- Write the small script/function: `renderView(position, direction) → camera frame`
|
| Need | Library |
|
||||||
- Manually feed in a handful of (position, direction) pairs corresponding to places you can physically stand
|
|---|---|
|
||||||
- Render and screenshot each
|
| WebView host | `react-native-webview` |
|
||||||
|
| Splat rendering (inside WebView) | three.js + a gsplat viewer (e.g. `@mkkellogg/GaussianSplats3D`), or `gsplat.js` |
|
||||||
|
| Device orientation | `expo-sensors` (`DeviceMotion`) or `react-native-sensors` |
|
||||||
|
| Camera (for validation photos) | `expo-camera` or `react-native-vision-camera` |
|
||||||
|
| RN ↔ WebView messaging | `webviewRef.injectJavaScript()` out, `window.ReactNativeWebView.postMessage()` back |
|
||||||
|
|
||||||
**4. Validation (45 min)**
|
### Session Agenda (~4 hrs)
|
||||||
- Stand in the real space at those same points, facing the same direction, take a phone photo
|
**1. Setup (30 min)** — scaffold RN app (Expo recommended for speed), embed WebView with a working splat viewer loading your test splat.
|
||||||
- Compare rendered splat view vs. real photo side by side
|
|
||||||
- Note: does it match well enough? Where does it break (occlusion, splat artifacts, alignment drift)?
|
|
||||||
|
|
||||||
**5. Fake "live" navigation (45 min)**
|
**2. Coordinate alignment (45 min)** — same as before: measure 2–3 real-world reference points, compute the transform between splat space and real-world metric space.
|
||||||
- Define a short path as a sequence of (position, direction) waypoints
|
|
||||||
- Interpolate between them (linear position, slerp direction) and animate the camera through the splat
|
|
||||||
- Does it feel smooth and directionally correct, like walking through the space?
|
|
||||||
|
|
||||||
**6. Stretch goal (remaining time, optional)**
|
**3. Pose bridge (45 min)** — build the RN → WebView pose channel: RN holds `{position, direction}` in state, pushes it into the WebView on each update via `injectJavaScript`, WebView-side JS sets the three.js camera accordingly. Confirm round-trip latency is low enough to feel live.
|
||||||
- Pipe live phone gyroscope data into the direction vector while position stays fixed/manual
|
|
||||||
- Test whether looking around live in the splat feels responsive enough
|
|
||||||
|
|
||||||
**7. Wrap-up (15–30 min)**
|
**4. Validation (45 min)** — manually set poses matching physical spots you can stand in; compare rendered screenshots to real photos taken with `expo-camera` at the same spot/orientation.
|
||||||
- Debrief: what worked, what didn't, what's the biggest open risk
|
|
||||||
- Capture screenshots/video of best result as a demo artifact
|
|
||||||
|
|
||||||
## Success Criteria
|
**5. Fake "live" navigation (45 min)** — animate through a sequence of waypoints (interpolated position, slerped direction) to simulate walking a route.
|
||||||
- Rendered splat view at a given pose is recognizably the same as the real view (same landmarks, correct orientation) to a human comparing them
|
|
||||||
- Camera can be moved through a sequence of poses smoothly enough to read as "navigation," not a slideshow
|
|
||||||
- Coordinate alignment error is small enough to be usable at ~1m positioning accuracy (i.e. rendering error doesn't dominate over positioning error)
|
|
||||||
|
|
||||||
## Key Risks / Open Questions to Surface
|
**6. Gyroscope stretch (remaining time)** — wire `expo-sensors` device orientation into the direction vector so looking around live with the phone updates the splat view.
|
||||||
- How much manual calibration does each splat capture need before poses map correctly? (Ship-wide, this needs to scale — can't hand-align every room.)
|
|
||||||
- Splat rendering performance on-device (mobile GPU) at usable frame rates
|
|
||||||
- How splat quality degrades in low-light, mirrored/repetitive ship corridors (relevant to capture quality later, not this session)
|
|
||||||
- Whether direction vector alone is enough, or roll/tilt from gyroscope is needed for a convincing AR feel
|
|
||||||
|
|
||||||
## Deliverables
|
**7. Wrap-up (15–30 min)** — debrief, capture demo video, note WebView performance ceiling (frame drops, memory) as a data point for whether Option B (native renderer) becomes necessary later.
|
||||||
- Short demo video: real photo vs. rendered splat view at 3–4 poses
|
|
||||||
- Video of interpolated "walk" through the splat along a path
|
### Success Criteria
|
||||||
- One-page write-up of the coordinate alignment method and error observed
|
- Rendered view at a given pose visually matches the real view at that spot/orientation
|
||||||
- List of blockers/questions for the next session (likely: scaling capture+alignment to full ship, and integrating with actual positioning data)
|
- Pose updates feel responsive enough (no visible lag) driving the camera through the WebView bridge
|
||||||
|
- Clear read on whether WebView performance is sufficient or whether a native renderer will be needed before ship-wide rollout
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session 2: Native BLE Beacon Positioning (React Native)
|
||||||
|
|
||||||
|
### Core Hypothesis
|
||||||
|
Native BLE scanning in a React Native app can detect nearby beacons and estimate a passenger's position to a useful accuracy (~1–3m as a first-pass target; ~1m is the eventual goal but likely needs fingerprinting/fusion beyond this session).
|
||||||
|
|
||||||
|
### Why Native (Not Web)
|
||||||
|
Web Bluetooth isn't available on Safari/iOS, and even on Android its `requestDevice()`-plus-GATT model isn't well suited to continuously scanning many beacons' advertisement packets at once. Native BLE modules (backed by CoreBluetooth on iOS, `BluetoothLeScanner` on Android) don't have either limitation.
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Scanning for BLE beacon advertisements (iBeacon or Eddystone format) continuously on both iOS and Android
|
||||||
|
- Reading RSSI per beacon
|
||||||
|
- A first-pass position estimate from RSSI (simple trilateration or nearest-beacon heuristic — not full fingerprinting yet)
|
||||||
|
- Testing signal behavior in a real indoor space with obstructions (walls, people) as a proxy for ship conditions
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Full RSSI fingerprinting map of a space
|
||||||
|
- Sensor fusion with IMU/dead-reckoning
|
||||||
|
- Production-grade filtering (Kalman/particle filters) — note as future work
|
||||||
|
- Splat rendering integration (that's Session 1 + a later integration session)
|
||||||
|
|
||||||
|
### Tooling
|
||||||
|
| Need | Library |
|
||||||
|
|---|---|
|
||||||
|
| BLE scanning | `react-native-ble-plx` (mature, cross-platform) or `react-native-ble-manager` |
|
||||||
|
| Beacon-specific (iBeacon/Eddystone parsing) | `react-native-beacons-manager` (wraps CoreLocation/AltBeacon) if you want OS-level beacon ranging instead of raw BLE scanning |
|
||||||
|
| Test beacons | A handful of cheap iBeacon/Eddystone hardware beacons (e.g. Estimote, Kontakt.io, or generic ones) |
|
||||||
|
|
||||||
|
Note: `react-native-beacons-manager` uses iOS's native CLBeaconRegion ranging (via CoreLocation) rather than raw BLE scans — often more stable for iBeacon-format positioning than manual RSSI parsing, worth trying first since it's less code.
|
||||||
|
|
||||||
|
### Session Agenda (~4 hrs)
|
||||||
|
**1. Setup (30 min)** — scaffold BLE scanning in the RN app, confirm permissions flow works on both iOS (Bluetooth + Location permission prompts) and Android (Location permission, and Nearby Devices permission on Android 12+).
|
||||||
|
|
||||||
|
**2. Basic detection (45 min)** — place 3–4 beacons around a real room/corridor, confirm the app reliably detects all of them and logs RSSI over time.
|
||||||
|
|
||||||
|
**3. RSSI behavior characterization (45 min)** — walk around the space, log RSSI vs. known distance at several points. Note noise, variance, and how much it degrades with obstructions (a person standing between phone and beacon, walls).
|
||||||
|
|
||||||
|
**4. First-pass position estimate (60 min)** — implement a simple estimate (nearest-beacon-wins, or basic trilateration from 3+ beacons) and compare estimated position to actual measured position at several test points.
|
||||||
|
|
||||||
|
**5. Cross-platform check (30 min)** — repeat a subset of the above on both an iOS and an Android device; note any platform-specific quirks (scan intervals, background restrictions, permission differences).
|
||||||
|
|
||||||
|
**6. Wrap-up (15–30 min)** — debrief: current accuracy vs. the ~1m target, what's driving the gap (beacon density? RSSI noise? algorithm?), and whether fingerprinting or sensor fusion is the likely next step.
|
||||||
|
|
||||||
|
### Success Criteria
|
||||||
|
- Reliable beacon detection on both iOS and Android with no missed beacons in range
|
||||||
|
- A documented RSSI-to-distance relationship (even if noisy) for this beacon hardware/environment
|
||||||
|
- A first accuracy number for the naive position estimate, to set a baseline before investing in fingerprinting or fusion
|
||||||
|
|
||||||
|
### Key Risks / Open Questions
|
||||||
|
- iOS background BLE scanning restrictions — may limit how "always on" positioning can be
|
||||||
|
- Beacon density needed for ~1m accuracy at ship scale (cost/logistics implication)
|
||||||
|
- Metal ship structure's effect on BLE signal propagation (multipath, attenuation) vs. this session's test environment
|
||||||
|
- Whether iBeacon ranging (CoreLocation-based) or raw RSSI scanning gives more stable results
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Integration Session (Future, Not This Round)
|
||||||
|
Once Session 1 and Session 2 are each independently validated, a follow-up session should feed Session 2's live position estimate as Session 1's position vector, and test the two together in the same real space. Keeping them separate now avoids debugging two unproven systems at once.
|
||||||
Reference in New Issue
Block a user