Add prototyping-session.md

2026-08-03 17:14:38 +00:00
parent 3199cba73b
commit 02578e1c78
1 changed files with 92 additions and 0 deletions
@@ -0,0 +1,92 @@
 # Prototyping Session: Pose → Splat Rendering
 ## Context
 Part of the cruise ship AR indoor navigation project. This session isolates and de-risks the **rendering half** of the pipeline (position + direction vector → correct camera view in a Gaussian Splat), independent of the indoor positioning problem. Indoor positioning accuracy is a separate, harder problem — here we assume we *have* a pose and just need to prove we can turn it into a convincing live AR view.
 ## Core Hypothesis
 Given a 3D position vector and a direction (viewing) vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, at a frame rate and visual fidelity good enough to feel like "AR navigation" rather than a static 3D viewer.
 ## Goal for the Session
 By the end, have a working demo: move a virtual camera through a splat of a real room/corridor using manually or programmatically fed (position, direction) pairs, and confirm the rendered view matches what a person standing there and looking that way would actually see.
 ## In Scope
 - Defining the pose representation (position vector + direction vector, or quaternion)
 - Capturing or reusing one small real-world splat (a room, hallway, or stairwell — not the whole ship)
 - Aligning the splat's internal coordinate system to a real-world/metric coordinate system
 - Feeding pose data into a splat viewer/renderer and rendering the correct view
 - Validating rendered view vs. reality at a few reference points
 - Simulating a "live navigation" feel by animating/interpolating pose along a path
 ## Out of Scope (for this session)
 - Actual indoor positioning tech (assume pose is given/manually set)
 - Full ship-scale splat capture
 - Route computation / pathfinding
 - Production mobile app integration
 - Gyroscope fusion (optional stretch only, see below)
 ## Pose Representation
 Decide and document early — this is the contract between "positioning" and "rendering":
 - **Position vector**: `(x, y, z)` in a fixed world/ship coordinate frame, metric units
 - **Direction vector**: either
  - a normalized forward vector `(dx, dy, dz)`, or
  - yaw/pitch (+ fixed roll) if you want to lean on the phone's gyroscope later, or
  - a full quaternion if you need roll too
 - Decide now whether "up" is world-up or splat-native up (splats often come with an arbitrary up axis — needs alignment)
 ## Tooling Options to Test
 - **Splat capture**: Polycam, Luma AI, or a quick Postshot/nerfstudio run from a phone video of a real room
 - **Splat rendering/viewer**:
  - Web: three.js gsplat viewers (e.g. antimatter15 splat viewer, `@mkkellogg/GaussianSplats3D`, or Luma's web SDK) — easiest to script custom camera poses into
  - Engine: Unity/Unreal with a Gaussian Splatting plugin, if you want mobile deployment sooner
 - Pick web-based first for this session — fastest to iterate on "set camera pose programmatically."
 ## Session Agenda (half-day, ~4 hrs)
 **1. Setup (30 min)**
 - Pick/capture one small real space (a room or corridor you can physically walk into)
 - Get it into a splat viewer that accepts programmatic camera pose input
 **2. Coordinate alignment (45 min)**
 - Establish 2–3 known reference points in the real space (measured with tape measure/laser) and their pixel/splat coordinates
 - Compute the transform (scale, rotation, offset) between splat space and real-world metric space
 - Sanity check: does a known real position map to the expected point in the splat?
 **3. Pose → render pipeline (45 min)**
 - Write the small script/function: `renderView(position, direction) → camera frame`
 - Manually feed in a handful of (position, direction) pairs corresponding to places you can physically stand
 - Render and screenshot each
 **4. Validation (45 min)**
 - Stand in the real space at those same points, facing the same direction, take a phone photo
 - Compare rendered splat view vs. real photo side by side
 - Note: does it match well enough? Where does it break (occlusion, splat artifacts, alignment drift)?
 **5. Fake "live" navigation (45 min)**
 - Define a short path as a sequence of (position, direction) waypoints
 - Interpolate between them (linear position, slerp direction) and animate the camera through the splat
 - Does it feel smooth and directionally correct, like walking through the space?
 **6. Stretch goal (remaining time, optional)**
 - Pipe live phone gyroscope data into the direction vector while position stays fixed/manual
 - Test whether looking around live in the splat feels responsive enough
 **7. Wrap-up (15–30 min)**
 - Debrief: what worked, what didn't, what's the biggest open risk
 - Capture screenshots/video of best result as a demo artifact
 ## Success Criteria
 - Rendered splat view at a given pose is recognizably the same as the real view (same landmarks, correct orientation) to a human comparing them
 - Camera can be moved through a sequence of poses smoothly enough to read as "navigation," not a slideshow
 - Coordinate alignment error is small enough to be usable at ~1m positioning accuracy (i.e. rendering error doesn't dominate over positioning error)
 ## Key Risks / Open Questions to Surface
 - How much manual calibration does each splat capture need before poses map correctly? (Ship-wide, this needs to scale — can't hand-align every room.)
 - Splat rendering performance on-device (mobile GPU) at usable frame rates
 - How splat quality degrades in low-light, mirrored/repetitive ship corridors (relevant to capture quality later, not this session)
 - Whether direction vector alone is enough, or roll/tilt from gyroscope is needed for a convincing AR feel
 ## Deliverables
 - Short demo video: real photo vs. rendered splat view at 3–4 poses
 - Video of interpolated "walk" through the splat along a path
 - One-page write-up of the coordinate alignment method and error observed
 - List of blockers/questions for the next session (likely: scaling capture+alignment to full ship, and integrating with actual positioning data)