diff --git a/prototyping-session.md b/prototyping-session.md new file mode 100644 index 0000000..7c36dc3 --- /dev/null +++ b/prototyping-session.md @@ -0,0 +1,92 @@ +# Prototyping Session: Pose → Splat Rendering + +## Context +Part of the cruise ship AR indoor navigation project. This session isolates and de-risks the **rendering half** of the pipeline (position + direction vector → correct camera view in a Gaussian Splat), independent of the indoor positioning problem. Indoor positioning accuracy is a separate, harder problem — here we assume we *have* a pose and just need to prove we can turn it into a convincing live AR view. + +## Core Hypothesis +Given a 3D position vector and a direction (viewing) vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, at a frame rate and visual fidelity good enough to feel like "AR navigation" rather than a static 3D viewer. + +## Goal for the Session +By the end, have a working demo: move a virtual camera through a splat of a real room/corridor using manually or programmatically fed (position, direction) pairs, and confirm the rendered view matches what a person standing there and looking that way would actually see. + +## In Scope +- Defining the pose representation (position vector + direction vector, or quaternion) +- Capturing or reusing one small real-world splat (a room, hallway, or stairwell — not the whole ship) +- Aligning the splat's internal coordinate system to a real-world/metric coordinate system +- Feeding pose data into a splat viewer/renderer and rendering the correct view +- Validating rendered view vs. reality at a few reference points +- Simulating a "live navigation" feel by animating/interpolating pose along a path + +## Out of Scope (for this session) +- Actual indoor positioning tech (assume pose is given/manually set) +- Full ship-scale splat capture +- Route computation / pathfinding +- Production mobile app integration +- Gyroscope fusion (optional stretch only, see below) + +## Pose Representation +Decide and document early — this is the contract between "positioning" and "rendering": +- **Position vector**: `(x, y, z)` in a fixed world/ship coordinate frame, metric units +- **Direction vector**: either + - a normalized forward vector `(dx, dy, dz)`, or + - yaw/pitch (+ fixed roll) if you want to lean on the phone's gyroscope later, or + - a full quaternion if you need roll too +- Decide now whether "up" is world-up or splat-native up (splats often come with an arbitrary up axis — needs alignment) + +## Tooling Options to Test +- **Splat capture**: Polycam, Luma AI, or a quick Postshot/nerfstudio run from a phone video of a real room +- **Splat rendering/viewer**: + - Web: three.js gsplat viewers (e.g. antimatter15 splat viewer, `@mkkellogg/GaussianSplats3D`, or Luma's web SDK) — easiest to script custom camera poses into + - Engine: Unity/Unreal with a Gaussian Splatting plugin, if you want mobile deployment sooner +- Pick web-based first for this session — fastest to iterate on "set camera pose programmatically." + +## Session Agenda (half-day, ~4 hrs) + +**1. Setup (30 min)** +- Pick/capture one small real space (a room or corridor you can physically walk into) +- Get it into a splat viewer that accepts programmatic camera pose input + +**2. Coordinate alignment (45 min)** +- Establish 2–3 known reference points in the real space (measured with tape measure/laser) and their pixel/splat coordinates +- Compute the transform (scale, rotation, offset) between splat space and real-world metric space +- Sanity check: does a known real position map to the expected point in the splat? + +**3. Pose → render pipeline (45 min)** +- Write the small script/function: `renderView(position, direction) → camera frame` +- Manually feed in a handful of (position, direction) pairs corresponding to places you can physically stand +- Render and screenshot each + +**4. Validation (45 min)** +- Stand in the real space at those same points, facing the same direction, take a phone photo +- Compare rendered splat view vs. real photo side by side +- Note: does it match well enough? Where does it break (occlusion, splat artifacts, alignment drift)? + +**5. Fake "live" navigation (45 min)** +- Define a short path as a sequence of (position, direction) waypoints +- Interpolate between them (linear position, slerp direction) and animate the camera through the splat +- Does it feel smooth and directionally correct, like walking through the space? + +**6. Stretch goal (remaining time, optional)** +- Pipe live phone gyroscope data into the direction vector while position stays fixed/manual +- Test whether looking around live in the splat feels responsive enough + +**7. Wrap-up (15–30 min)** +- Debrief: what worked, what didn't, what's the biggest open risk +- Capture screenshots/video of best result as a demo artifact + +## Success Criteria +- Rendered splat view at a given pose is recognizably the same as the real view (same landmarks, correct orientation) to a human comparing them +- Camera can be moved through a sequence of poses smoothly enough to read as "navigation," not a slideshow +- Coordinate alignment error is small enough to be usable at ~1m positioning accuracy (i.e. rendering error doesn't dominate over positioning error) + +## Key Risks / Open Questions to Surface +- How much manual calibration does each splat capture need before poses map correctly? (Ship-wide, this needs to scale — can't hand-align every room.) +- Splat rendering performance on-device (mobile GPU) at usable frame rates +- How splat quality degrades in low-light, mirrored/repetitive ship corridors (relevant to capture quality later, not this session) +- Whether direction vector alone is enough, or roll/tilt from gyroscope is needed for a convincing AR feel + +## Deliverables +- Short demo video: real photo vs. rendered splat view at 3–4 poses +- Video of interpolated "walk" through the splat along a path +- One-page write-up of the coordinate alignment method and error observed +- List of blockers/questions for the next session (likely: scaling capture+alignment to full ship, and integrating with actual positioning data) \ No newline at end of file