splat-indoor-navigation/prototyping-session.md

# Prototyping Session: Pose → Splat Rendering

## Context
Part of the cruise ship AR indoor navigation project. This session isolates and de-risks the **rendering half** of the pipeline (position + direction vector → correct camera view in a Gaussian Splat), independent of the indoor positioning problem. Indoor positioning accuracy is a separate, harder problem — here we assume we *have* a pose and just need to prove we can turn it into a convincing live AR view.

## Core Hypothesis
Given a 3D position vector and a direction (viewing) vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, at a frame rate and visual fidelity good enough to feel like "AR navigation" rather than a static 3D viewer.

## Goal for the Session
By the end, have a working demo: move a virtual camera through a splat of a real room/corridor using manually or programmatically fed (position, direction) pairs, and confirm the rendered view matches what a person standing there and looking that way would actually see.

## In Scope
- Defining the pose representation (position vector + direction vector, or quaternion)
- Capturing or reusing one small real-world splat (a room, hallway, or stairwell — not the whole ship)
- Aligning the splat's internal coordinate system to a real-world/metric coordinate system
- Feeding pose data into a splat viewer/renderer and rendering the correct view
- Validating rendered view vs. reality at a few reference points
- Simulating a "live navigation" feel by animating/interpolating pose along a path

## Out of Scope (for this session)
- Actual indoor positioning tech (assume pose is given/manually set)
- Full ship-scale splat capture
- Route computation / pathfinding
- Production mobile app integration
- Gyroscope fusion (optional stretch only, see below)

## Pose Representation
Decide and document early — this is the contract between "positioning" and "rendering":
- **Position vector**: `(x, y, z)` in a fixed world/ship coordinate frame, metric units
- **Direction vector**: either
  - a normalized forward vector `(dx, dy, dz)`, or
  - yaw/pitch (+ fixed roll) if you want to lean on the phone's gyroscope later, or
  - a full quaternion if you need roll too
- Decide now whether "up" is world-up or splat-native up (splats often come with an arbitrary up axis — needs alignment)

## Tooling Options to Test
- **Splat capture**: Polycam, Luma AI, or a quick Postshot/nerfstudio run from a phone video of a real room
- **Splat rendering/viewer**:
  - Web: three.js gsplat viewers (e.g. antimatter15 splat viewer, `@mkkellogg/GaussianSplats3D`, or Luma's web SDK) — easiest to script custom camera poses into
  - Engine: Unity/Unreal with a Gaussian Splatting plugin, if you want mobile deployment sooner
- Pick web-based first for this session — fastest to iterate on "set camera pose programmatically."

## Session Agenda (half-day, ~4 hrs)

**1. Setup (30 min)**
- Pick/capture one small real space (a room or corridor you can physically walk into)
- Get it into a splat viewer that accepts programmatic camera pose input

**2. Coordinate alignment (45 min)**
- Establish 2–3 known reference points in the real space (measured with tape measure/laser) and their pixel/splat coordinates
- Compute the transform (scale, rotation, offset) between splat space and real-world metric space
- Sanity check: does a known real position map to the expected point in the splat?

**3. Pose → render pipeline (45 min)**
- Write the small script/function: `renderView(position, direction) → camera frame`
- Manually feed in a handful of (position, direction) pairs corresponding to places you can physically stand
- Render and screenshot each

**4. Validation (45 min)**
- Stand in the real space at those same points, facing the same direction, take a phone photo
- Compare rendered splat view vs. real photo side by side
- Note: does it match well enough? Where does it break (occlusion, splat artifacts, alignment drift)?

**5. Fake "live" navigation (45 min)**
- Define a short path as a sequence of (position, direction) waypoints
- Interpolate between them (linear position, slerp direction) and animate the camera through the splat
- Does it feel smooth and directionally correct, like walking through the space?

**6. Stretch goal (remaining time, optional)**
- Pipe live phone gyroscope data into the direction vector while position stays fixed/manual
- Test whether looking around live in the splat feels responsive enough

**7. Wrap-up (15–30 min)**
- Debrief: what worked, what didn't, what's the biggest open risk
- Capture screenshots/video of best result as a demo artifact

## Success Criteria
- Rendered splat view at a given pose is recognizably the same as the real view (same landmarks, correct orientation) to a human comparing them
- Camera can be moved through a sequence of poses smoothly enough to read as "navigation," not a slideshow
- Coordinate alignment error is small enough to be usable at ~1m positioning accuracy (i.e. rendering error doesn't dominate over positioning error)

## Key Risks / Open Questions to Surface
- How much manual calibration does each splat capture need before poses map correctly? (Ship-wide, this needs to scale — can't hand-align every room.)
- Splat rendering performance on-device (mobile GPU) at usable frame rates
- How splat quality degrades in low-light, mirrored/repetitive ship corridors (relevant to capture quality later, not this session)
- Whether direction vector alone is enough, or roll/tilt from gyroscope is needed for a convincing AR feel

## Deliverables
- Short demo video: real photo vs. rendered splat view at 3–4 poses
- Video of interpolated "walk" through the splat along a path
- One-page write-up of the coordinate alignment method and error observed
- List of blockers/questions for the next session (likely: scaling capture+alignment to full ship, and integrating with actual positioning data)