splat-indoor-navigation/prototyping-session.md

Prototyping Session: Pose → Splat Rendering

Context

Part of the cruise ship AR indoor navigation project. This session isolates and de-risks the rendering half of the pipeline (position + direction vector → correct camera view in a Gaussian Splat), independent of the indoor positioning problem. Indoor positioning accuracy is a separate, harder problem — here we assume we have a pose and just need to prove we can turn it into a convincing live AR view.

Core Hypothesis

Given a 3D position vector and a direction (viewing) vector, we can render the corresponding camera view inside a pre-captured splat of a real space, live, at a frame rate and visual fidelity good enough to feel like "AR navigation" rather than a static 3D viewer.

Goal for the Session

By the end, have a working demo: move a virtual camera through a splat of a real room/corridor using manually or programmatically fed (position, direction) pairs, and confirm the rendered view matches what a person standing there and looking that way would actually see.

In Scope

• Defining the pose representation (position vector + direction vector, or quaternion)
• Capturing or reusing one small real-world splat (a room, hallway, or stairwell — not the whole ship)
• Aligning the splat's internal coordinate system to a real-world/metric coordinate system
• Feeding pose data into a splat viewer/renderer and rendering the correct view
• Validating rendered view vs. reality at a few reference points
• Simulating a "live navigation" feel by animating/interpolating pose along a path

Out of Scope (for this session)

• Actual indoor positioning tech (assume pose is given/manually set)
• Full ship-scale splat capture
• Route computation / pathfinding
• Production mobile app integration
• Gyroscope fusion (optional stretch only, see below)

Pose Representation

Decide and document early — this is the contract between "positioning" and "rendering":

• Position vector: (x, y, z) in a fixed world/ship coordinate frame, metric units
• Direction vector: either
  • a normalized forward vector (dx, dy, dz), or
  • yaw/pitch (+ fixed roll) if you want to lean on the phone's gyroscope later, or
  • a full quaternion if you need roll too
• Decide now whether "up" is world-up or splat-native up (splats often come with an arbitrary up axis — needs alignment)

Tooling Options to Test

• Splat capture: Polycam, Luma AI, or a quick Postshot/nerfstudio run from a phone video of a real room
• Splat rendering/viewer:
  • Web: three.js gsplat viewers (e.g. antimatter15 splat viewer, @mkkellogg/GaussianSplats3D, or Luma's web SDK) — easiest to script custom camera poses into
  • Engine: Unity/Unreal with a Gaussian Splatting plugin, if you want mobile deployment sooner
• Pick web-based first for this session — fastest to iterate on "set camera pose programmatically."

Session Agenda (half-day, ~4 hrs)

1. Setup (30 min)

• Pick/capture one small real space (a room or corridor you can physically walk into)
• Get it into a splat viewer that accepts programmatic camera pose input

2. Coordinate alignment (45 min)

• Establish 2–3 known reference points in the real space (measured with tape measure/laser) and their pixel/splat coordinates
• Compute the transform (scale, rotation, offset) between splat space and real-world metric space
• Sanity check: does a known real position map to the expected point in the splat?

3. Pose → render pipeline (45 min)

• Write the small script/function: renderView(position, direction) → camera frame
• Manually feed in a handful of (position, direction) pairs corresponding to places you can physically stand
• Render and screenshot each

4. Validation (45 min)

• Stand in the real space at those same points, facing the same direction, take a phone photo
• Compare rendered splat view vs. real photo side by side
• Note: does it match well enough? Where does it break (occlusion, splat artifacts, alignment drift)?

5. Fake "live" navigation (45 min)

• Define a short path as a sequence of (position, direction) waypoints
• Interpolate between them (linear position, slerp direction) and animate the camera through the splat
• Does it feel smooth and directionally correct, like walking through the space?

6. Stretch goal (remaining time, optional)

• Pipe live phone gyroscope data into the direction vector while position stays fixed/manual
• Test whether looking around live in the splat feels responsive enough

7. Wrap-up (15–30 min)

• Debrief: what worked, what didn't, what's the biggest open risk
• Capture screenshots/video of best result as a demo artifact

Success Criteria

• Rendered splat view at a given pose is recognizably the same as the real view (same landmarks, correct orientation) to a human comparing them
• Camera can be moved through a sequence of poses smoothly enough to read as "navigation," not a slideshow
• Coordinate alignment error is small enough to be usable at ~1m positioning accuracy (i.e. rendering error doesn't dominate over positioning error)

Key Risks / Open Questions to Surface

• How much manual calibration does each splat capture need before poses map correctly? (Ship-wide, this needs to scale — can't hand-align every room.)
• Splat rendering performance on-device (mobile GPU) at usable frame rates
• How splat quality degrades in low-light, mirrored/repetitive ship corridors (relevant to capture quality later, not this session)
• Whether direction vector alone is enough, or roll/tilt from gyroscope is needed for a convincing AR feel

Deliverables

• Short demo video: real photo vs. rendered splat view at 3–4 poses
• Video of interpolated "walk" through the splat along a path
• One-page write-up of the coordinate alignment method and error observed
• List of blockers/questions for the next session (likely: scaling capture+alignment to full ship, and integrating with actual positioning data)