Research: on-device visual relocalization against a gaussian splat #2
Notifications
Due Date
No due date set.
Blocks
#10 Decide: bake-off design — comparands, ground truth, decision rule
lars/splat-indoor-navigation
#11 Decide: success gates and the measurement protocol
lars/splat-indoor-navigation
Reference: lars/splat-indoor-navigation#2
Reference in New Issue
Block a user
Question
What is the 2026 state of the art for a phone localizing itself against a pre-captured gaussian splat or 3D model, on-device, and is it reachable from React Native?
Answer specifically:
Part of the wayfinder map #1.
Answer
Researched 2026-08-11. Short version: nothing localizes a phone against a Gaussian splat on-device today. Splat-based localization is a 2024–2026 paper field that runs on desktop GPUs and, critically, nearly all of it is pose refinement needing a prior pose. What is shipping is a different thing wearing similar clothes: sparse-feature VPS maps (Immersal, MultiSet, Vuforia Area Targets) that you build yourself from a phone scan, plus ARKit VIO as the between-fixes backbone. The splat stays what it always was for us — the renderer, not the localizer. Two separate artefacts of the same room.
1. The approaches, and which work against a splat we captured
ARWorldMapVuforia: Area Targets is the one product designed around exactly our capture device — the Creator App scans with iPhone/iPad Pro LiDAR and generates a device database that tracks fully on-device, and there is an Area Target Capture API for runtime capture (PTC library; default quota 10 generations). But 2025/2026 reports say PTC stopped selling new Vuforia Engine licences in late 2025, supporting existing installs only (summary — this is a competitor's page, treat as a lead, not proof). Do not start a multi-year ship programme on it without a written answer from PTC.
Niantic: Niantic sold the games business to Scopely (2025); Niantic Spatial is the spun-out geospatial company. 8th Wall went open-source with hosted access ending 2026-02-28 and hosted campaigns dead 2027-02-28; Niantic Spatial VPS, Lightship Maps and the Geospatial Browser are explicitly not available to offline/open-source 8th Wall projects after that (8th Wall blog, Road to VR). Treat Niantic as platform risk, not a foundation.
Splat-native relocalization, honestly:
I found no shipping app, SDK or demo doing splat-based relocalization on a phone. That is the single most important finding.
2. Accuracy and latency — and the corridor failure mode
VIO backbone (the part we can trust): Kim et al., An Empirical Evaluation of Four Off-the-Shelf Proprietary VIO Systems (arXiv 2207.06780, 2022-07-14) — closed-loop final drift error (FDE), handheld, walking:
Their indoor set deliberately includes "repetitive motion in stairs, temporary occlusions, and areas lacking visual features," and ARKit was the most consistent of the four overall. ~0.5% of distance travelled indoors is the number to plan against: ARKit VIO alone holds ~1 m over roughly 150–200 m of walking. That is a much stronger result than the BLE side is likely to produce — provided the absolute origin is fixed.
VPS fix accuracy (vendor-claimed, not independently verified):
Repetitive corridors — the real risk, and no vendor addresses it. The failure has a name in the literature: perceptual aliasing. Surveys and indoor-VPR work state it plainly — indoor environments are symmetric and repetitive; corridors with bland walls and repeated doors and lights cause confusion between distinct places, producing wrong loop closures and bad localization (Lowry et al., Visual Place Recognition: A Survey, PDF; NYC-Indoor-VPR, arXiv 2404.00504). A mirrored ship with identical port/starboard corridors and hundreds of identical cabin doors is close to the adversarial worst case for retrieval-based relocalization. Google says the quiet part out loud for Cloud Anchors: hosting works best where there are many unique features, avoid plain or dimly lit spaces — and many anchors in a small area slow resolution down.
Mitigations that exist and that we would have to build ourselves: constrain the VPS query to a small candidate region using the VIO prior plus deck knowledge (never search the whole ship), require geometric verification, use sequence-based rather than single-image matching, and reject fixes that disagree with the VIO track (a mirrored-corridor mismatch is typically a large, self-consistent jump — detectable). None of this is off-the-shelf.
3. Absolute heading — yes, and this is the strongest argument for the visual approach
A VPS fix returns full 6-DoF pose in the map frame, so it yields absolute yaw with no magnetometer involved. MultiSet claims 2° on lock; ARCore Geospatial documents 5° rotational. This genuinely sidesteps the compass-on-steel problem, and it is something BLE trilateration cannot do at all — RSSI gives position only, so a beacon-only system still needs the magnetometer (or heading inferred from motion) to point an AR arrow. Caveat on magnitude: 2° is fine, but 5° of heading error is ~1.7 m of lateral error at 20 m, so heading must be re-fixed periodically too, not just once.
ARKit's own
gravityAndHeadingworld alignment derives heading from the compass, and is therefore exactly what we are trying to avoid. (I could not fetch Apple's docs pages — developer.apple.com is JS-rendered and returned title-only — so the specific caveat wording there is unverified.)4. React Native availability
Blunt: no VPS vendor ships a React Native package. Verified npm state (2026-08-11):
react-native-arkit— latest 0.9.0, published 2018-03-10. Dead.@reactvision/react-viro— 2.57.5, 2026-07-27. Actively maintained, and the one useful find:ViroARSceneexposesonCameraTransformUpdate, i.e. ARKit/ARCore camera position + rotation delivered into JS (docs). A working RN path to VIO pose without writing our own module.react-native-vision-camera— 5.2.2, 2026-08-05. Frame processors give us camera frames, which is how we'd feed a localizer.react-native-fast-tflite3.0.1 (2026-04-21),onnxruntime-react-native1.24.3 (2026-03-05) — on-device inference from RN is real, if we ever ship our own model.So the RN path is: Viro for VIO pose today, and if we want VPS, write an Expo Module around a native SDK (MultiSet is the only credible candidate with real native SDKs). Budget real Swift/Kotlin work — this is not an npm install.
5. Chest height, and when the camera sees nothing
.limitedtracking and keeps dead-reckoning off the IMU, with drift growing. Practically: the system keeps working off VIO and gets worse until the next successful fix. This is the architecture: VIO continuous, VPS as an intermittent absolute-pose corrector. Design the UX to tolerate minutes without a fix, and to ask the user to raise the phone when confidence decays.6. Licensing, cost, offline
Offline capability is a hard requirement for a cruise ship (no usable connectivity at sea, passengers not on ship wifi). That knocks out anything cloud-only, which is most of the field.
Verdict
Keep BLE as the primary comparand for Session 0. Do not put splat-based visual relocalization in the bake-off — it does not exist. But add one visual comparand, because the interesting result is not "visual vs BLE," it is that they are complementary:
onCameraTransformUpdategives us ARKit pose in RN today: no vendor, no licence, no ship hardware. In the office room, then in a corridor, measure closed-loop drift over 50–150 m walks. If we reproduce anything near the published 0.5%-of-distance figure, VIO alone beats 1 m across a whole deck and the problem collapses to "how do we re-fix the origin every few hundred metres" — a far cheaper problem than covering a ship in beacons. My expectation: this comparand wins the accuracy column outright, BLE wins the robustness column.Open / unresolved
sessionShouldAttemptRelocalizationbehaviour,gravityAndHeading's documented accuracy notes, and current ARGeoAnchor region availability are all unverified here — someone should read these in Xcode's docs viewer. Whether iOS 26 / recent ARKit added anything for persistent multi-room relocalization is likewise unestablished.Addendum — VPS is less closed off than the answer above concluded, and the real risk is recall, not accuracy
A late vendor-focused pass corrects several specifics and one framing.
Corrections to the answer above
xcframeworkon public GitHub with a tiny API (localize()→ pose plus confidence) — genuinely bridgeable to RN. Catch: on-device offline is Enterprise-only (quote), and on-device map bundles run "100s of MBs" per ~50k sq ft./localizereturns a full 3×3 rotation matrix, so cloud localization from JS works today; what's weak is offline-from-RN (native iOS plugin exists, sample stale since 2024-08-20) and that it publishes no accuracy or latency figure anywhere.developer.vuforia.comfetch 403'd — so the EOL claim in the answer above remains uncorroborated in both directions.The framing that changes
Accuracy is not the risk — recall is. Every vendor's published figure (6–8 cm) beats 1 m by an order of magnitude. Nobody publishes a recall or success rate, and nobody documents behaviour at chest height while walking. That is a thing to prototype, not to research further, and it should be an explicit measurement in #10 rather than an assumption.
Two findings that shape the ship-scale story
ARGeoAnchor, VPS2 coarse geoposition — all wrong the moment the ship leaves the berth. Use map-relative pose only. Bonus: map-relative pose also yields absolute yaw, which sidesteps the magnetometer for Immersal, MultiSet, Vuforia, precise-mode Niantic, andARWorldMap. (Niantic's coarse mode fuses the magnetometer — don't use it for heading.)Net effect on the bake-off: a VPS arm is more viable than the answer above implied — MultiSet specifically, via a native SDK bridge — but its offline gating, per-map bundle size, and unmeasured recall keep it behind ARKit VIO + markers for a first prototype. Worth recording as a costed option rather than a closed door.
Caveats: this pass exhausted its search budget, and all Vuforia detail comes from search extracts rather than direct doc reads.
Addendum 2 — the aliasing failure mode, now quantified, and it is worse than assumed
A third pass corroborates this ticket's verdict and puts numbers on the risk that was previously only named.
Confirms the core finding. 3DGS localization is a pose-refinement literature: the sub-centimetre figures everyone quotes (GS-CPR 0.8 cm, GS-SMC 1.13 cm, UGS-Loc 0.64 cm) are refinements of an already-good prior, usually ACE's output which is already 1.1 cm. iComMa literally parameterises the prior it needs (±10–40° rotation). Prior-free methods do exist — STDLoc (CVPR 2025, 0.76 cm, but 152 ms on an RTX 4090), 6DGS, SplatLoc, SGLoc — all young and unvalidated at scale. Exactly one claim of on-phone splat localization exists (SplatPose, ACM MM '25) and it is paywalled with no code, no preprint, outdoors. The only embedded number anywhere is iGaussian at 2.87 FPS (~350 ms/pose) on a robot SoC.
Repetitive, mirrored space is measurably catastrophic for retrieval-based localization:
Two consequences. First, a mirrored port/starboard fix is a 180° error — an ambiguity problem, not an accuracy problem, so no amount of refinement precision helps. Second, InLoc did improve enormously on repetitive indoor scenes (42.6% → 94.4% @1 m since 2018) but only via dense transformer matchers that cannot run on a phone. Accuracy in repetitive space and mobile-runnability are in direct conflict.
Latency kills on-device retrieval anyway: hloc matching costs 510–1900 ms per image pair on a Jetson Orin (≈ an iPhone Pro for sustained compute) → 5–19 s for k=10 candidates. Server-side reloc is 150–400 ms. On-device is only viable as XFeat + LightGlue against 1–3 VIO-constrained candidates (200–600 ms) — which requires a good prior, closing the loop back to #15.
Licensing traps, verified by reading LICENSE files — relevant if a localizer is ever built rather than bought: ACE is Niantic non-commercial with "Patent Pending", MASt3R is CC BY-NC-SA, VGGT is a Meta Research Materials licence (not OSI). The commercially clean stack is XFeat + LightGlue + gsplat + COLMAP/GLOMAP (Apache-2.0/BSD), plus
be2rlab/gsplatlocas the only permissively-licensed 3DGS localizer shipping code.Architecture worth copying: Niantic Scaniverse builds two maps from one capture — localize against the sparse map, render the splat. That is precisely the separation this ticket concluded, already productised.
OCR of signage is the cheap symmetry-breaker, and a ship is the ideal case for it. Deck numbers, cabin numbers and frame markings are painted everywhere by regulation. +17.9 points on a repetitive building, and it needs no added hardware at all — a much better fit for the project's constraint than beacons.
Two sobering gaps: there is zero published visual localization work inside any ship — novel territory, and the shipboard AR that does exist (US Navy ARMS, 5 vessels) is remote-expert overlay that avoids global localization entirely. And barometric deck classification has never been validated aboard a moving vessel — see the correction on the map.