Research: on-device visual relocalization against a gaussian splat #2

Closed
opened 2026-08-11 15:41:37 +00:00 by lars · 3 comments
Owner

Question

What is the 2026 state of the art for a phone localizing itself against a pre-captured gaussian splat or 3D model, on-device, and is it reachable from React Native?

Answer specifically:

  • What approaches exist (ARKit/ARCore VIO + periodic visual relocalization, splat-based pose regression, feature-map relocalization a la ARCore Geospatial / Immersal / Niantic VPS), and which are usable against a splat we captured ourselves.
  • Reported accuracy and relocalization latency, and how they degrade in repetitive, self-similar corridors — the ship failure mode.
  • Whether it yields absolute heading (which would sidestep the magnetometer-on-steel problem).
  • Availability from React Native: native SDK with an RN bridge, an existing RN package, or nothing shipping.
  • Whether it works with the phone held at chest height in normal use, and what happens when the camera sees nothing useful.
  • Licensing and cost of any hosted service, and whether an offline/on-device mode exists.

Part of the wayfinder map #1.

## Question What is the 2026 state of the art for a phone localizing itself **against a pre-captured gaussian splat or 3D model**, on-device, and is it reachable from React Native? Answer specifically: - What approaches exist (ARKit/ARCore VIO + periodic visual relocalization, splat-based pose regression, feature-map relocalization a la ARCore Geospatial / Immersal / Niantic VPS), and which are usable against a splat we captured ourselves. - Reported accuracy and relocalization latency, and how they degrade in **repetitive, self-similar corridors** — the ship failure mode. - Whether it yields **absolute heading** (which would sidestep the magnetometer-on-steel problem). - Availability from React Native: native SDK with an RN bridge, an existing RN package, or nothing shipping. - Whether it works with the phone held at chest height in normal use, and what happens when the camera sees nothing useful. - Licensing and cost of any hosted service, and whether an offline/on-device mode exists. --- Part of the wayfinder map #1.
lars added this to the Wayfinder: RN prototyping spec milestone 2026-08-11 15:41:37 +00:00
lars added the wayfinder:research label 2026-08-11 15:41:37 +00:00
lars added a new dependency 2026-08-11 15:42:13 +00:00
Author
Owner

Answer

Researched 2026-08-11. Short version: nothing localizes a phone against a Gaussian splat on-device today. Splat-based localization is a 2024–2026 paper field that runs on desktop GPUs and, critically, nearly all of it is pose refinement needing a prior pose. What is shipping is a different thing wearing similar clothes: sparse-feature VPS maps (Immersal, MultiSet, Vuforia Area Targets) that you build yourself from a phone scan, plus ARKit VIO as the between-fixes backbone. The splat stays what it always was for us — the renderer, not the localizer. Two separate artefacts of the same room.


1. The approaches, and which work against a splat we captured

Approach What it localizes against Usable with our own capture? Shipping?
ARKit/ARCore VIO (relative odometry) nothing — integrates motion from origin n/a Yes, mature
ARKit ARWorldMap Apple's opaque feature map from a prior session Yes, but Apple's own map format, not our splat; viewpoint/lighting-sensitive Yes, iOS 12+
ARCore Cloud Anchors Google-hosted feature map from a hosting scan Yes (you host anchors) Yes; free-tier anchors expire in 24h, business tier up to 365 days; quota 300/min/project (docs)
ARCore Geospatial API / Google VPS Google's Street View–derived global map No. You cannot supply your own indoor map; coverage is Street View, i.e. not ship interiors Yes but irrelevant to us
Immersal (Hexagon) your own sparse point-feature map, built from phone images Yes — this is the core use case Yes
MultiSet AI your own scan (scan-agnostic; accepts LiDAR/photogrammetry) Yes Yes
Vuforia Area Targets (PTC) your own iPhone/iPad Pro LiDAR scan Yes — closest to our exact capture path Effectively dying, see below
Niantic Spatial VPS (ex-Lightship/8th Wall) Niantic's public wayspot map Mostly no for private ship interiors Platform in upheaval, see below
3DGS relocalization (GSplatLoc, SplatLoc, GSFeatLoc, Hi2-GSLoc…) the splat itself Yes in principle Research only, no mobile runtime

Vuforia: Area Targets is the one product designed around exactly our capture device — the Creator App scans with iPhone/iPad Pro LiDAR and generates a device database that tracks fully on-device, and there is an Area Target Capture API for runtime capture (PTC library; default quota 10 generations). But 2025/2026 reports say PTC stopped selling new Vuforia Engine licences in late 2025, supporting existing installs only (summary — this is a competitor's page, treat as a lead, not proof). Do not start a multi-year ship programme on it without a written answer from PTC.

Niantic: Niantic sold the games business to Scopely (2025); Niantic Spatial is the spun-out geospatial company. 8th Wall went open-source with hosted access ending 2026-02-28 and hosted campaigns dead 2027-02-28; Niantic Spatial VPS, Lightship Maps and the Geospatial Browser are explicitly not available to offline/open-source 8th Wall projects after that (8th Wall blog, Road to VR). Treat Niantic as platform risk, not a foundation.

Splat-native relocalization, honestly:

  • GSFeatLoc (arXiv 2504.20379, 2025-04-29): matches query features against features rendered from the splat. <5° rotation / 0.05 scene-units translation on 90% of images, 0.1 s per query (down from >10 s). But it requires an initial pose estimate, tolerating up to 55° / 1.1 scene-units of error. It refines; it does not find you from scratch.
  • GSplatLoc (arXiv 2412.20056, Dec 2024, rev. May 2025): sub-millimetre translation on Replica/TUM-RGBD — but it is gradient-based optimisation of pose against rendered depth, i.e. again refinement, on desktop GPU, on room-scale benchmark scenes.
  • SplatLoc (TVCG 2025, repo), GSplatLoc IROS 2025 (repo), Hi2-GSLoc (arXiv 2507.15683) — same shape: Python/CUDA, desktop, benchmark scenes.

I found no shipping app, SDK or demo doing splat-based relocalization on a phone. That is the single most important finding.

2. Accuracy and latency — and the corridor failure mode

VIO backbone (the part we can trust): Kim et al., An Empirical Evaluation of Four Off-the-Shelf Proprietary VIO Systems (arXiv 2207.06780, 2022-07-14) — closed-loop final drift error (FDE), handheld, walking:

Sequence Length ARKit FDE ARCore FDE
Indoor long corridor 145.2 m 0.79 m (0.54%) 0.12 m
Indoor hallway 84.0 m 0.14 m 0.09 m
Indoor stairs 114.1 m 0.19 m 3.98 m

Their indoor set deliberately includes "repetitive motion in stairs, temporary occlusions, and areas lacking visual features," and ARKit was the most consistent of the four overall. ~0.5% of distance travelled indoors is the number to plan against: ARKit VIO alone holds ~1 m over roughly 150–200 m of walking. That is a much stronger result than the BLE side is likely to produce — provided the absolute origin is fixed.

VPS fix accuracy (vendor-claimed, not independently verified):

  • MultiSet: "5 cm median accuracy," "sub-5 cm and 2° on initial lock," "<1 cm drift at 10 m," "52 ms median pose on Snapdragon 8 Gen 3" (pricing, SDKs). Marketing figures; no third-party benchmark found.
  • Immersal: publishes no accuracy or latency figures I could find on developers.immersal.com. Sizing instead: 100 images covers 100–120 m² indoors (FAQ).
  • ARCore Geospatial VPS: ~5 m position, 5° rotation typical (docs) — an order of magnitude short of our 1 m target, and outdoor anyway.

Repetitive corridors — the real risk, and no vendor addresses it. The failure has a name in the literature: perceptual aliasing. Surveys and indoor-VPR work state it plainly — indoor environments are symmetric and repetitive; corridors with bland walls and repeated doors and lights cause confusion between distinct places, producing wrong loop closures and bad localization (Lowry et al., Visual Place Recognition: A Survey, PDF; NYC-Indoor-VPR, arXiv 2404.00504). A mirrored ship with identical port/starboard corridors and hundreds of identical cabin doors is close to the adversarial worst case for retrieval-based relocalization. Google says the quiet part out loud for Cloud Anchors: hosting works best where there are many unique features, avoid plain or dimly lit spaces — and many anchors in a small area slow resolution down.

Mitigations that exist and that we would have to build ourselves: constrain the VPS query to a small candidate region using the VIO prior plus deck knowledge (never search the whole ship), require geometric verification, use sequence-based rather than single-image matching, and reject fixes that disagree with the VIO track (a mirrored-corridor mismatch is typically a large, self-consistent jump — detectable). None of this is off-the-shelf.

3. Absolute heading — yes, and this is the strongest argument for the visual approach

A VPS fix returns full 6-DoF pose in the map frame, so it yields absolute yaw with no magnetometer involved. MultiSet claims 2° on lock; ARCore Geospatial documents 5° rotational. This genuinely sidesteps the compass-on-steel problem, and it is something BLE trilateration cannot do at all — RSSI gives position only, so a beacon-only system still needs the magnetometer (or heading inferred from motion) to point an AR arrow. Caveat on magnitude: 2° is fine, but 5° of heading error is ~1.7 m of lateral error at 20 m, so heading must be re-fixed periodically too, not just once.

ARKit's own gravityAndHeading world alignment derives heading from the compass, and is therefore exactly what we are trying to avoid. (I could not fetch Apple's docs pages — developer.apple.com is JS-rendered and returned title-only — so the specific caveat wording there is unverified.)

4. React Native availability

Blunt: no VPS vendor ships a React Native package. Verified npm state (2026-08-11):

  • react-native-arkit — latest 0.9.0, published 2018-03-10. Dead.
  • @reactvision/react-viro2.57.5, 2026-07-27. Actively maintained, and the one useful find: ViroARScene exposes onCameraTransformUpdate, i.e. ARKit/ARCore camera position + rotation delivered into JS (docs). A working RN path to VIO pose without writing our own module.
  • react-native-vision-camera5.2.2, 2026-08-05. Frame processors give us camera frames, which is how we'd feed a localizer.
  • react-native-fast-tflite 3.0.1 (2026-04-21), onnxruntime-react-native 1.24.3 (2026-03-05) — on-device inference from RN is real, if we ever ship our own model.
  • Immersal: Unity-only SDK plus a REST API; supported platforms are enumerated purely as Unity/AR Foundation, Magic Leap, Xreal, PICO (compatibility). No native iOS/Android SDK to bridge. The REST API is callable from RN — send an image, get a pose — but that path is cloud-only.
  • MultiSet: native iOS (Swift Package Manager) and Android (AAR/Kotlin) SDKs (SDK page) — bridgeable via an Expo Module. No RN wrapper exists; we would write it.
  • Vuforia: Unity plus native iOS/Android; no RN wrapper.
  • WebXR-in-WebView is not a route on iOS.

So the RN path is: Viro for VIO pose today, and if we want VPS, write an Expo Module around a native SDK (MultiSet is the only credible candidate with real native SDKs). Budget real Swift/Kotlin work — this is not an npm install.

5. Chest height, and when the camera sees nothing

  • Chest height, walking, camera forward: fine for VIO (the Kim et al. sequences are handheld walking), and it is the standard AR-navigation posture. But every VPS wants the camera pointed at feature-rich geometry; a chest-height camera in a corridor sees mostly blank wall, ceiling lights and carpet. Mapping must therefore be captured at the same height and viewpoint the app will use, or matching will silently degrade. No vendor documents chest-height performance — I could not establish this from primary sources.
  • Camera sees nothing useful: VPS localization simply fails (no pose returned / retries) — it does not degrade gracefully, it just doesn't fire. ARKit meanwhile drops to .limited tracking and keeps dead-reckoning off the IMU, with drift growing. Practically: the system keeps working off VIO and gets worse until the next successful fix. This is the architecture: VIO continuous, VPS as an intermittent absolute-pose corrector. Design the UX to tolerate minutes without a fix, and to ask the user to raise the phone when confidence decays.

6. Licensing, cost, offline

  • Immersal (Hexagon): Free (non-commercial, 100 images/map, Immersal logo must be displayed throughout the experience), Pro (commercial, 100–500 images/map), Enterprise (unlimited images, map stitching to building/city scale, and the option to run the Immersal server inside your own private network) (pricing). Prices not public. Offline: yes — maps can be embedded in the app for "completely on-device localization" (FAQ). For a ship with no reliable internet, that matters enormously.
  • MultiSet: free tier, $29/mo Lite, $149/mo Plus, Enterprise custom; deployment cloud / private cloud / on-prem / on-device, including a "fully on-device runtime engine" for air-gapped sites (pricing).
  • Vuforia: 10 free Area Target generations, then licensing; on-device tracking. New licence sales reportedly ended — cost unknown.
  • ARCore Geospatial / Cloud Anchors: ARCore API free with quotas; Cloud Anchors need the business tier for >24h persistence. Cloud-dependent, so unsuitable for a ship regardless.

Offline capability is a hard requirement for a cruise ship (no usable connectivity at sea, passengers not on ship wifi). That knocks out anything cloud-only, which is most of the field.


Verdict

Keep BLE as the primary comparand for Session 0. Do not put splat-based visual relocalization in the bake-off — it does not exist. But add one visual comparand, because the interesting result is not "visual vs BLE," it is that they are complementary:

  1. Bake off BLE trilateration against ARKit-VIO-with-a-manual-origin. This is nearly free — Viro's onCameraTransformUpdate gives us ARKit pose in RN today: no vendor, no licence, no ship hardware. In the office room, then in a corridor, measure closed-loop drift over 50–150 m walks. If we reproduce anything near the published 0.5%-of-distance figure, VIO alone beats 1 m across a whole deck and the problem collapses to "how do we re-fix the origin every few hundred metres" — a far cheaper problem than covering a ship in beacons. My expectation: this comparand wins the accuracy column outright, BLE wins the robustness column.
  2. Treat BLE beacons as the re-fix mechanism, not the positioning system. A beacon at each lift lobby and stairwell gives an absolute fix every ~50–100 m, which is exactly what VIO needs and exactly what BLE is good at (proximity, not trilateration). That fusion plausibly hits 1 m with far fewer beacons than trilateration-to-1 m would need. It does not give heading.
  3. Park splat-based localization; re-check in 12 months. The papers move fast, but there is no mobile runtime and the methods need a prior pose — so even when they arrive they will be a refiner sitting on top of exactly the VIO+fix architecture above. Building that architecture now is not wasted work.
  4. If we want the heading benefit, spike MultiSet, not Immersal. MultiSet is the only vendor with native iOS/Android SDKs (bridgeable to RN), a public price, on-device/air-gapped deployment, and scan-agnostic ingestion of our LiDAR capture. Its accuracy claims are unverified marketing, so the spike's whole purpose is to measure them — in a corridor, at chest height, phone held normally. Immersal is Unity-only and therefore effectively closed to us despite being the more established name. Vuforia is the best technical fit for iPhone-LiDAR capture and is what I'd have recommended two years ago; its licensing status makes it unsafe now.
  5. Whatever we pick, assume perceptual aliasing will bite on a mirrored ship, and design fix-acceptance logic (VIO-constrained candidate region, deck prior, reject fixes inconsistent with the VIO track) as a first-class component. One accepted mirrored-corridor mismatch teleports the user to the wrong side of the ship — worse than no navigation at all.

Open / unresolved

  • Apple's own documentation could not be read. developer.apple.com returns title-only to WebFetch (JS-rendered). So ARWorldMap's stated caveats, size limits and cross-device portability, sessionShouldAttemptRelocalization behaviour, gravityAndHeading's documented accuracy notes, and current ARGeoAnchor region availability are all unverified here — someone should read these in Xcode's docs viewer. Whether iOS 26 / recent ARKit added anything for persistent multi-room relocalization is likewise unestablished.
  • No independent accuracy benchmark exists for any of these VPS vendors. Every centimetre figure above except the Kim et al. VIO numbers is vendor marketing.
  • Nothing quantifies VPS relocalization failure rate in repetitive corridors. The literature says perceptual aliasing is a known, serious problem; nobody publishes a false-fix rate for Immersal/MultiSet/Vuforia in a mirrored building. We would be the ones measuring it.
  • PTC's Vuforia Engine end-of-life is second-hand (a competitor's blog plus aggregator posts). Confirm directly with PTC before treating it as fact — or before using it to rule Vuforia out.
  • Chest-height performance is undocumented across the board, as is behaviour with crowds of moving passengers occluding the camera. Both are central to our actual use case.
  • Immersal and Vuforia pricing are not public (enterprise quotes only). Cost at ship or fleet scale is unknown for every vendor except MultiSet, and MultiSet's public tiers may not extend to that scale.
  • Niantic Spatial's post-8th-Wall offering was not fully pinned down — specifically whether private, self-scanned indoor VPS maps are available to third parties in 2026 and on what terms. Given the platform churn I did not chase it.
  • Map size at ship scale is unquantified. Immersal's 100 images ≈ 100–120 m² implies a large vessel needs many hundreds of stitched maps (an Enterprise-tier feature). The operational cost of capturing and maintaining that (ships get redecorated, furniture moves) is unestimated and may dominate everything else.
  • The three background research agents I dispatched returned nothing; this write-up is my own direct search pass, so it is thinner on some vendors (notably Niantic and Apple) than intended.
## Answer Researched 2026-08-11. Short version: **nothing localizes a phone against a Gaussian splat on-device today.** Splat-based localization is a 2024–2026 paper field that runs on desktop GPUs and, critically, nearly all of it is *pose refinement* needing a prior pose. What *is* shipping is a different thing wearing similar clothes: **sparse-feature VPS maps** (Immersal, MultiSet, Vuforia Area Targets) that you build yourself from a phone scan, plus **ARKit VIO** as the between-fixes backbone. The splat stays what it always was for us — the *renderer*, not the localizer. Two separate artefacts of the same room. --- ### 1. The approaches, and which work against a splat we captured | Approach | What it localizes against | Usable with our own capture? | Shipping? | |---|---|---|---| | **ARKit/ARCore VIO** (relative odometry) | nothing — integrates motion from origin | n/a | Yes, mature | | **ARKit `ARWorldMap`** | Apple's opaque feature map from a prior session | Yes, but Apple's own map format, not our splat; viewpoint/lighting-sensitive | Yes, iOS 12+ | | **ARCore Cloud Anchors** | Google-hosted feature map from a hosting scan | Yes (you host anchors) | Yes; free-tier anchors expire in **24h**, business tier up to **365 days**; quota 300/min/project ([docs](https://developers.google.com/ar/develop/cloud-anchors)) | | **ARCore Geospatial API / Google VPS** | Google's Street View–derived global map | **No.** You cannot supply your own indoor map; coverage is Street View, i.e. not ship interiors | Yes but irrelevant to us | | **Immersal (Hexagon)** | your own sparse point-feature map, built from phone images | **Yes** — this is the core use case | Yes | | **MultiSet AI** | your own scan (scan-agnostic; accepts LiDAR/photogrammetry) | **Yes** | Yes | | **Vuforia Area Targets (PTC)** | your own **iPhone/iPad Pro LiDAR** scan | **Yes** — closest to our exact capture path | **Effectively dying, see below** | | **Niantic Spatial VPS** (ex-Lightship/8th Wall) | Niantic's public wayspot map | Mostly no for private ship interiors | Platform in upheaval, see below | | **3DGS relocalization** (GSplatLoc, SplatLoc, GSFeatLoc, Hi2-GSLoc…) | the splat itself | Yes in principle | **Research only, no mobile runtime** | **Vuforia:** Area Targets is the one product designed around exactly our capture device — the Creator App scans with iPhone/iPad Pro LiDAR and generates a device database that tracks **fully on-device**, and there is an Area Target Capture API for runtime capture ([PTC library](https://developer.vuforia.com/library/vuforia-engine/environments/area-targets/area-targets/); default quota 10 generations). But 2025/2026 reports say **PTC stopped selling new Vuforia Engine licences in late 2025**, supporting existing installs only ([summary](https://www.multiset.ai/post/vuforia-area-target-alternative-upgrade-to-a-faster-scan-agnostic-vps) — this is a competitor's page, treat as a lead, not proof). Do not start a multi-year ship programme on it without a written answer from PTC. **Niantic:** Niantic sold the games business to Scopely (2025); Niantic Spatial is the spun-out geospatial company. 8th Wall went open-source with **hosted access ending 2026-02-28** and hosted campaigns dead **2027-02-28**; Niantic Spatial VPS, Lightship Maps and the Geospatial Browser are explicitly **not available** to offline/open-source 8th Wall projects after that ([8th Wall blog](https://www.8thwall.com/blog/post/202888018234/8th-wall-update-engine-distribution-and-open-source-plans), [Road to VR](https://roadtovr.com/niantic-8th-wall-open-source/)). Treat Niantic as platform risk, not a foundation. **Splat-native relocalization, honestly:** - *GSFeatLoc* (arXiv [2504.20379](https://arxiv.org/abs/2504.20379), 2025-04-29): matches query features against features rendered from the splat. <5° rotation / 0.05 scene-units translation on 90% of images, **0.1 s** per query (down from >10 s). But it **requires an initial pose estimate**, tolerating up to 55° / 1.1 scene-units of error. It refines; it does not find you from scratch. - *GSplatLoc* (arXiv [2412.20056](https://arxiv.org/abs/2412.20056), Dec 2024, rev. May 2025): sub-millimetre translation on Replica/TUM-RGBD — but it is gradient-based optimisation of pose against rendered depth, i.e. again refinement, on desktop GPU, on room-scale benchmark scenes. - *SplatLoc* (TVCG 2025, [repo](https://github.com/zhaihongjia/SplatLoc)), *GSplatLoc* IROS 2025 ([repo](https://github.com/haksorus/gsplatloc)), *Hi2-GSLoc* (arXiv [2507.15683](https://arxiv.org/pdf/2507.15683)) — same shape: Python/CUDA, desktop, benchmark scenes. I found **no** shipping app, SDK or demo doing splat-based relocalization on a phone. That is the single most important finding. ### 2. Accuracy and latency — and the corridor failure mode **VIO backbone (the part we can trust):** Kim et al., *An Empirical Evaluation of Four Off-the-Shelf Proprietary VIO Systems* (arXiv [2207.06780](https://arxiv.org/abs/2207.06780), 2022-07-14) — closed-loop final drift error (FDE), handheld, walking: | Sequence | Length | ARKit FDE | ARCore FDE | |---|---|---|---| | Indoor **long corridor** | 145.2 m | **0.79 m** (0.54%) | 0.12 m | | Indoor hallway | 84.0 m | **0.14 m** | 0.09 m | | Indoor stairs | 114.1 m | 0.19 m | 3.98 m | Their indoor set deliberately includes "repetitive motion in stairs, temporary occlusions, and areas lacking visual features," and ARKit was the most consistent of the four overall. **~0.5% of distance travelled indoors** is the number to plan against: ARKit VIO alone holds ~1 m over roughly 150–200 m of walking. That is a much stronger result than the BLE side is likely to produce — *provided* the absolute origin is fixed. **VPS fix accuracy (vendor-claimed, not independently verified):** - MultiSet: "5 cm median accuracy," "sub-5 cm and 2° on initial lock," "<1 cm drift at 10 m," "52 ms median pose on Snapdragon 8 Gen 3" ([pricing](https://multiset.ai/pricing), [SDKs](https://www.multiset.ai/arsdks)). Marketing figures; no third-party benchmark found. - Immersal: publishes **no** accuracy or latency figures I could find on developers.immersal.com. Sizing instead: 100 images covers **100–120 m² indoors** ([FAQ](https://developers.immersal.com/docs/immersal-sdk/faq/)). - ARCore Geospatial VPS: ~**5 m position, 5° rotation** typical ([docs](https://developers.google.com/ar/develop/unity-arf/geospatial/check-vps-availability)) — an order of magnitude short of our 1 m target, and outdoor anyway. **Repetitive corridors — the real risk, and no vendor addresses it.** The failure has a name in the literature: **perceptual aliasing**. Surveys and indoor-VPR work state it plainly — indoor environments are symmetric and repetitive; corridors with bland walls and repeated doors and lights cause confusion between distinct places, producing wrong loop closures and bad localization (Lowry et al., *Visual Place Recognition: A Survey*, [PDF](https://nikosuenderhauf.github.io/assets/papers/visual_place_recognition_a_survey.pdf); *NYC-Indoor-VPR*, arXiv [2404.00504](https://arxiv.org/pdf/2404.00504)). A mirrored ship with identical port/starboard corridors and hundreds of identical cabin doors is close to the adversarial worst case for retrieval-based relocalization. Google says the quiet part out loud for Cloud Anchors: hosting works best where there are many unique features, avoid plain or dimly lit spaces — and many anchors in a small area *slow resolution down*. Mitigations that exist and that we would have to build ourselves: constrain the VPS query to a small candidate region using the VIO prior plus deck knowledge (never search the whole ship), require geometric verification, use sequence-based rather than single-image matching, and reject fixes that disagree with the VIO track (a mirrored-corridor mismatch is typically a *large*, self-consistent jump — detectable). None of this is off-the-shelf. ### 3. Absolute heading — yes, and this is the strongest argument for the visual approach A VPS fix returns full **6-DoF** pose in the map frame, so it yields **absolute yaw with no magnetometer involved**. MultiSet claims 2° on lock; ARCore Geospatial documents 5° rotational. This genuinely sidesteps the compass-on-steel problem, and it is something **BLE trilateration cannot do at all** — RSSI gives position only, so a beacon-only system still needs the magnetometer (or heading inferred from motion) to point an AR arrow. Caveat on magnitude: 2° is fine, but 5° of heading error is ~1.7 m of lateral error at 20 m, so heading must be re-fixed periodically too, not just once. ARKit's own `gravityAndHeading` world alignment derives heading from the compass, and is therefore exactly what we are trying to avoid. (I could not fetch Apple's docs pages — developer.apple.com is JS-rendered and returned title-only — so the specific caveat wording there is unverified.) ### 4. React Native availability Blunt: **no VPS vendor ships a React Native package.** Verified npm state (2026-08-11): - `react-native-arkit` — latest **0.9.0, published 2018-03-10**. Dead. - `@reactvision/react-viro` — **2.57.5, 2026-07-27**. Actively maintained, and the one useful find: `ViroARScene` exposes **`onCameraTransformUpdate`**, i.e. ARKit/ARCore camera position + rotation delivered into JS ([docs](https://viro-community.readme.io/docs/viroarscene)). A working RN path to VIO pose without writing our own module. - `react-native-vision-camera` — **5.2.2, 2026-08-05**. Frame processors give us camera frames, which is how we'd feed a localizer. - `react-native-fast-tflite` **3.0.1 (2026-04-21)**, `onnxruntime-react-native` **1.24.3 (2026-03-05)** — on-device inference from RN is real, if we ever ship our own model. - **Immersal: Unity-only SDK** plus a REST API; supported platforms are enumerated purely as Unity/AR Foundation, Magic Leap, Xreal, PICO ([compatibility](https://developers.immersal.com/docs/immersal-sdk/compatibility/)). No native iOS/Android SDK to bridge. The REST API *is* callable from RN — send an image, get a pose — but that path is cloud-only. - **MultiSet: native iOS (Swift Package Manager) and Android (AAR/Kotlin) SDKs** ([SDK page](https://www.multiset.ai/arsdks)) — bridgeable via an Expo Module. No RN wrapper exists; we would write it. - Vuforia: Unity plus native iOS/Android; no RN wrapper. - WebXR-in-WebView is not a route on iOS. So the RN path is: **Viro for VIO pose today**, and if we want VPS, **write an Expo Module around a native SDK (MultiSet is the only credible candidate with real native SDKs)**. Budget real Swift/Kotlin work — this is not an npm install. ### 5. Chest height, and when the camera sees nothing - **Chest height, walking, camera forward:** fine for VIO (the Kim et al. sequences are handheld walking), and it is the standard AR-navigation posture. But every VPS wants the camera pointed at feature-rich geometry; a chest-height camera in a corridor sees mostly blank wall, ceiling lights and carpet. Mapping must therefore be captured *at the same height and viewpoint the app will use*, or matching will silently degrade. No vendor documents chest-height performance — **I could not establish this from primary sources.** - **Camera sees nothing useful:** VPS localization simply **fails** (no pose returned / retries) — it does not degrade gracefully, it just doesn't fire. ARKit meanwhile drops to `.limited` tracking and keeps dead-reckoning off the IMU, with drift growing. Practically: the system keeps working off VIO and gets worse until the next successful fix. **This is the architecture**: VIO continuous, VPS as an intermittent absolute-pose corrector. Design the UX to tolerate minutes without a fix, and to ask the user to raise the phone when confidence decays. ### 6. Licensing, cost, offline - **Immersal (Hexagon):** Free (non-commercial, 100 images/map, Immersal logo must be displayed throughout the experience), Pro (commercial, 100–500 images/map), Enterprise (unlimited images, map stitching to building/city scale, **and the option to run the Immersal server inside your own private network**) ([pricing](https://developers.immersal.com/docs/immersal-sdk/pricing/)). Prices **not public**. Offline: yes — maps can be embedded in the app for "completely on-device localization" ([FAQ](https://developers.immersal.com/docs/immersal-sdk/faq/)). For a ship with no reliable internet, that matters enormously. - **MultiSet:** free tier, **$29/mo Lite, $149/mo Plus**, Enterprise custom; deployment cloud / private cloud / on-prem / **on-device**, including a "fully on-device runtime engine" for air-gapped sites ([pricing](https://multiset.ai/pricing)). - **Vuforia:** 10 free Area Target generations, then licensing; on-device tracking. New licence sales reportedly ended — cost unknown. - **ARCore Geospatial / Cloud Anchors:** ARCore API free with quotas; Cloud Anchors need the business tier for >24h persistence. Cloud-dependent, so unsuitable for a ship regardless. Offline capability is a hard requirement for a cruise ship (no usable connectivity at sea, passengers not on ship wifi). That knocks out anything cloud-only, which is most of the field. --- ### Verdict **Keep BLE as the primary comparand for Session 0. Do not put splat-based visual relocalization in the bake-off — it does not exist.** But add one visual comparand, because the interesting result is not "visual vs BLE," it is that they are complementary: 1. **Bake off BLE trilateration against ARKit-VIO-with-a-manual-origin.** This is nearly free — Viro's `onCameraTransformUpdate` gives us ARKit pose in RN today: no vendor, no licence, no ship hardware. In the office room, then in a corridor, measure closed-loop drift over 50–150 m walks. If we reproduce anything near the published 0.5%-of-distance figure, VIO alone beats 1 m across a whole deck and the problem collapses to *"how do we re-fix the origin every few hundred metres"* — a far cheaper problem than covering a ship in beacons. My expectation: this comparand wins the accuracy column outright, BLE wins the robustness column. 2. **Treat BLE beacons as the re-fix mechanism, not the positioning system.** A beacon at each lift lobby and stairwell gives an absolute fix every ~50–100 m, which is exactly what VIO needs and exactly what BLE is good at (proximity, not trilateration). That fusion plausibly hits 1 m with far fewer beacons than trilateration-to-1 m would need. It does **not** give heading. 3. **Park splat-based localization; re-check in 12 months.** The papers move fast, but there is no mobile runtime and the methods need a prior pose — so even when they arrive they will be a *refiner* sitting on top of exactly the VIO+fix architecture above. Building that architecture now is not wasted work. 4. **If we want the heading benefit, spike MultiSet, not Immersal.** MultiSet is the only vendor with native iOS/Android SDKs (bridgeable to RN), a public price, on-device/air-gapped deployment, and scan-agnostic ingestion of our LiDAR capture. Its accuracy claims are unverified marketing, so the spike's whole purpose is to measure them — in a corridor, at chest height, phone held normally. Immersal is Unity-only and therefore effectively closed to us despite being the more established name. Vuforia is the best technical fit for iPhone-LiDAR capture and is what I'd have recommended two years ago; its licensing status makes it unsafe now. 5. **Whatever we pick, assume perceptual aliasing will bite on a mirrored ship,** and design fix-acceptance logic (VIO-constrained candidate region, deck prior, reject fixes inconsistent with the VIO track) as a first-class component. One accepted mirrored-corridor mismatch teleports the user to the wrong side of the ship — worse than no navigation at all. ### Open / unresolved - **Apple's own documentation could not be read.** developer.apple.com returns title-only to WebFetch (JS-rendered). So ARWorldMap's stated caveats, size limits and cross-device portability, `sessionShouldAttemptRelocalization` behaviour, `gravityAndHeading`'s documented accuracy notes, and current ARGeoAnchor region availability are all **unverified here** — someone should read these in Xcode's docs viewer. Whether iOS 26 / recent ARKit added anything for persistent multi-room relocalization is likewise unestablished. - **No independent accuracy benchmark exists for any of these VPS vendors.** Every centimetre figure above except the Kim et al. VIO numbers is vendor marketing. - **Nothing quantifies VPS relocalization failure rate in repetitive corridors.** The literature says perceptual aliasing is a known, serious problem; nobody publishes a false-fix rate for Immersal/MultiSet/Vuforia in a mirrored building. We would be the ones measuring it. - **PTC's Vuforia Engine end-of-life is second-hand** (a competitor's blog plus aggregator posts). Confirm directly with PTC before treating it as fact — or before using it to rule Vuforia out. - **Chest-height performance is undocumented across the board**, as is behaviour with crowds of moving passengers occluding the camera. Both are central to our actual use case. - **Immersal and Vuforia pricing are not public** (enterprise quotes only). Cost at ship or fleet scale is unknown for every vendor except MultiSet, and MultiSet's public tiers may not extend to that scale. - **Niantic Spatial's post-8th-Wall offering was not fully pinned down** — specifically whether private, self-scanned indoor VPS maps are available to third parties in 2026 and on what terms. Given the platform churn I did not chase it. - **Map size at ship scale is unquantified.** Immersal's 100 images ≈ 100–120 m² implies a large vessel needs many hundreds of stitched maps (an Enterprise-tier feature). The operational cost of capturing and *maintaining* that (ships get redecorated, furniture moves) is unestimated and may dominate everything else. - The three background research agents I dispatched returned nothing; this write-up is my own direct search pass, so it is thinner on some vendors (notably Niantic and Apple) than intended.
lars closed this issue 2026-08-11 15:51:10 +00:00
Author
Owner

Addendum — VPS is less closed off than the answer above concluded, and the real risk is recall, not accuracy

A late vendor-focused pass corrects several specifics and one framing.

Corrections to the answer above

  • MultiSet AI pricing was wrong. Public tiers are $0 / $49 / $249 / custom, not $149/mo. It publishes ~6 cm median positional error, ~3–4 s per on-device fix on a high-end iPhone, maps captured with a LiDAR iPhone app, and ships a downloadable native Swift xcframework on public GitHub with a tiny API (localize() → pose plus confidence) — genuinely bridgeable to RN. Catch: on-device offline is Enterprise-only (quote), and on-device map bundles run "100s of MBs" per ~50k sq ft.
  • Immersal is not simply Unity-only. Offline is the least gated of any vendor — embed maps, on-device localization, no tier restriction — at $0/$99/custom (free tier forbids commercial use). Its REST /localize returns a full 3×3 rotation matrix, so cloud localization from JS works today; what's weak is offline-from-RN (native iOS plugin exists, sample stale since 2024-08-20) and that it publishes no accuracy or latency figure anywhere.
  • Niantic: no offline mode is documented anywhere. Near-disqualifying at sea. VPS survived the Scopely split (Niantic Spatial, NSDK 4.x/VPS2, real Swift SDK, $0/$20/$50 credits), but lightship.dev shut down 2026-02-28 and Lightship is fully decommissioned 2027-02-20 — two breaking migrations in 18 months.
  • Vuforia Area Targets have a hard scale cap: phone-scanned targets are limited to ~50 m² each (5-minute recording limit); above that PTC directs you to Matterport/Leica/NavVis rigs. A ship deck would mean hundreds of targets. Licence availability could not be confirmed — every developer.vuforia.com fetch 403'd — so the EOL claim in the answer above remains uncorroborated in both directions.
  • ARCore Geospatial is ruled out for a reason beyond RN reach: you cannot supply your own map ("relies exclusively on Google's VPS localization model derived from Street View data"), and ~5 m / ~5° fails the 1 m target by 5×.

The framing that changes

Accuracy is not the risk — recall is. Every vendor's published figure (6–8 cm) beats 1 m by an order of magnitude. Nobody publishes a recall or success rate, and nobody documents behaviour at chest height while walking. That is a thing to prototype, not to research further, and it should be an explicit measurement in #10 rather than an assumption.

Two findings that shape the ship-scale story

  • Mirrored port/starboard corridors will need our own coarse prior regardless of vendor — deck from a barometer, zone from BLE or Wi-Fi — narrowing the query before visual matching runs. Immersal has prior-pose localization (SDK 2.1.0), MultiSet has per-map query activation, and Vuforia has an explicit location-prior API and is the only vendor that names perceptual aliasing in its docs ("tracking locations may be mistaken for one another"). This gives the mirrored-ship problem a known shape instead of leaving it unsolved.
  • Ignore all geo-referencing. WGS84/GeoPose outputs, ARGeoAnchor, VPS2 coarse geoposition — all wrong the moment the ship leaves the berth. Use map-relative pose only. Bonus: map-relative pose also yields absolute yaw, which sidesteps the magnetometer for Immersal, MultiSet, Vuforia, precise-mode Niantic, and ARWorldMap. (Niantic's coarse mode fuses the magnetometer — don't use it for heading.)

Net effect on the bake-off: a VPS arm is more viable than the answer above implied — MultiSet specifically, via a native SDK bridge — but its offline gating, per-map bundle size, and unmeasured recall keep it behind ARKit VIO + markers for a first prototype. Worth recording as a costed option rather than a closed door.

Caveats: this pass exhausted its search budget, and all Vuforia detail comes from search extracts rather than direct doc reads.

## Addendum — VPS is less closed off than the answer above concluded, and the real risk is **recall, not accuracy** A late vendor-focused pass corrects several specifics and one framing. ### Corrections to the answer above - **MultiSet AI pricing was wrong.** Public tiers are **$0 / $49 / $249 / custom**, not $149/mo. It publishes **~6 cm median positional error**, ~3–4 s per on-device fix on a high-end iPhone, maps captured with a LiDAR iPhone app, and ships a **downloadable native Swift `xcframework` on public GitHub** with a tiny API (`localize()` → pose **plus confidence**) — genuinely bridgeable to RN. **Catch: on-device offline is Enterprise-only (quote), and on-device map bundles run "100s of MBs" per ~50k sq ft.** - **Immersal is not simply Unity-only.** Offline is the *least* gated of any vendor — embed maps, on-device localization, no tier restriction — at $0/$99/custom (free tier forbids commercial use). Its REST `/localize` returns a full 3×3 rotation matrix, so **cloud localization from JS works today**; what's weak is offline-from-RN (native iOS plugin exists, sample stale since 2024-08-20) and that it **publishes no accuracy or latency figure anywhere**. - **Niantic: no offline mode is documented anywhere.** Near-disqualifying at sea. VPS survived the Scopely split (Niantic Spatial, NSDK 4.x/VPS2, real Swift SDK, $0/$20/$50 credits), but lightship.dev shut down 2026-02-28 and Lightship is fully decommissioned 2027-02-20 — two breaking migrations in 18 months. - **Vuforia Area Targets have a hard scale cap:** phone-scanned targets are limited to **~50 m² each** (5-minute recording limit); above that PTC directs you to Matterport/Leica/NavVis rigs. A ship deck would mean **hundreds** of targets. Licence availability could not be confirmed — every `developer.vuforia.com` fetch 403'd — so the EOL claim in the answer above remains uncorroborated in both directions. - **ARCore Geospatial is ruled out for a reason beyond RN reach:** you cannot supply your own map (*"relies exclusively on Google's VPS localization model derived from Street View data"*), and ~5 m / ~5° fails the 1 m target by 5×. ### The framing that changes **Accuracy is not the risk — recall is.** Every vendor's published figure (6–8 cm) beats 1 m by an order of magnitude. **Nobody publishes a recall or success rate, and nobody documents behaviour at chest height while walking.** That is a thing to prototype, not to research further, and it should be an explicit measurement in #10 rather than an assumption. ### Two findings that shape the ship-scale story - **Mirrored port/starboard corridors will need our own coarse prior regardless of vendor** — deck from a barometer, zone from BLE or Wi-Fi — narrowing the query before visual matching runs. Immersal has prior-pose localization (SDK 2.1.0), MultiSet has per-map query activation, and Vuforia has an explicit **location-prior API** and is the only vendor that names perceptual aliasing in its docs (*"tracking locations may be mistaken for one another"*). This gives the mirrored-ship problem a known shape instead of leaving it unsolved. - **Ignore all geo-referencing.** WGS84/GeoPose outputs, `ARGeoAnchor`, VPS2 coarse geoposition — all wrong the moment the ship leaves the berth. Use **map-relative pose only**. Bonus: map-relative pose also yields **absolute yaw**, which sidesteps the magnetometer for Immersal, MultiSet, Vuforia, precise-mode Niantic, and `ARWorldMap`. (Niantic's *coarse* mode fuses the magnetometer — don't use it for heading.) **Net effect on the bake-off:** a VPS arm is more viable than the answer above implied — MultiSet specifically, via a native SDK bridge — but its offline gating, per-map bundle size, and unmeasured recall keep it behind ARKit VIO + markers for a first prototype. Worth recording as a costed option rather than a closed door. *Caveats: this pass exhausted its search budget, and all Vuforia detail comes from search extracts rather than direct doc reads.*
Author
Owner

Addendum 2 — the aliasing failure mode, now quantified, and it is worse than assumed

A third pass corroborates this ticket's verdict and puts numbers on the risk that was previously only named.

Confirms the core finding. 3DGS localization is a pose-refinement literature: the sub-centimetre figures everyone quotes (GS-CPR 0.8 cm, GS-SMC 1.13 cm, UGS-Loc 0.64 cm) are refinements of an already-good prior, usually ACE's output which is already 1.1 cm. iComMa literally parameterises the prior it needs (±10–40° rotation). Prior-free methods do exist — STDLoc (CVPR 2025, 0.76 cm, but 152 ms on an RTX 4090), 6DGS, SplatLoc, SGLoc — all young and unvalidated at scale. Exactly one claim of on-phone splat localization exists (SplatPose, ACM MM '25) and it is paywalled with no code, no preprint, outdoors. The only embedded number anywhere is iGaussian at 2.87 FPS (~350 ms/pose) on a robot SoC.

Repetitive, mirrored space is measurably catastrophic for retrieval-based localization:

finding number
NetVLAD Recall@1, five-floor repetitive building 58.2% — and the failures are specifically cross-floor confusion between identical levels (per-floor is 64–88%)
Adding OCR of signage +17.9 points
Single-pose PoseNet in a symmetric meeting room 5% accuracy — it places the camera on the opposite side
ROVER (Aug 2025), sparse geometric verification in repetitive scenes 0% max-recall at 100% precision
ORB-SLAM2 with loop closures enabled vs disabled 3× worse: 0.67 m → 2.17 m
ActLoc — effect of where the camera points 60.7% → 92.1%

Two consequences. First, a mirrored port/starboard fix is a 180° error — an ambiguity problem, not an accuracy problem, so no amount of refinement precision helps. Second, InLoc did improve enormously on repetitive indoor scenes (42.6% → 94.4% @1 m since 2018) but only via dense transformer matchers that cannot run on a phone. Accuracy in repetitive space and mobile-runnability are in direct conflict.

Latency kills on-device retrieval anyway: hloc matching costs 510–1900 ms per image pair on a Jetson Orin (≈ an iPhone Pro for sustained compute) → 5–19 s for k=10 candidates. Server-side reloc is 150–400 ms. On-device is only viable as XFeat + LightGlue against 1–3 VIO-constrained candidates (200–600 ms) — which requires a good prior, closing the loop back to #15.

Licensing traps, verified by reading LICENSE files — relevant if a localizer is ever built rather than bought: ACE is Niantic non-commercial with "Patent Pending", MASt3R is CC BY-NC-SA, VGGT is a Meta Research Materials licence (not OSI). The commercially clean stack is XFeat + LightGlue + gsplat + COLMAP/GLOMAP (Apache-2.0/BSD), plus be2rlab/gsplatloc as the only permissively-licensed 3DGS localizer shipping code.

Architecture worth copying: Niantic Scaniverse builds two maps from one capture — localize against the sparse map, render the splat. That is precisely the separation this ticket concluded, already productised.

OCR of signage is the cheap symmetry-breaker, and a ship is the ideal case for it. Deck numbers, cabin numbers and frame markings are painted everywhere by regulation. +17.9 points on a repetitive building, and it needs no added hardware at all — a much better fit for the project's constraint than beacons.

Two sobering gaps: there is zero published visual localization work inside any ship — novel territory, and the shipboard AR that does exist (US Navy ARMS, 5 vessels) is remote-expert overlay that avoids global localization entirely. And barometric deck classification has never been validated aboard a moving vessel — see the correction on the map.

## Addendum 2 — the aliasing failure mode, now quantified, and it is worse than assumed A third pass corroborates this ticket's verdict and puts numbers on the risk that was previously only named. **Confirms the core finding.** 3DGS localization is a pose-*refinement* literature: the sub-centimetre figures everyone quotes (GS-CPR 0.8 cm, GS-SMC 1.13 cm, UGS-Loc 0.64 cm) are refinements of an already-good prior, usually ACE's output which is already 1.1 cm. iComMa literally parameterises the prior it needs (±10–40° rotation). Prior-free methods do exist — **STDLoc** (CVPR 2025, 0.76 cm, but **152 ms on an RTX 4090**), **6DGS**, **SplatLoc**, **SGLoc** — all young and unvalidated at scale. Exactly one claim of on-phone splat localization exists (**SplatPose**, ACM MM '25) and it is paywalled with no code, no preprint, outdoors. The only embedded number anywhere is **iGaussian at 2.87 FPS (~350 ms/pose)** on a robot SoC. **Repetitive, mirrored space is measurably catastrophic for retrieval-based localization:** | finding | number | |---|---| | NetVLAD Recall@1, five-floor repetitive building | **58.2%** — and the failures are specifically *cross-floor confusion between identical levels* (per-floor is 64–88%) | | Adding **OCR of signage** | **+17.9 points** | | Single-pose PoseNet in a *symmetric* meeting room | **5% accuracy** — it places the camera on the opposite side | | ROVER (Aug 2025), sparse geometric verification in repetitive scenes | **0% max-recall at 100% precision** | | ORB-SLAM2 with loop closures **enabled** vs disabled | **3× worse**: 0.67 m → 2.17 m | | ActLoc — effect of *where the camera points* | 60.7% → **92.1%** | Two consequences. First, **a mirrored port/starboard fix is a 180° error — an ambiguity problem, not an accuracy problem**, so no amount of refinement precision helps. Second, InLoc *did* improve enormously on repetitive indoor scenes (42.6% → 94.4% @1 m since 2018) **but only via dense transformer matchers that cannot run on a phone**. Accuracy in repetitive space and mobile-runnability are in direct conflict. **Latency kills on-device retrieval anyway:** hloc matching costs **510–1900 ms per image pair** on a Jetson Orin (≈ an iPhone Pro for sustained compute) → 5–19 s for k=10 candidates. Server-side reloc is 150–400 ms. On-device is only viable as **XFeat + LightGlue against 1–3 VIO-constrained candidates** (200–600 ms) — which requires a good prior, closing the loop back to #15. **Licensing traps, verified by reading LICENSE files** — relevant if a localizer is ever built rather than bought: **ACE is Niantic non-commercial with "Patent Pending"**, **MASt3R is CC BY-NC-SA**, **VGGT is a Meta Research Materials licence (not OSI)**. The commercially clean stack is **XFeat + LightGlue + gsplat + COLMAP/GLOMAP** (Apache-2.0/BSD), plus `be2rlab/gsplatloc` as the only permissively-licensed 3DGS localizer shipping code. **Architecture worth copying: Niantic Scaniverse builds two maps from one capture** — localize against the sparse map, render the splat. That is precisely the separation this ticket concluded, already productised. **OCR of signage is the cheap symmetry-breaker, and a ship is the ideal case for it.** Deck numbers, cabin numbers and frame markings are painted everywhere by regulation. +17.9 points on a repetitive building, and it needs no added hardware at all — a much better fit for the project's constraint than beacons. **Two sobering gaps:** there is **zero published visual localization work inside any ship** — novel territory, and the shipboard AR that does exist (US Navy ARMS, 5 vessels) is remote-expert overlay that avoids global localization entirely. And **barometric deck classification has never been validated aboard a moving vessel** — see the correction on the map.
Sign in to join this conversation.