Radiance fields, Gaussian Splatter, Photogrammetry, and other computational photographic techniques are where this is headed. It's not that much overhead to render a stereoscopic view from the same dataset. That camera array on the back of your phone is more than capable of hand tracking.
Being able to capture "spatial video" is the hook Apple is using to sell their top-of-the-line iPhone 15 Pro Max handsets. You can already achieve this to a degree[^1] with tools like Lifecast[^2], but when Apple throws their billion-dollar marketing chops behind it, you get it integrated into a dominant ecosystem.
