Apple technology could create AR, VR content based on LiDAR mesh maps

Apple is continuing to refine its virtual and augmented reality technology, including work on a system that can map an AR or VR experience to a physical setting.

The company is thought to be working on multiple devices that rely on synthesized reality (SR), a broad category including both AR, VR, and mixed reality (MR). Apple is rumored to be developing both an AR "Apple Glass" device, as well as another MR visor device.

Both of those devices could use new methods and technology detailed in a patent application published Thursday and titled "Method and Device for Tailoring a Synthesized Reality Experience to a Physical Setting."

The patent application lays out a method that could obtain local environment data, including the spatial information of a "volumetric region around a user." It would then create a mesh map of that data, likely through LiDAR technology. From there, Apple's system could use the mesh map to composite or tailor an SR experience.

"For example, while a user is watching a movie in his/her living room on a television (TV), the user may wish to experience a more immersive version of the movie where portions of the user's living room may become part of the movie scenery," the patent reads, giving one example of a use case.

Other possibilities in the movie example include projecting portions of the movie onto walls or floors, or providing additional content like maps, graphs, or educational information. The SR system could also "skin" at least some of a living room to reflect content in the movie. Apple gives the example of an "at-home holodeck."

It doesn't stop there. The patent also includes uses tailored for AR or MR experiences. For example, presenting synthesized information to a user wearing a pair of glasses. Apple also notes that the system isn't restricted to head-mounted devices — tablets or smartphones could provide SR experiences too.

Apple also details how the experience could change as a user moves. Using a location tracking system, it could update the synthesized experience as someone moves around a room or environment.

Different display types that could use the SR generating system could include holographic, digital light processing, LCD, or OLED displays. The patent notes that users may also be able to interact with the environment or give voice or gesture commands to change it. It could different feedback mechanisms — like audio, haptic, skin shear, temperature — as a user interacts with their environment.

The patent lists Ian M. Richter; Maxime Meilland; and Patrick W. O'Keefe as its inventors. Of them, Richter has been named in past SR-related patents.

Apple files numerous patents on a weekly basis, so there's no guarantee that the technology in them will make it to market or any sense of a possible debut timeline. Given the state of Apple AR and Apple VR technology, however, there's a good chance this specific system could end up in a consumer device sooner than later.

The company is also expanding the use of its LiDAR technology, which can create depth maps of an environment. It's currently in use in the iPhone 12 Pro and iPad Pro lineups, and rumors suggest Apple could bring it to all iPhone models going forward.