Apple refining room & floor mapping for mobile devices, virtual reality headsets

Apple is researching several types of 3D floor-mapping systems that could integrate with a head-mounted device like a virtual reality headset or "Apple Glass."

In a trio of patent applications published by the U.S. Patent and Trademark Office on Thursday, Apple details various methods for generating three-dimensional representations of physical environments using sensor data.

All three patents share the same name, "Floorplan Generation based on Room Scanning," but their exact details differ. For example, one patent lays out a system that could obtain 3D data based on depth-mapping and light intensity image data using a scanning process, while another adds that this type of data can be obtained through a live camera feed.

It appears that the specific floor-mapping goals are slightly different, too. One of the patents details how this data can identify floor-to-ceiling walls in a physical environment based on 2D semantic data, while the second and third patent focuses on corners and small walls.

The goal of obtaining and parsing this environment data would be to provide accurate and efficient measurements or floorpans using a mobile device. At present, Apple notes, existing techniques don't allow for good floor plan generation from mobile devices.

Apple notes that this floor-mapping data can be provided to a user through a specific device like an iPhone, iPad, or something that they wear on their head.

A figure representing the collection of room data using a mobile device.

"The system flow of the example environment 400 can be displayed on a device (e.g., device 120 of FIGS. 1 and 3) that has a screen for displaying images and/or a screen for viewing stereoscopic images such as a head-mounted display (HMD)," says a sentence included in all three patents.

The patents note that 3D representations of a physical environment could be generated by sensor data, such as image or depth sensor information. That's the type of data modern iPhones are equipped to collect. Presumably, an Apple head-mounted device would be as well.

"In some implementations, a floorplan includes a 2D top-down view of a room. A floorplan may graphically depict a boundary of a room, e.g., by graphically depicting walls, barriers, or other limitations of the extent of a room, using lines or other graphical features," the patent reads.

"A floorplan may graphically depict the locations and geometries of wall features such as wall edges, doors, and windows."

The patents name Feng Tang, Afshin Dehghan, Kai Kang, Yang Yang, Yikang Liao, and Guangyu Zhao as their inventors. Among them, Feng Tang has worked on Face ID systems for Apple, which also relates to using sensors to generate 3D models.

Apple has filed patent applications related to indoor floor mapping in the past, including for use cases specifically related to virtual or augmented reality. In 2020, it applied for a patent detailing a system that could prevent users from bumping into walls or furniture when using an Apple VR device.

The Cupertino tech giant is thought to be working on multiple head-mounted devices, including a mixed-reality visor that could immerse a user in a virtual or partially virtual environment and a lighter pair of glasses that would overlay digital information onto the physical world.

Apple files for numerous patents on a weekly basis, so they're a poor indicator of the company's current plans or the time to market for new features or devices.