A future Apple mixed-reality headset, like the rumored "Apple Glass," could leverage precision sensors to provide a highly immersive experience and allow users to control nearby electronic devices.
Credit: Oculus
Two patents, granted to Apple on Tuesday cover different aspects of a virtual- or mixed-reality headset or goggles. In addition to Apple's existing work on platforms like ARKit, Apple is thought to be working on at least one head-mounted device of some type that could provide MR or VR experiences.
The first patent, "Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking," is a foundational patent describing a suite of sensors that could provide high levels of immersion when in a virtual or mixed-reality setting.
The suite of sensors on a potential head-mounted device. Credit: Apple
As Apple points out in the patent, the "objective of immersion in a virtual world is to convince a user's mind to perceive a non-physical world as if it were real. The concept of reality here refers more to the notion of perceptual plausibility rather than representing a real world."
To provide that level of immersion, the patent contends that tracking of various elements is crucial That includes tracking of the head's position, the body's position, and the environment around a user.
The suite of immersion-providing sensors could include a pair of RGB cameras with infrared cut-off filters; another pair of mono camera sensors with near infrared filters; an inertial measurement unit; a time-of-flight sensor; a speckle pattern projector; and other associated components.
According to this patent, these components would work together in different ways. The cameras, for example, could provide a pass-through view for MR, but could also provide image data that would be processed to analyze and understand the environment.
Environmental analysis is essential for realistic object placement in augmented reality or MR, but the patent also lays out ways that a head-mounted device could understand a user better as well. That includes real-time tracking of a user's body, head, and their place in an environment.
All of this data collection and analysis takes processing power, and over a connection, could lead to high latency. For immersion purposes, the patent notes that some processing could be conducted on the head-mounted device itself for low-latency, with an external computing device providing additional power when needed.
Simon Fortin-Desch nes; Vincent Chapdelaine-Couture; Yan Cote; and Anthony Ghannoum are listed as the patent's inventors. Fortin-Desch nes and Chapdelaine-Couture have both worked on a couple of imaging-related patents for the Cupertino tech giant in the past.
In a separate patent, Apple describes a system that could allow a similar head-mounted device to control nearby electronics and computers.
A figure depicting in-air gestures that could control various smart or electronic devices. Credit: Apple
An electronic device, such as a head-mounted one, could also input-output components that would "provide an intuitive way for a user to control, gather information about, or otherwise communicate with external electronic devices."
"The input-output devices may include sensors for identifying external electronic devices within the user's environment, a display system for displaying information about the external electronic devices, and user input components such as touch sensors, force sensors, and other sensors for gathering touch input and/or gesture from a user's fingers. The electronic device may include communications circuitry for sending signals to and/or receiving signals from the external electronic devices based on the received user input," the patent reads.
As Apple notes, users may find it challenging to "identify and control which devices are connected and how the devices are connected. Typically, a user must navigate to a settings menu on each individual device, which then provides a list of external devices to which that device is connected."
To get mitigate this, Apple contends that devices could be displayed in an image or on a separate device. From here, users could view and tap on specific electronics within an environment to easily select them for settings purposes.
"A finger swipe from a first external electronic device to a second electronic device on the display or in the environment may cause the control circuitry to send wireless signals to at least one of the first and second external electronic devices to establish a wireless communications link between the first and second external electronic devices," the patent reads.
The system would not necessarily need to be a AR or MR-related patent. For example, a user could hold up an iPad and use its camera view to see and control various devices in a setting. However, the patent does specify that the control device could incorporate a "head-mounted display," with control gestures being made in the air. In other words, it hints at HomeKit functionality for "Apple Glass" or another HMD.
This patent lists Colin M. Ely; Erik G. de Jong; Stephen Brian Lynch; and Fletcher R. Rothkopf. All four have worked on various Apple-related patents in the past, including ones related to the Apple Watch as well as embedded sensors in OLED displays.
Apple files numerous patents on a weekly basis, and is granted many of them. Because of that, there's no indication of when -- or if -- technology described within them will make it to a consumer-facing device.