Apple aiming for immaculate images in AR glasses & VR headset

Apple has made little secret that it has plans for Virtual Reality and Augmented Reality, but a series of five new patent applications show just how detailed those plans have become. The patents, filed separately and each credited to different teams of inventors, cover everything from what you see when you're wearing a head-mounted display — to what you see when you're not.

Three of the five are chiefly concerned with making sure that the image presented to a user is crisp and clear, with even lighting and sufficient brightness. Each is also concerned with how this can be accomplished within the constraints of a head-mounted display.

Displaying high resolution AR or VR images

"Waveguided Display Systems," specifically covers a way of getting high quality images without the head-mounted display being nose-heavy with cumbersome equipment.

"Electronic devices with displays may be used to display content for a user," it explains. "If care is not taken, the components used in displaying content for a user in an electronic device may be unsightly and bulky and may not exhibit desired levels of optical performance."

VR gets around this by dividing the projector that creates the image, and the lens that the user sees through. For an Augmented Reality system, that image is combined with that of the real world that the user is looking at. Routing the AR image, and combining it with the real one, relies on an optical system called a waveguide. Apple plans to embed "beam splitter structures" to keep the different light levels from AR and reality consistent.

"An electronic device may have a display that emits image light, a waveguide, and an input coupler that couples the image light into the waveguide," continues the patent. "The beam splitter structures may replicate the beams across two dimensions (e.g., across the lateral area of the waveguide). In this way, an eye box may be provided with uniform-intensity light from the display across its area and for a wide field of view."

This waveguide patent is credited to ten inventors, including Guolin Peng, Eric J. Hansotte, and four more who also share credit on the previous "Optical Systems for Displays" patent.

Next, the separate patent application, "Scanning Display Systems," is also concerned with the image the user sees, but specifically about its brightness and resolution.

"It is challenging to form a projection and display system with sufficient optical brightness, display resolution, and compactness for a scalable use," says this patent, which also notes that head-mounted displays must be robust. "Additional care must be taken to consider user cases that include drop shock and thermal loads."

Mirrors could be damaged by misuse of the headset, but in some systems they can deformed in regular operation. The systems described use scanning, or rotating, mirrors and there are issues with the size of the mirrors plus the speed of rotation.

"In general," continues the patent, "it is desirable to provide the scanning mirrors in optical system as large a reflective area as possible (e.g., 1.2-2.0 mm in diameter) in order to maximize image resolution in the far-field domain."

"However, scanning the mirrors at high speeds can cause physical deformation in the mirror (particularly for relatively large mirrors), which serves to diverge the image light and thus limit image resolution in the far-field," it continues.

The system proposed here would instead use "an array of staggered light emitting elements arranged in diagonal rows and aligned vertical columns." The aim is to provide optical systems that can "overcome these difficulties to provide high resolution (low divergence) images at high frame rates while still allowing the optical system to fit within the constrained form factor of [a headset]."

Mixing real-world objects with VR

As well as displaying an AR image alongside any real-world view the user has, head-mounted displays need that user to be able to see and manipulate external objects. If they are watching an entirely virtual reality space, users cannot see devices such as controllers that they may even be holding.

"Tracking and Drift Correction," by inventor Mohamed Selim Ben Himane, aims to help with that.

"[Existing] systems are unable to adequately track the locations of such separate devices relative to the content providing devices,"says this patent, "and thus are unable to display such separate devices or representations of them to guide the users' interactions."

"For example, a user wearing a head-mounted device (HMD) presenting a CGR [Computer Generated Reality] environment would need to see a representation of his [or her] mobile phone in the CGR environment in order to use the touch screen of the mobile phone as an input device," it continues. "However, without being able to accurately, consistently, and efficiently track the relative location of the mobile phone to the HMD, the representation of the mobile phone cannot be displayed at locations in the CGR environment corresponding to the real-world locations."

In Apple's proposed system, the phone in the real world would be precisely tracked so that a virtual version could be displayed in the correct place. "When the user virtually touches a virtual button on the virtual remote control, the user actually touches a corresponding portion of the touch screen of the second device, which is recognized as input that controls or otherwise initiates an interaction with the virtual environment."

Audio issues

Each of these patents goes into great detail specifying methods and applications, but not all of them are solely concerned with the images that are displayed to a user. "Display Devices With Multimodal Audio," concentrates on what the user hears.

Alongside the near-eye displays that present images, the head-mounted display would have earpieces that can be adjusted to suit the type of audio being presented.

"Flexibility in audio operation is often desirable in that it allows for use of the system in a variety of settings or environments," says the patent, which is credited to five inventors. These include Robert D. Silfvast, Christopher T. Eubank, and Neal D. Evans, who were recently listed as inventors on a patent regarding removable earphones on the "Apple Glasses."

"For example, in the context of virtual reality (VR), a more immersive audio experience may be desirable (e.g., to block out or cancel external noise), whereas in the context of augmented reality (AR) or mixed reality (MR), external noise may be of less import," it says.

Head-mounted displays not worn on the head

Perhaps the most unexpected of the five patent applications is "Electronic Device With Inner Display and Externally Accessible Input-Output Device" by Paul X. Wang. This is about what happens with a head-mounted display when it is not being worn on the head.

The idea is that a user could take off such a headset, and still be able to use it in some fashion.

"In some situations," says the patent, "it can be challenging for a user to interact with an electronic device such as a head-mounted electronic device. When the head-mounted electronic device is not being worn, however, the user cannot view content on the display, which hinders the ability of the user to interact with the device and associated equipment."

Rather than projecting an image onto the head-mounted display's eyepieces, the system could project it out "onto nearby surfaces." For an example, Apple suggests that this could present a way for the user to control the device.

"The images may include a virtual keyboard with which a user may interact," it says.

Apple is exploring how an AR/VR head-mounted display will look and sound when worn on the head, plus how it can be utilized when not. Then these five new patent applications are broad-ranging, but they are just the latest of a series that goes back more than a decade.