Apple refining AR and VR tech, including sensors to detect food freshness

Apple is refining technology for use in a head-mounted device like the rumored "Apple Glass," including infrared sensors that could be used to measure the freshness or calorie content of food.

That's according to a trio of Apple patent applications published by the U.S. Patent and Trademark Office Thursday. All three detail technology that could be used in an Apple VR or AR device, such as the rumored visor-like mixed-reality headset.

Optical infrared sensing for food detection

Credit: Apple

The first patent, titled "Electronic devices with beam-steered infrared light sensing," outlines a system that uses infrared sensors to gather the light spectrum of an object. It could include beam steerers and adjustable components like lenses and polarizers.

"Infrared spectra, visible camera images, and/or data from other sensors may be used alone and in combination for characterizing target objects, so that notifications can be provided to a user and other actions taken," the patent reads.

That's similar to existing technology found in Face ID. What's interesting about the new patent application is its potential uses, particularly in a head-mounted Apple AR device.

For example, Apple says that the system could be used in tasks such as image recognition. An example is food. "A user may desire to evaluate food items using optical sensing (e.g., to evaluate ripeness, caloric content, etc.). This preference may be stored in memory in [the device] or may be selected dynamically by a user," it says.

According to Apple, this image recognition could then be used to trigger information stored in a databse. That data could include food characteristics such as freshness, fat content, food type, sweetness, and other metrics.

It wouldn't end at food, however. "In general, any type of object can be analyzed using the optical sensor (e.g., plants, animals, inanimate objects, food, paint, portions of a house or other building, automobile systems, electronic equipment, furniture, etc.)" the patent adds. Additionally, because infrared light can excite vibrational and rotational molecular resonances in a material, the system could be used to provide information on a user's body. Apple says, combined with other sensors, the system could monitor a user's breath for even more data

This patent lists Sheng Liu; Guocheng Shao; and Kenneth J. Vampola as its inventors. Of them, Shao and Vampola have been named in past Apple patents related to ambient light sensing technology.

Adaptable Facial Interfaces for HMDs

Credit: Apple

The second Apple patent application published Thursday, "Head-Mounted Display Unit With Adaptable Facial Interface," covers an interface system meant to be worn on the head of a user and comprising two separate parts.

In the patent, Apple says that a head-mounted display could be composed of a display assembly and a facial interface. The latter component would be coupled to the display assembly and its main purpose would be to support it while worn on the face.

"The upper portion engages an upper facial region above eyes of the user. The lower portion that engages a lower facial region below the eyes of the user. The lower portion has a lower shear compliance in that is greater than an upper shear compliance of the upper portion," the patent reads.

This facial interface could contain light seals meant to keep ambient light out. It could feature a pivot mechanism to "provide vertical compliance between the display assembly and the lower facial region." A stiff portion could provide support on a user's nose.

In other words, this describes a system meant to enclose a user's eyes and provide an environment for virtual reality or mixed reality experiences. The facial interface are similar to the foam gaskets on a pair of swim goggles. They sit flush on the face and allow space between a user's head and a head-mounted device's displays.

Apple's system, of course, appears more complicated than simple foam gaskets or seals. It goes to great lengths to provide a comfortable and stable fit, and could include more complicated pivot mechanisms — seemingly for adjustability.

The patent lists Dustin A. Hatfield and Kristi E. Bauerly as its inventors. Bauerly has been named in both AR headset-related patents, as well as a patent for "an improved set of headphones."

Eye-tracking refinement

Credit: Apple

Lastly, Apple is continuing to refine its eye-tracking technology for both AR and VR applications. The patent application published Thursday is far from the first Apple has filed for.

Compared to past eye-tracking patent applications, there appear to be a few differences.

Like those past systems, Apple says an eye-tracking suite would use at least one camera and an illumination source. Instead of "hot mirrors" mentioned in past patents, Apple's new patent application mentions diffraction gratings as a way to receive light.

"The diffraction gratings redirect or reflect at least a portion of infrared light reflected off the user's eyes, while allowing visible light to pass. The cameras capture images of the user's eyes from the infrared light that is redirected or reflected by the diffraction gratings," the patent reads.

There's clear evidence of Apple's refinement, too. In one portion, the company says a system that integrates transmissive or reflection diffraction gratings in the eyepieces allows for a slimmer headset than one that includes hot mirrors.

Additionally, "Integrating reflective gratings in the eyepieces allows the user's eyes to be imaged through the eyepieces while improving the images (e.g., by reducing distortion) captured by the IR cameras when compared to systems in which the IR cameras view the user's eyes directly through the eyepieces."

The company also says that gratings also improve the viewing angle of the IR cameras when compared to systems in which IR cameras view a user's eyes directly.

This patent lists Kathrin Berkner-Cieslicki; Se Baek Oh; Scott M. DeLapp; Christopher F. Griffo; Bradley C. Steele; Ting Sun; Kenichi Saito; and Noah D. Bedard as its inventors. Of them, Berkner-Cieslicki, Oh, and Saito have been named in past Apple AR patents.