Apple is working on fingertip and face mapping tools for VR and AR

Two new Apple patents describe methods to improve how a user can touch and interact with VR objects, plus how headsets could be more easily adjusted to fit better.

Apple continues to investigate methods of making better Virtual and Augmented Reality experiences for users. Two separate new patents both attempt to make wearing VR/AR technology less cumbersome and more useful.

The first of two, "Computer Systems with Finger Devices," is concerned with making it easier for users to interact with items in a virtual space.

"In virtual reality systems, force-feedback gloves can be used to control virtual objects," it says. "Cellular telephones may have touch screen displays and vibrators that are used to create haptic feedback in response to touch input. Devices such as these may not be convenient for a user, may be cumbersome or uncomfortable, or may provide inadequate feedback."

Instead of large gloves that are awkward to wear, or haptic feedback displays that can only provide minimal feedback, Apple proposes slim sensors that can be worn on the finger.

"A touch sensor may be mounted in a finger device housing to gather input from an external object as the object moves along an exterior surface of the housing," it continues. "The external object may be, for example, the tip of finger other than a finger received within the finger device."

"Finger input from a user may be used to manipulate virtual objects," it says. "For example, a virtual object corresponding to a movable control can be moved by a user based on finger input gathered using a finger device or a displayed menu option can be selected. In some arrangements, a user may interact with real-world objects while computer-generated content is overlaid over some or all of the objects."

An example of a sensor device on a finger

Of the 20 inventors listed on this finger device patent, Chang Zhang and Tianjia Sun have recently also been named on "Cross Device Interactions." Similarly, Paul X. Wang, is credited on "Finger-Mounted Device with Fabric," while Nicolai Georg is an inventor of "Techniques for Locating Virtual Objects Relative to Real Physical Objects."

Part of being able to locate and manipulate virtual objects concerns being able to see them, and a separate Apple patent looks to address one of the chief obstacles to AR and VR. Bulky headsets may eventually be replaced by glasses, but for now they are required. Their cost means they are made to fit as many people as possible, and their awkwardness means it's difficult to simply adjust them to suit each individual user.

"It can be challenging to ensure that a head-mounted device fits a user's head satisfactorily," says Apple in "Adjustable Electronic Device System With Facial Mapping." "If care is not taken, a head-mounted device will not fit properly and will be uncomfortable to use."

Apple proposes that such headsets could use sensors similar to those in Face ID to scan the user's face.

"The head-mounted device or external equipment that communicates with the head-mounted device may include a three-dimensional image sensor," it explains. "The three-dimensional image sensor may capture a three-dimensional image of a user's face."

"Control circuitry may analyze the three-dimensional image to determine which of multiple versions of a customized face-fit module should be used in the head-mounted device to optimize the fit of the head-mounted device on the user's head," it continues.

The headset would not change or adjust itself around a person's face when they put it on, however. Instead, an initial scan would display information on separate, external screens to help the user or support staff select the best "customized face-fit module."

"[The module] may be selected based on user facial feature characteristics in the three-dimensional image of the user's face," says the patent, "such as facial size and the shapes of a user's forehead, nose, and cheeks."

Detail showing a headset with fixed elements and adjustable or replaceable ones to make a better fit

A headset will have its main body remain the same but include swappable elements that can be changed to best match this "face-fit" calculation.

"After identifying which version of the face-fit module is to be used by the user," it continues, "that version of the face-fit module may be coupled to a non-custom portion of a main unit for the head-mounted device using magnets or other coupling structures."

Apple has previously been predicted to release AR Smart Glasses this year.

It has also previously patented a system for an AR user to wear an "iRing" on their fingers.