Apple AR motion controllers may detect fingers without covering the hand

Apple's AR and VR systems could use wearable controllers with extending sections, allowing for the sensing of the user's fingers pressing a surface without covering the fingertips.

Augmented reality and virtual reality introduce numerous interaction challenges, in part caused by the user's possible inability to see a real-world button to press if their vision is obscured.

Current systems rely on hand-held controllers with a selection of buttons and sensors to detect the user's hand positions, and in some cases, even the fingers. Attempts have also been made to enable hand tracking using gloves, freeing the hand to grasp and interact with other things.

Even gloves provide their own issues, as they can still cover the entire user's hands, reducing the sensation of touch at the fingertips. Since such finger-detection systems typically rely on some sort of temporary placement on the fingers themselves, it makes designing a system that accomplishes the task quite tricky.

In a patent granted to Apple by the U.S. Patent and Trademark Office on Tuesday titled "Electronic device system with controllers," Apple attempts to solve the problem with more refined hand-held controller concepts.

The hand-gripped controller would allow for free monitored finger movement.

The patent suggests a number of different solutions, but they all seem to have the same outcome of leaving the fingers as unencumbered as possible. Finger detection is possible with each, but typically with the result of keeping the fingertip free.

The basic design appears to involve a tubular body gripped by the thumb and fingers, with ring sections for the fingers to slip through, like a form of knuckleduster. The main differences in design largely relate to elements that stem from the rings, with a view to interacting with the fingers.

Some designs suggest a form of sleeve covering the sides and top of the finger, while leaving the sensitive underside open. A pinching of the sides could allow the mechanism to stay attached to the finger, and in turn enabling a detection of finger movement.

The controller could be gripped by the thumb, but have holes for fingers.

Others also propose extendable stalks with onboard sensors, for detecting the range of the finger without actually touching it. This may also offer the opportunity for the sensor to detect when a user touches another surface, which can help aid with interactivity.

A wristband could be used to both anchor the mechanism on the hand and to provide another mounting point for electronics and sensors.

The list of sensors used to detect the finger and external surfaces could include optical sensors, radio-frequency sensors, acoustic sensors, two-dimensional force sensors, and motion sensors.

This is all on top of other typical sensors used for hand-held controllers, which can range from radio-based communications with a host device to accelerometers and other positioning elements.

The sensor many not necessarily need to touch the finger, and may even sense surface touches.

The patent lists its inventors as Paul X. Wang and Pol Pla I Conesa. It was originally filed on March 24, 2020.

Apple files many patent applications on a weekly basis, but while the existence of a filing indicates areas of interest for its research and development teams, it doesn't guarantee the idea will appear in a future product or service.

Many filings in the past have suggested some form of controller or finger sensing mechanism is in development by Apple. Such systems could be employed in its long-rumored AR and VR headsets, or even the "Apple Glass" AR smartglasses.

In February 2019, Apple applied for "Systems for modifying finger sensations during finger press input events," which used a form of clamp to lightly squeeze the sides of a finger to provide haptic feedback when typing or pressing a surface. This is similar to the mechanism described by Apple in the latest patent, though without the squeezing haptic element.

November 2020 brought with it the idea that Apple may be creating an "Expanding ring device" that could cover an entire finger. Doing so and using force sensors could allow a similar finger movement tracking while keeping the fingertip free.

Another smart glove patent in February 2021 proposed the use of a magnetic field generator and fingertip sensors to determine finger movements and placement in relation to the palm of the hand.

Stay on top of all Apple news right from your HomePod. Say, "Hey, Siri, play AppleInsider," and you'll get latest AppleInsider Podcast. Or ask your HomePod mini for "AppleInsider Daily" instead and you'll hear a fast update direct from our news team. And, if you're interested in Apple-centric home automation, say "Hey, Siri, play HomeKit Insider," and you'll be listening to our newest specialized podcast in moments.