'Apple Glass' could place virtual controls in the wearer's hand

An Apple AR device such as "Apple Glass," or another head-mounted display, may be able to detect when a wearer puts their hand out, and position a virtual control panel there.

Even if "Apple Glass" or another head-mounted display (HMD) is lighter and more fashionable than, say, an Oculus headset, it's still a headset. Apple doesn't want you to have to take it off, or search around with your hands, for the controls to use it.

The company has previously investigated turning any surface into a display with touch controls. Now it has also been revealed to be looking at ways of bringing controls to the user's hands — just virtually.

"Devices, Methods, and Graphical User Interfaces for Displaying Applications in Three-Dimensional Environments," is a newly-revealed patent application where the only thing longer than the title is the description. Seven credited inventors — including Jony Ive — detail their proposals in a mere 50,000 words.

At the heart of it, though, are two related concerns. First, how to know when an "Apple Glass" wearer has turned their hand just so. Then, second, how to present them with a control panel that appears to be in their grasp.

The first issue covers problems that sound as if they may have been solved, or at least investigated, over the Apple Watch.

"A computer system detects a wrist," begins the patent application. " In accordance with a determination that first criteria that require an inner side of the wrist facing toward a viewpoint are met, the computer system displays a first user interface object..."

In other words, if any system detects that the wearer has turned their wrist to look at something, it displays something for them to look at. With the Apple Watch, that would be a face as the user turns the outside of their wrist toward them.

With a proposed virtual control system, it could be that the wearer has turned their wrist away, such that they are then able to open their hand. "While displaying the first user interface object, the computer system detects that the wrist's position or orientation has changed to satisfying second criteria that requires an outer side of the wrist facing toward the viewpoint," continues the patent.

"In response, the computer system switches from displaying the first user interface object at the first position to displaying a second user interface object including a plurality of controls for controlling functions at a second position corresponding to a location on a back of a hand attached to the wrist," says Apple.

Left: the Luke Skywalker version

It's far from a straightforward process, though, as if Apple weren't careful, users who gesticulate a lot could keep getting control panels thrown at them.

So the patent application spends a lot of time and attention on determining a wearer's intent, and then providing what that user wants. It's not especially concerned with what specific controls it displays, more with the process of displaying it.

That means a system that can determine when a user wants controls, can also then confer with an AR application to provide the relevant controls.

Keep up with everything Apple in the weekly AppleInsider Podcast — and get a fast news update from AppleInsider Daily. Just say, "Hey, Siri," to your HomePod mini and ask for these podcasts, and our latest HomeKit Insider episode too.

If you want an ad-free main AppleInsider Podcast experience, you can support the AppleInsider podcast by subscribing for $5 per month through Apple's Podcasts app, or via Patreon if you prefer any other podcast player.