Apple AR will intelligently identify and annotate items of interest

As the wearer of "Apple Glass," or any Apple AR device, sees an object they're interested in, Apple wants the system to understand that and provide useful details.

Amongst its very many patents regarding every possible detail of Apple AR, Apple has applied for one that builds on its comparison shopping research. That was a proposal where if a user wearing "Apple Glass" held up two items, such as iPhones, the system could present pop-up comparison charts.

Now the newly-revealed "Method, Device, and System for Delivering Recommendations," patent application wants to extend that. The aim is that whenever a wearer sees an object they're interested in, "Apple Glass" could display useful pop-up information.

The patent application is really concerned primarily with how the decision is made to offer the information, rather than necessarily what that information is. And despite the word "recommendation" in the title, it doesn't refer to, say, movies the wearer might like.

Instead, the word recommendation here refers to how various sensors and sources calculate the likelihood that the user is interested, and then offers that information to the system. Then, perhaps in combination with other data available, "Apple Glass" can elect to display it or not.

So for instance, the "Apple Glass" system could use gaze detection to see what a user is looking at in an environment that may be a mix of AR and reality. The patent application uses the example of a cupcake, which may be a real one or, disappointingly, a computer-generated (CGR) one.

"For example, a cupcake recipe or nutritional information for a cupcake are generated and displayed to the user when a cupcake is recognized within the user's field of view," says the patent application. That could mean the entire field of view is examined, or that it is narrowed down by gaze detection.

Having identified a cupcake and delivered that information to the "Apple Glass" system, the system could then calculate the odds of the user wanting to know more. Apple's example is that it could, for instance, know from medical history that the user has some condition that means they shouldn't eat cupcakes.

In that case, "Apple Glass" might even display a "forbidden" icon, a circle with a line through it, over the view of the cupcake.

However, once an object has been identified and the information can be combined with other available data, there are countless possibilities. Maybe the user has some of the ingredients for making cupcakes in their Reminders app, in which case the system might then display a recipe for how to make the one they're looking at.

The possible applications all come down, though, to this business of identifying an object and determining whether more information is wanted. "The user-specific content is generated and displayed to a user based on likelihoods of user interests," says Apple.

No cupcakes for you, not if "Apple Glass" knows about your dietary requirements.

There is definitely an issue of convenience here as the patent application says doing this automatically would "allow the user to remain immersed in their experience without having to manually enter in search queries or indicate preferences."

More, the "seamless integration also reduces power usage and improves battery life" because it enables "use the device more quickly and efficiently."

Indicating interesting AR objects with colored markers

A separate newly-revealed patent application, "Colored Visual Markers for Variable Use," also features a way to distinguish between objects in a mixed reality environment. In this case, items in the environment can be labelled with markers, which might be simple color swatches, or QR codes.

"[Visual] are 2D objects that encode binary data such as strings (e.g., QR codes) or other payloads used to access remotely-based experiences (e.g., remotely-based experiences)," says the patent application. "In some implementations, the links to the remotely-based experiences include links to initiate payments (e.g., sanctioned payment endpoints), links to websites (e.g., URLs), or links that launch into web-based experiences."

So you could have a virtual "Pay Here" marker, or you could choose a virtual object that takes you through to some other environment.

"[The visual markers may be] used to launch only into or link only to sanctioned remotely-based experiences authorized by the creator of the visual marker," continues Apple. "In some implementations, the creator of the visual markers includes an entity that designs the visual marker, an entity that prints (e.g., makes) the visual marker (e.g., developer), as well as an entity that manages/hosts the visual markers."

Detail from the patent showing one mechanism for adjusting "Apple Glass" for precision optical alignment

Keeping Head Mounted Displays functioning correctly

Both of these inventions suppose that the user is likely to be looking around an environment. A third patent application wants to make sure that no matter how quickly they turn their head, the head mounted display (HMD) maintains "optimal alignment of optical components".

"Optical Alignment for Head-Mountable Device," is concerned partly with how precision is achieved during manufacture, but it's chiefly about keeping that in general use.

It goes into around 11,000 words of detail, but the summary is that "Apple Glass," or other devices, "can include adjustment mechanisms."

"The alignment mechanisms can be integrated into the head-mountable device itself," continues Apple. "A light projecting display element can be adjustable based on operation of one or more actuators within the head-mountable device (e.g., within an arm) to adjust a position and/or orientation of the light projecting display element relative to the waveguide onto which it projects light."

"The adjustment mechanisms can adjust the display element during initial assembly and/or be operated by actuators that actively adjust the alignment as needed over time," says the patent application.

Apple applies for hundreds of patents and, even if granted, it does not mean that any of them will result in shipping devices. However, the sheer volume of patents regarding "Apple Glass," "Apple AR," and so on, at least means that Apple is extremely focused on the details of how such systems could work.

Stay on top of all Apple news right from your HomePod. Say, "Hey, Siri, play AppleInsider," and you'll get latest AppleInsider Podcast. Or ask your HomePod mini for "AppleInsider Daily" instead and you'll hear a fast update direct from our news team. And, if you're interested in Apple-centric home automation, say "Hey, Siri, play HomeKit Insider," and you'll be listening to our newest specialized podcast in moments.