Apple Vision Pro research and early work was hidden in plain sight

Why didn't we see what was right in front of our faces?

Over the last decade, there must have been hundreds of Apple patents and patent applications that were immediately and indisputably concerned with headsets. They even include the word "headset."

But there were also so many other patents that did not say "headset," and yet we now know that's what they were really for.

Such as one from 2015, eight years ago now, in which it appeared that Apple wanted to create a Mac keyboard out of thin air. Some kind of camera or sensor would be on the Mac's screen, and the user could make a typing gesture in front of it.

Data from the cameras would provide a "best guess of the user's intended input," said the original filing reads, which seemingly would allow Mac users to type more quickly.

Flash forward to Vision Pro, and it comes with exactly that kind of virtual keyboard. Rather than a camera facing out from a Mac to its user, it uses cameras looking from the Vision Pro wearer's perspective, out toward their fingers.

Or in 2020, it appeared that some future iMac would be able to have its display spill out onto the walls behind the device. Assuming you had a plain wall or a good splashback, the desktop could appear to extend beyond the boundaries of the Mac screen.

That would involve projectors built into the iMac, but it would also rely on the device being able to allow for screen displays to be wider than on the physical Mac. Now Vision Pro lets you take your Mac's screen and resize it, reposition it, in just the same way.

Gestures in thin air

One of the most impressive and Apple-like features of Vision Pro is that it does not come with any kind of separate controller. Instead, you just shrug, or you press two finger together — and you can do that pretty much wherever your hands are.

You couldn't pinch your fingers together behind your back, but you can do it while casually sitting with your hands and arm to your side.

It's just that we should have guessed that no later than 2020. That's when a patent application called "Electronic Device with Sensing Strip" told us.

We just thought it was to do with a future MacBook Pro, since Apple worked quite hard to make us think so. Despite drawings that implied a laptop, some of the descriptions are now unmistakeable.

"An electronic device may have an elongated sensing strip," said the patent application. "Control circuitry may use the sensing strip to gather air gesture input from the fingers or other body part of a user."

It's not only iPhones that have cameras

Once you suspect that Apple has been hiding headset details in plain sight, you do start seeing conspiracy theories everywhere. Not every patent can be a secret headset patent, but so very many concern technologies that we can now see must inform Vision Pro as well as any other Apple device.

In 2021, there was a patent application called "System For Producing A Continuous Image From Separate Image Sources." At the time, it looked like it was quite a cumbersome idea of having one person being able to take a photograph using a whole group's iPhones, all at once.

Perhaps it is that, but while much of the patent talked about mounting multiple devices, some of it was about the stitching together of images from all of these cameras.

"A photographer, videographer, or other person may desire to capture images using several image-capture devices," said Apple, "and combine the images captured by each of the individual devices into one continuous image."

"The combined, continuous image may have a greater field-of-view and include more image data than the individual images captured by the image-capture devices," continued the patent application.

We won't know just how Vision Pro's recording features work until it is released, but it sounds like this was paving the way.

Bezels and thin displays

There was reportedly a long-standing difference of opinion within Apple over whether the headset should come in two pieces or not. Ultimately, it appears a compromise was reached with just the battery being a separate segment, rather than any of the controls.

But whether it was to be one device, or the user would have to carry a connected iPhone as well, there must inevitably have been a pressure to make everything thin.

Such as the bezels around the screens. Apple Vision Pro still resembles goggles, but it's thin enough to be closer to ski glasses than most of its rivals.

To make thin bezels around the screen, and also to hide sensors behind the glass, Apple had another patent in 2021. "Sensing System for Detection of Light Incident to a Light Emitting Layer of an Electronic Device Display," cut the bezels, and embedded sensors.

""[This] protective outer cover defines an exterior surface of a housing of the electronic device," said Apple's patent application. "An electronic device can also include a light sensing system, such as an ambient light sensor."

"Typically, a light sensing system is positioned below the protective outer cover, adjacent to, and separated from, a light emitting region of the display stack," it continued.

Detecting the user

It was obvious that a headset would be a more personal device than a phone, and that it would have be adjustable to suit different levels of eyesight. What was less obvious was that Apple would seemingly limit the device to a user and one guest.

Then Apple didn't exactly spend a lot of time at WWDC talking about biometric identification. We now know that Vision Pro uses a new Optic ID system to verify the user, much like all the smart glasses in "Mission: Impossible."

Maybe we couldn't have guessed the term Optic ID, but we should have guessed at least Face ID would be involved. Ultimately there would be patents and patent applications that were more blatantly about headset biometrics, but in 2020 there was one about verifying identity that was ostensibly for iPhone users.

So that's why we got Spatial Audio

Spatial audio was also ostensibly for iPhone users, but now we know — and, for once, we did guess. In 2021, one of several patent applications regarding spatial audio (SR) actually talked about its use in AR.

"For example, a SR system may detect an individual walking a few paces forward and, responsive thereto, adjust graphics and audio presented to the individual in a manner similar to how such scenery and sounds would change in a physical setting," said the patent application.

"For example, an individual may interact with and/or sense aural objects that create a multi-dimensional (e.g., three dimensional) or spatial aural setting, and/or enable aural transparency," it continued

Forget smart rings

It's debatable which rumor was more keenly followed, the "Apple Glass" headset or the Apple Car. But behind them both was the equally long-lasting rumor that Apple was working on a smart ring.

Some patents involved extending a smart ring to cover almost all of a finger, and others concerned accessories for such a device.

But there were two patent applications in 2020 that in retrospect now seem so obviously intended to be about headsets. A patent application for a "wearable ring device" was refined in 2020 to be about a "wireless finger-worn device," that was intended to control "an external electronic device."

Also in 2020, though, there was a patent application for what was called an "expandable ring device."

Notice that there's no smokescreen here. It specifically says that the wearable device is part of a system that also includes a "head-mounted device comprising a display viewable by the user."

The headset would feature "control circuitry configured to adjust the displayed images based on the received data." And the system as a whole would support "bidirectional communication between the ring device and the head-mounted device."

Given these, it's easy to see now that the one where a ring was extended into covering a whole finger could be to give a kind of headset trackpad. Then the granted application plainly concerns smart ring accessories actually talks about virtual reality, too.

"As the user is viewing virtual content... the finger devices may be used in gathering finger input," it says.

As we now know, Vision Pro does not come with any external control devices. But also as we now know, finger movement is central to the operation of the headset.

It's just that instead of some wireless connection between a ring and the headset, Apple Vision Pro cameras watch for the user pinching their fingers together.

So while this can't ever be definite, it does look likely that Apple researched smart rings specifically for use with its headset. And now that Vision Pro has a better system, the ring research may never appear in a shipping device.

Patent pending

All patent applications, by any firm, are written so that they are close to preposterously wide-ranging. The aim is lock out rivals from using the idea, or even doing anything similar.

So it's conceivable that any one example of a patent appearing to really be about the headset is a chance result of just how wide-ranging it was written to be.

But the sheer volume of research that has been hiding in plain sight mean it can't all be coincidence.

Perhaps Apple really will put thin-air gesture sensors in a MacBook Pro, and certainly all work on smaller bezels must benefit every one of its devices.

Or maybe Apple will decide it can't waste all that effort and chooses to release smart wedding rings that detect infidelity.

But we've been listening to Apple AR headset rumors for so long, and it now looks like we should've seen it coming in far more detail than we did.