Vision Pro will turn any surface into a display with touch control

Vision Pro can make any surface appear to be a control, or an app display

Developers using Apple Vision Pro have learned they will be able to create controls and displays, and have them appear to be on any surface in the user's room.

Apple has been working on Vision Pro for a long time, and along the way it has applied for countless patents that were to do with it — even if it wasn't always obvious what the company's plans were. Now one intriguing patent application from 2020 has been revealed to be a part of Vision Pro that will help developers.

Developer Steve Troughton Smith has discovered that it's possible to pick a surface within the headset's field of view, and then place any app so that it appears to be actually on that surface.

"A natural way for humans to interact with (real) objects is to touch them with their hands," said Apple's 2020 patent application. "[Screens] that allow detecting and localizing touches on their surface are commonly known as touch screens and are nowadays common part of, e.g., smartphones and tablet computers."

Troughton Smith used an Apple Music app for his experimentation, but it can be any app and seemingly any controls. So where Apple's virtual keyboard for the Vision Pro is not practical for long typing sessions, it could be that a user's desk is turned into a keyboard.

That would still make for a stunted typing experience, as there would be no travel on the keys. But there are already keyboard projectors that use lasers to display a QWERTY layout directly onto a flat surface.

8 Comments

danox

said about 5 months ago

A long and steady drum beat from developers over the next 6 to 9 months priceless, taking advantage of a whole new Apple ecosystem………

tht

said about 5 months ago

Waiting to see if typing with 10 fingers will be possible, both with the "in-air" keyboard seen in the screenshots and videos and on a virtual keyboard on flat desktop surface. Ten finger typing is low on the priority list though as I assume Apple thinks people will just use a real keyboard.

Wondering how the dexterous you can be with the hand tracking. 10 finger typing is one example. Another is say, can you work a virtual Rubik's cube with your hands?

FileMakerFeller

said about 5 months ago

tht said:

Waiting to see if typing with 10 fingers will be possible, both with the "in-air" keyboard seen in the screenshots and videos and on a virtual keyboard on flat desktop surface. Ten finger typing is low on the priority list though as I assume Apple thinks people will just use a real keyboard.

Wondering how the dexterous you can be with the hand tracking. 10 finger typing is one example. Another is say, can you work a virtual Rubik's cube with your hands?

I suspect that the hand tracking won't be incredibly precise, but that won't be an issue if you are looking at the object because the eye tracking is reportedly phenomenal. We won't know for sure until devices get shipped.

williamh

said about 5 months ago

AppleInsider said:

So where Apple's virtual keyboard for the Vision Pro is not practical for long typing sessions, it could be that a user's desk is turned into a keyboard. That would still make for a stunted typing experience, as there would be no travel on the keys. But there are already keyboard projectors that use lasers to display a QWERTY layout directly onto a flat surface.

I think you COULD have key travel by overlaying a virtual keyboard on a real keyboard. It wouldn't matter if the keyboard was plugged in or not. You could type on the real keyboard and the Vision Pro would read the virtual one occupying the same space.

mike eggleston

said about 5 months ago

williamh said:

AppleInsider said:

So where Apple's virtual keyboard for the Vision Pro is not practical for long typing sessions, it could be that a user's desk is turned into a keyboard. That would still make for a stunted typing experience, as there would be no travel on the keys. But there are already keyboard projectors that use lasers to display a QWERTY layout directly onto a flat surface.

I think you COULD have key travel by overlaying a virtual keyboard on a real keyboard. It wouldn't matter if the keyboard was plugged in or not. You could type on the real keyboard and the Vision Pro would read the virtual one occupying the same space.

I won't lie, that seems like it could be an AWESOME alternative.