Developers using Apple Vision Pro have learned they will be able to create controls and displays, and have them appear to be on any surface in the user's room.
Apple has been working on Vision Pro for a long time, and along the way it has applied for countless patents that were to do with it — even if it wasn't always obvious what the company's plans were. Now one intriguing patent application from 2020 has been revealed to be a part of Vision Pro that will help developers.
Developer Steve Troughton Smith has discovered that it's possible to pick a surface within the headset's field of view, and then place any app so that it appears to be actually on that surface.
"A natural way for humans to interact with (real) objects is to touch them with their hands," said Apple's 2020 patent application. "[Screens] that allow detecting and localizing touches on their surface are commonly known as touch screens and are nowadays common part of, e.g., smartphones and tablet computers."
Troughton Smith used an Apple Music app for his experimentation, but it can be any app and seemingly any controls. So where Apple's virtual keyboard for the Vision Pro is not practical for long typing sessions, it could be that a user's desk is turned into a keyboard.
That would still make for a stunted typing experience, as there would be no travel on the keys. But there are already keyboard projectors that use lasers to display a QWERTY layout directly onto a flat surface.
8 Comments
A long and steady drum beat from developers over the next 6 to 9 months priceless, taking advantage of a whole new Apple ecosystem………
Waiting to see if typing with 10 fingers will be possible, both with the "in-air" keyboard seen in the screenshots and videos and on a virtual keyboard on flat desktop surface. Ten finger typing is low on the priority list though as I assume Apple thinks people will just use a real keyboard.
Wondering how the dexterous you can be with the hand tracking. 10 finger typing is one example. Another is say, can you work a virtual Rubik's cube with your hands?