Apple's push for 3D will require voxel map of the world, claims VR evangelist

Apple is thought to be working on "Apple Glass," wearable augmented reality or smart glasses that can provide users with a mix of digitally-produced and real-world views. While the bulk of rumors and speculation have revolved around the hardware, Robert Scoble has forecast the kind of user experience to expect from Apple over the coming years.

Starting at WWDC 2021, Apple will start to make announcements throughout the year about technological advances that it will be incorporating into its products, writes Scoble on Saturday. The announcements will apparently lead into "many new products, services, and experiences that will come for decades."

According to Scoble, this will be a fourth paradigm shift for Apple, following its introduction of the personal computer, the graphical user interface, and the iPhone. While many of the changes will center around headsets and glasses, Scoble believes it will be "way deeper" than the hardware, with Apple moving into a number of new areas.

The list of changes is vast, with arguably the most ambitious being a "realtime 3D map of the entire world." The map will be a "basis for the new Apple," and will be used to power its products and user experiences.

The sources, identified as "developers who are building for Apple," also apparently offered that many shipping products have elements that could help Apple's new strategy take form. Such items include the Neural Engine included in the M1 chip that is dedicated to AI workloads, but "hasn't really been used much yet," as well as Apple's buy-in to Ultra Wide-band.

Scoble believes the Apple Mapping car is being used to map the world with voxels, dividing Earth into small cubes. Each cube allegedly has a unique identifier, a virtualized microphone and speaker, a virtualized display, a virtualized database, and a virtualized computer.

The scans would also be performed in buildings by its products, taking advantage of things like computer vision and LiDAR on an iPhone or iPad.

By building the map and introducing hardware that can view and take advantage of the data, Apple will be able to offer other major applications.

As part of this, Scoble says Apple will be working to overhaul Siri to react to what the user is looking at through a headset or another device. By taking advantage of computer vision or the map, Siri will be able to infer meaning from what the user is looking at, and to answer questions like "How much are 20 of these on Amazon?"

The overhauled Siri will also be more flexible and hooked into more data sources. For example, Siri may access a Foursquare API to answer a question about check-ins at a hotel.

The "HoloDeck" will be a "TV killer" application, Scoble suggests, when the headset becomes available in 2022. Claiming to have been given a "sneak look" at software being built for the headset, he believes it will introduce "a stunning set of capabilities that will blow away any physical TV you have ever seen"

The headset or visor will "cover your eyes and your ears," showing a representation of the living room and playing a representation of environmental audio to the user. It is offered that this could be similar to the "transparency" mode of the AirPods Max.

The screens will apparently be made of two 8K "Sony chips," which will allow Apple to virtualize TVs to be bigger than any in a store, and in high quantities. By taking advantage of Ultra Wideband, these screens may also be better locked into position in the environment, and for devices to automatically know where they are located in the world without requiring resources.

Scoble offers that this is the equivalent of Apple putting a TV "on every inch of the world," which can enable new experiences. For example, a Super Bowl halftime show could pop out of the TV and show the 3D performer in front of the user, rather than on a screen.

A volumetric football service is also apparently being prepared, giving the experience of being sat at the stadium through a volumetric surface alongside 2D screens.

On sound, Scoble believes the experience will offer a system with extremely high numbers of channels. In the case of a VR band, a user would feasibly be able to stand in the middle and hear instruments in different positions.

The Holodeck-style experience will apparently be announced sometime before the end of 2021, and could be teased at WWDC itself, the VR evangelist believes.

For CarPlay, a 3D audio experience can allow individual buttons or parts of a road to "talk to you." While such a system could be included in the "Apple Car," Scoble insists it could land in current vehicles in 2021.

Scoble also suggests a new portable gaming device is on the way that would be able to work with headsets, as well as a next-level noise-cancellation system.

"Put it all together and you can see Apple is about to unleash a new paradigm-shifting strategy. One that will change all of our lives very deeply and bring us many exciting new things," says Scoble. "This has deep implications on the future of a number of companies. Spotify looks threatened. So does Google."

Stay on top of all Apple news right from your HomePod. Say, "Hey, Siri, play AppleInsider," and you'll get latest AppleInsider Podcast. Or ask your HomePod mini for "AppleInsider Daily" instead and you'll hear a fast update direct from our news team. And, if you're interested in Apple-centric home automation, say "Hey, Siri, play HomeKit Insider," and you'll be listening to our newest specialized podcast in moments.