How virtual and augmented reality could play a role in Apple's future

Apple CEO Tim Cook suggests that the company is investigating the possibilities of virtual reality and augmented reality. Here's an explanation of what the technology is, how Apple could adopt it, and how it could be practically implemented in a user's daily life.

Video gaming and education has depended for many decades now on computer simulations of the real world. Physics engines for engineers visualizing and testing products are both billion-dollar industries. Similar technologies are utilized in the gaming market, which is making more money that the entire movie industry and has for several years.

Virtual reality promises users a more immersive experience, by sinking a user in the environment with video and audio cues. Instead of a large screen presenting a singular vision of a scene, a user of VR can look around at the scene, and interact with the environment, to the limits of the programmed scenario.

Apple CEO Tim Cook has made remarks, most recently on Sunday, heralding the technology as one of the cornerstones of the company's future.

While similar, any discussion of the future of Virtual reality (VR) or augmented reality (AR) needs to have the differences between the two spelled out. Both utilize similar core technologies, but have different implementations.

Virtual reality

For about three decades, the term "virtual reality" has been used as a catch-all for any sort of real-world simulation or modification. Initial implementations were used by the entertainment industry for relatively compact rides in amusement parks, with additional user by the military for flight simulator training.

Both early uses employed conventional displays in conjunction with a series of hydraulics and other mechanical haptics to provide full feedback of the environment to the user. The most famous VR entertaiment venue was the BattleTech Center which launched in 1990, and have yet to fully close.

Military simulators often lagged behind the technology of "civilian" ones. However, one military flight simulator in use in the nineties would keep a pilot trainee upside-down in the simulator while the plane being flown was in inverted flight.

Later, VR headsets were introduced, and were initially a massive edifice of plastic and silicon strapped to a wearer's head, with the user tethered to a bank of computers supplying all audio and visual input to the wearer.

Augmented reality

Since the advent of VR, the immense hardware requirements over time have become apparent. What is less difficult to manipulate and generate are overlays over the existing environment. Augmented reality can be used to guide a user on a street, or highlight businesses as a user travels through a town. Another possible implementation is capture of virtual creatures with the flick of a finger, after spotting them in the park on a daily walk.

Google's currently shelved Glass project is an example of augmented reality, with the user getting an information overlay in one eye. Several copycats of Google Glass have arisen, but none have met with any commercial success.

Google Cardboard is a simple frame for insertion of a smartphone to use as a rudimentary AR or VR system. The technology is being used in schools all over the world, and the company offers virtual field trips to remote and culturally significant areas to expand the horizons of students that are geographically or financially bound.

The Microsoft HoloLens is a full-face AR system, tethered to the Windows ecosystem. Immensely popular mobile title Pokemon Go is also considered augmented reality, with the user aiming a phone to "spot" a virtual creature in the real world.

Enter the Oculus Rift

Oculus, the parent company behind the most public VR effort yet, was formed in 2012. Two months later, crowdfunding campaign launched which would garner $2.5 million dollars. The company promised a practical, light-weight, and easy to use solution for the consumer to experience full virtual reality in the home.

Ultimately, two developer models were released, with the development process hampered somewhat by the shift in available components. Facebook purchased Oculus in 2014, and it was thought that a commercial version would soon follow.

In the beginning of 2016, the first consumer model of the Oculus Rift shipped. It has two OLED displays running at 1080x1200, one for each eye. Positional input is provided through an infrared LED, with user controls executed by an Xbox One controller.

The Oculus Rift isn't a standalone device. It relies on a fairly powerful gaming PC, with a GPU that has more heft than anything currently shipping from Apple. Support was originally promised for macOS, but it has not been updated in well over a year, and has been "paused."

"That is up to Apple and if they ever release a good computer we will do it," Oculus founder Palmer Luckey said in January. "It just boils down to the fact that Apple doesn't prioritize high-end GPUs. You can buy a $6,000 Mac Pro with the top of the line AMD FirePro D700, and it still doesn't match our recommended spec."

SteamVR

A competitor to the Oculus Rift is the HTC Vive. The Vive was revealed in 2015 and released in April 2016, and has been developed by HTC and game developer Valve. Players uses hand held controllers to map a player's motion into the virtual world, with the headset and associated tether allowing the user to navigate freely in a hopefully clear play area.

The HTC Vive's required computer specifications are similar to that of the Oculus Rift. While the processor requirement is easily met, the GPU is an issue for OS X users — at least for now.

The main difference between the HTC Vive and the Oculus Rift is the SDK. The Oculus Rift uses a closed SDK, while the HTC Vive's key motion sensing technologies were made open source in the beginning of August.

Tim Cook, AR, and VR

In 2016, Apple CEO Tim Cook has made some remarks about the company and potential VR and AR implementations. During Apple's January's quarterly report question and answer session, Cook said that VR was "cool" and noted that he didn't think it was a niche, but at the time he had little to say about its future at Apple.

During July's question and answer session, Cook had a bit more to say. "AR can be really great," Cook said. "We have been and continue to invest a lot in this. We are high on AR for the long run, we think there's great things for customers and a great commercial opportunity. So we're investing."

On Sunday, Cook said to the Washington Post that "I think AR is extremely interesting and sort of a core technology," adding that "it's something we're doing a lot of things on behind that curtain that we talked about."

VR and AR are open fields

There is no clear leader, or de facto standard behind AR or VR yet. The Oculus Rift led the field, but the HTC Vive is likely to have more third party support with the opening of hardware and software for the device, plus tight integration with the popular Steam digital distribution platform.

Apple's iOS devices can be used at the core of rudimentary AR and VR now. The Mattel ViewMaster VR is an outgrowth of Google Cardboard, and uses software to resurrect the 50-year old toy, that once used stereographic slides. With insertion of a four-inch smartphone or larger, an app now gives the same functionality as the aged ViewMaster in a more effective way for a modern audience.

Modern VR and AR technologies aren't perfect. The major source of user physical illness in the technology is illness induced by input lag.

When the displayed images don't respond immediately to user input, or a head rotation, the human body's vestibular system is confused by the input lag, causing the nausea and light-headedness associated with motion sickness.

Input lag affects people differently. If one can read in a moving car without ill-effect, then in all likelihood a VR headset won't pose a problem. However, people prone to seasickness or motion sickness nearly always suffer from bad VR implementations.

On Google Play's ViewMaster VR page, it lists hardware that is compatible with the toy. Motorola and Samsung phones are listed, as well as every iPhone newer than the iPhone 5 and 5c. It also notes that the ViewMaster VR "may also work with newer model Android smartphones not listed above" but no guarantees are made, because of the wide disparity of device capabilities out of the control of Google.

Apple's tight integration of software and hardware down to the iPhone's casing size can do a great deal to eliminate problems inherent with both AR and VR technology. Input lag can be minimized by leveraging Apple's strict control over the sensors used in a device, as well as managing the communication between the sensors and development SDKs — much like Xcode does now for iOS.

Much of the work that Apple needs to do is simply refinement of existing technologies. If Apple should utilize the open source nature of the HTC Vive for positional tracking in a future full-VR implementation, both the Apple VR and Windows-based VR ecosystem can flourish.

While Apple was the first to market with a PC, it didn't set the standard — IBM did that in 1981. Apple wasn't the first to release a MP3 player, but it did it better, and won the market in the end. Samsung released its smart watch a year before the Apple Watch came out, and in every regard, the Apple Watch is the superior product, with Samsung floundering with multiple models and operating systems.

VR and AR stand now where wearables did just a few years ago, and only have advanced lately after decades of work on computer and imaging technology making it practical and reasonably affordable to have a setup in the home. Apple can seize the day like it did with the iPod and iPad.

Based on remarks from Tim Cook, Apple appears to be looking at the possibilities.