Apple is working on virtual reality/augmented reality that could be incorporated into future iOS devices and/or hardware products. There aren’t any concrete details about the products or when they might launch, but the company’s focus in the area has increased over the past several months.
Tim Cook said in a 2016 interview that Apple is “doing a lot of things” in the AR space and has called it a “core technology." He even hints at favoring it over VR. "There's virtual reality and there's augmented reality — both of these are incredibly interesting," Cook said in the interview. "But my own view is that augmented reality is the larger of the two, probably by far." In August, Cook said that "I think AR is extremely interesting and sort of a core technology," adding that "it's something we're doing a lot of things on behind that curtain that we talked about." Then in October, he again touted the benefits of AR over VR: "There's no substitute for human contact," Cook said. "And so you want the technology to encourage that."
In a February 2017 interview, Cook elaborated on his thoughts on AR, calling it a "big idea" concept, like a smartphone. Cook also suggested AR holds more promise than VR, as the latter "closes the world out" while AR keeps it all visible.
"The smartphone is for everyone, we don't have to think the iPhone is about a certain demographic, or country, or vertical market; it's for everyone. I think AR is that big, it's huge," said Cook. "I get excited because of the things that could be done that could improve a lot of lives. And be entertaining."
"I view AR like I view the silicon here in my iPhone, it's not a product per se, it's a core technology. But there are things to discover before that technology is good enough for the mainstream."
New miniaturization techniques and advanced sensor technology built for smartphones are the basis of many hardware kits, including those built by Oculus and Microsoft, but the world leader in portable technology, Apple, has yet to debut its own system.
Apple's "Visual-based inertial navigation" patent
What is virtual reality/augmented reality?
Simply put, virtual reality is a way for users to see a computer-generated world, freely able to look around by moving their head, usually with stereoscopic vision and a way to interact with the digital environment. Typically this technology uses a head-mounted display with a motion tracking mechanism to monitor the head's movement, and a similarly-tracked controller.
Virtual reality, as a concept, has been around for years, but has yet to see much commercial success. Companies tried, and failed, to make VR a socially-acceptable experience in the 90's, but these efforts were considered to be bulky, heavy, and difficult to use. For example, the Nintendo Virtual Boy, the Japanese game company's attempt at creating such a device for public consumption, was discontinued just one year after its release.
In the last few years, VR has undergone a renaissance, thanks in part to the miniaturization of technology, and the processing power of modern computing systems. Hardware such as the Oculus Rift and the HTC Vive make VR relatively accessible, though the lightweight hardware is still relatively prohibitive for most people unwilling to spend $1,000 on an early-generational technology.
The improvements in mobile technology has also led to a cheaper way to try out VR, using a smartphone. Google created Cardboard, a VR head-mounted display that uses cardboard and lenses with a smartphone's display to create a VR experience, while Samsung has pushed forward with its own Gear VR system that follows a similar path.
Augmented Reality is a markedly different concept to VR, in that instead of a completely virtual environment, the user instead sees the real world, albeit with computer-generated imagery partly obscuring their vision. This can be used to overlay data on a person's viewpoint, providing extra information about things they are looking at.
For example, this could include labels and supporting materials, cut-away side views of an object, or even translated signs overlaying those of a different language. The key here is that AR converges the real and virtual worlds into one singular view.
There are two main ways AR has developed, again in varying levels of cost and practicality. Some companies, such as Microsoft, have come up with headsets that users can see through, rather than looking at a tiny screen. While still in development, these are relatively inaccessible for most people due to the need for expensive hardware, and typically tethering the user to a specific room while it is being used.
Much like VR, there is a smartphone-based alternative. Using a combination of the rear camera of the smartphone and the screen, users can see “through” their mobile device to look at generated images. This concept has been available for a number of years, and is already employed in a number of different ways.
In Pokemon Go, players can see creatures they want to capture on their display, “existing” on a spot on the real-world landscape, with other games and apps performing similar tricks. Some apps can provide visitors to a museum more information about exhibits, or travelers can see road signs translated into their native language.
For this AR technique, all a user requires is a relevant app that offers AR functionality and a smartphone, something that the majority of likely users will already own, giving it an extremely low barrier for entry compared to the hardware-encumbered version of AR, and VR as a whole.
Analysts say AR perfect partnership for Apple
KGI analyst Ming-Chi Kuo told investors in October 2016 that he believes Apple’s aptitude for delivering innovative user experiences through human-machine interfaces will help the company move naturally into the AR space.
Just as iPod helped pave the way for the iPhone, the iPhone may be able to provide the necessary building blocks for a full-blown AR solution. He didn’t provide details on what this might look like. One example might be Apple testing the waters with a system like the iOS game Pokemon Go, which uses the iPhone’s camera and display to provide users with a seamless AR experience.
In general, Kuo sees Apple integrating AR to redefine key product lines, perhaps leapfrogging competitors by three to five years. For example, augmented user interfaces could drastically change the way users interact with Apple Watch and Apple TV, eliminating obstacles like small screens and clunky controls.
At the same time, Apple might leverage AR tech to break into other fields, Kuo says. One such area of interest is automotive technology, or more specifically autonomous driving systems. Apple was widely rumored to be working on a self-driving car, dubbed "Project Titan," since March of 2015, but recent reports claim the company has abandoned those plans. Instead of a full-fledged car, Apple is scaling back its ambitious project to focus on underlying technology.
The notion that AR is one of the next big technologies to be embraced in Apple's products is also held by Steven Milunovich, an analyst for UBS. A note to investors in late February 2017 hypes the potential for AR, citing an interview with a developer suggesting it could make the current smartphone experience seem like "the dark ages."
"Thanks to advanced cameras, consumers will hold their phones up with images superimposed onto the screen in cars, rooms, or walking down the street," writes Milunovich. "3D mapping through Simultaneous Localization and Mapping (SLAM) will be key."
Suggesting sources claim Apple has over 1,000 engineers in Israel working on AR-related technology, the analyst expects Apple to offer advanced AR applications in new ways, with supporting technology arriving in future iPhone iterations.
Milunovich believes Apple will slowly roll out the technology over the coming years, though could bypass the competition with a "superior user experience," possibly drawing more Android users over to iOS in the process. In the near future, the rumored upcoming "iPhone 8" is said to include "moderate 3D mapping using stereoscopic vision," while developers could be provided an AR software development kit by Apple as soon as this year.
AR glasses to debut in 2018
Rumors of Apple's intent to enter the augmented reality hardware space gained traction in January, as a report from AR/VR evangelist Robert Scoble claimed the company is partnering with optics manufacturer Carl Zeiss on a pair of lightweight augmented reality/mixed reality glasses.
Further, Scoble says the partnership explains why the Zeiss booth at CES 2017, located in the middle of the AR section, had no AR, VR or mixed reality optics to demonstrate. The theory is that Apple muzzled the company until the supposed tie-up is announced —or falls through.
Refresh rates and the "iPhone 8"
In March, developer Steven Troughton-Smith discovered code in a pre-release beta for iOS 10.3 that would allow an app to specify the device's screen refresh rate, potentially signifying a display with a higher refresh rate may be on the way in a future launch. While this can help improve the experience of using the Apple Pencil in a refreshed iPad Pro, it also has applications in VR.
Virtual reality relies on having high refresh rate displays to make motion as fluid as possible. Lower refresh rates means fewer updates to the image the user sees, making movements seem choppier and destroying the "illusion" for the user.
Current iPhones use an LCD panel with a refresh rate of 60Hz, which is acceptable for the majority of smartphone uses. The rumored "iPhone 8," expected to be introduced this fall, is currently thought to have an OLED display, a technology that has a far lower response time than LCD, and has the potential to be run at far higher refresh rates.
This switch to OLED theoretically makes the "iPhone 8" an extremely good candidate for use with VR or AR, when used in a Cardboard or Gear VR-style system.
One other item that the "iPhone 8" has that may help with VR and AR is the rumored laser-based 3D scanning system supposedly being used for facial recognition. The technology, effectively a miniaturized LIDAR mapper or rangefinder, could also be used to scan the environment if it is also mounted to the back of the device.
For VR, scanning the local area potentially allows the user to "see" potential obstructions while their view is obscured by a head-mounted display accessory. More likely would be its use in AR, which could help give apps direct positioning data for items, allowing for labels and other interactive elements to be correctly placed "on" an object of interest.
Apple virtual reality/augmented reality hiring
Two Apple hires in late 2016 suggested the company was getting serious about building out its own virtual and augmented reality technologies, though it had some catching up to do as Google, Facebook and Microsoft forge ahead with mature projects.
Zeyu Li, a former Magic Leap employee, joined Apple as a Senior Computer Vision Algorithm Engineer, reported Business Insider. According to Li's LinkedIn profile, he worked first as Lead 3D Engineer, then as Principal Engineer.
Apple's second hire came from Facebook's Oculus. Yury Petrov, who worked as a research scientist at the VR firm since 2013, took an identical position at Apple in June. Petrov's LinkedIn profile said his job at Oculus entailed “psychophysical and physiological studies of visual and multisensory experience of virtual reality (VR) including user experience factors in head-mounted displays (HMD).”
A report from March 2017 from sources of Bloomberg suggest Apple has filled out the rest of the team with high-profile individuals from a number of other major companies.
The person said to be heading up the team is Mike Rockwell, a 2015 hire who previously led the hardware and new technologies groups at Dolby. Rockwell is believed to be reporting to Dan Riccio, senior VP of Hardware Engineering.
Fletcher Rothkopf, one of the designers of the original Apple Watch, was allegedly assigned to work for Rockwell in spring 2016, alongside THX audio standard creator Tomlinson Holman.
Former lead engineer of Amazon's Lumberyard game engine Cody White is also said to be on the team, as well as Duncan McRoberts. McRoberts was previously a director of software development at Meta, a firm that produces high-end AR glasses.
Other members of the team are said to include iPhone, camera, and optical lens engineers, along with 3D animation veterans who previously worked on special effects for movies. Apple has reportedly managed to hire some employees away from Weta Digital, known for work on the "Lord of the Rings" films, with the new hires thought to be working from a new office in Wellington, New Zealand.
Sending employees to Stanford's virtual reality lab
It's been revealed on a couple of occasions that Apple has taken interest in Stanford University's Virtual Human Interaction Lab, sending representatives to visit the facility at least three times in three months earlier in 2016. Employees were put through immersive VR experiences, including a project that aims to teach empathy through forced perspective virtual reality interventions. For example, a male subject entering the VR world might be given a female persona and exposed to prejudice.
Apple has made a number of key acquisitions in the virtual reality/augmented reality field that would further hint of developments in this realm.
Most recently, Apple purchased both Emotient - a company that builds tools for facial expression analysis -- and Flyby Media in 2016. Emotient's tools are used to capture direct emotional response from customers, and has been used in marketing/advertising. It has also been tested in the medical setting to measure pain levels. Flyby focuses on augmented reality projects.
Apple also acquired motion capture specialist Faceshift and German AR firm Metaio in 2015, as well as PrimseSense in 2013.
Apple has been mum about the Faceshift purchase, but has finally admitted to being behind the mysterious acquisition of the company but has declined to offer detail. The buyout helped Apple continue to build out its portfolio of facial recognition technologies. It’s been similarly quiet about the Metaio acquisition.
The PrimeSense acquisition sparked rumors that motion-based capabilities may be in store for Apple TV, as well an iPad app for 3D printing. PrimeSense's 3D depth technology and motion sensing capabilities were used in Microsoft Kinect's platform.
In addition to ongoing in-house research and development, Apple holds a variety of patents covering a gamut of augmented reality applications, including transparent displays, mapping solutions and iPhone-powered virtual displays.
In November 2016, Apple obtained a patent detailing an augmented reality mapping system that harnesses iPhone hardware to overlay visual enhancements onto live video, lending credence to recent rumors suggesting the company plans to implement an iOS-based AR strategy in the near future.
Apple's U.S. Patent No. 9,488,488 for "Augmented reality maps" describes a mapping app capable of tapping into iPhone's advanced sensor suite to present users with real-time augmented views of their surrounding environment.
Apple was also granted a patent detailing a method of device localization -- mapping -- using computer vision and inertial measurement sensors, one of the first inventions to be reassigned from the acquisition of AR reality startup Flyby Media.
Two patents that surfaced in January 2017 stem from Apple's acquisition of German AR firm Metaio. One relates to the hardware framework for an AR device with enhanced computer vision capabilities, with power-efficient object recognition being a main focus for the patent.
The second, a method for "representing virtual information in a real environment," details a way to label points of interest in an AR environment, taking occlusion perception into account. Using a combination of depth sensing and positioning data, Apple's system would be able to show only labels for points of interest that can be seen by the user, hiding those for places out of view behind a wall or a building, for example.
EPGL & Apple parntership for AR iOS apps
Medical supply company EPGL in conjunction with Apple is utilizing its intellectual property to develop iOS apps to project an image on a contact lens oriented around the perimeters of a contact lens for use in AR applications.
The app requires low power, can be adjusted quickly, and can be incorporated into the elastic material of a contact lens. The lenses may utilize a prism to redirect the image onto the retina, potentially aiding those with vision cuts, where part of the users’s vision is absent or restricted due to stroke or another malady.
This AR tech avoid the stigma of bulkier apparatuses like Google Glass, which was banned in some places because of the possibility of covert surveillance by a wearer.
Why an Apple virtual headset is unlikely
Although patents may hint otherwise, an Apple virtual headset is unlikely. They're too niche. Instead, Apple will more likely create a platform where developers can tap into its hardware/software to create VR experiences. This could mean simple apps or connected headsets, etc.
Piper Jaffray analyst Gene Munster believes an iOS ecosystem support might be ripe for launch as soon as 2018 due to the aforementioned acquisitions, hires and serious assets earmarked specifically for AR/VR research and development. There's a natural progression from current cutting-edge personal technology — smartphones — to AR/VR devices, which could see mass adoption as wearable devices priced in line with modern handsets. Munster believes Apple is currently looking at VR like it does the Apple Watch, which is to say a peripheral for iPhone. However, he doesn't see the company releasing its own hardware at least not in the near term.
Modern virtual reality and augmented reality technologies aren't perfect. The major source of user physical illness in the technology is illness induced by input lag. Apple's tight integration of software and hardware down to the iPhone's casing size can do a great deal to eliminate problems inherent with both AR and VR technology. Input lag can be minimized by leveraging Apple's strict control over the sensors used in a device, as well as managing the communication between the sensors and SDKs — much like Xcode does now for iOS.
Much of the work that Apple needs to do is simply refinement of existing technologies. If Apple should utilize the open source nature of the HTC Vive for positional tracking in a future full-VR implementation, both the Apple VR and Windows-based VR ecosystem can flourish.
While Apple was the first to market with a PC, it didn't set the standard — IBM did that in 1981. Apple wasn't the first to release a MP3 player, but it did it better, and won the market in the end. Samsung released its smart watch a year before the Apple Watch came out, and in every regard, the Apple Watch is the superior product, with Samsung floundering with multiple models and operating systems.