Apple is working on virtual reality/augmented reality that could be incorporated into future iOS devices and/or hardware products. There aren’t any concrete details about the products or when they might launch, but the company’s focus in the area has increased over the past several months.
Tim Cook recently said in an interview that Apple is “doing a lot of things” in the AR space and has called it a “core technology." He even hints at favoring it over VR. "There's virtual reality and there's augmented reality — both of these are incredibly interesting," Cook said in the interview. "But my own view is that augmented reality is the larger of the two, probably by far." In August, Cook said that "I think AR is extremely interesting and sort of a core technology," adding that "it's something we're doing a lot of things on behind that curtain that we talked about." Then in October, he again touted the benefits of AR over VR: "There's no substitute for human contact," Cook said. "And so you want the technology to encourage that."
New miniaturization techniques and advanced sensor technology built for smartphones are the basis of many hardware kits, including those built by Oculus and Microsoft, but the world leader in portable technology, Apple, has yet to debut its own system.
Apple's "Visual-based inertial navigation" patent
What is virtual reality/augmented reality
Virtual reality has been around for years, but hasn't seen much commercial success. Virtual reality focuses on creating a virtual world with which users can interact. Companies as far back as the 90s have tried (and failed) to make virtual reality devices happen. Just think back to the Nintendo Virtual Boy that was discontinued just a year after its release.
Augmented reality on the other hand, has found a higher success rate with consumers. Augmented reality is a blend of virtual and reality where developers can produce images that blend in with the real world around us. Think apps like Pokemon Go along with video game and hardware devices like Google Glass.
Both virtual reality and augmented realty interact with users and immerses them in a different reality. They just have different ways of doing so.
Analyst says AR perfect partnership for Apple
KGI analyst Ming-Chi Kuo told investors in October 2016 that he believes Apple’s aptitude for delivering innovative user experiences through human-machine interfaces will help the company move naturally into the AR space.
Just as iPod helped pave the way for the iPhone, the iPhone may be able to provide the necessary building blocks for a full-blown AR solution. He didn’t provide details on what this might look like. One example might be Apple testing the waters with a system like the iOS game Pokemon Go, which uses the iPhone’s camera and display to provide users with a seamless AR experience.
In general, Kuo sees Apple integrating AR to redefine key product lines, perhaps leapfrogging competitors by three to five years. For example, augmented user interfaces could drastically change the way users interact with Apple Watch and Apple TV, eliminating obstacles like small screens and clunky controls.
At the same time, Apple might leverage AR tech to break into other fields, Kuo says. One such area of interest is automotive technology, or more specifically autonomous driving systems. Apple was widely rumored to be working on a self-driving car, dubbed "Project Titan," since March of 2015, but recent reports claim the company has abandoned those plans. Instead of a full-fledged car, Apple is scaling back its ambitious project to focus on underlying technology.
AR glasses to debut in 2018
Rumors of Apple's intent to enter the augmented reality hardware space gained traction in January, as a report from AR/VR evangelist Robert Scoble claimed the company is partnering with optics manufacturer Carl Zeiss on a pair of lightweight augmented reality/mixed reality glasses.
Further, Scoble says the partnership explains why the Zeiss booth at CES 2017, located in the middle of the AR section, had no AR, VR or mixed reality optics to demonstrate. The theory is that Apple muzzled the company until the supposed tie-up is announced —or falls through.
Apple virtual reality/augmented reality hiring
Two Apple hires in late 2016 suggest the company is getting serious about building out its own virtual and augmented reality technologies, though they has some catching up to do as Google, Facebook and Microsoft forge ahead with mature projects.
Zeyu Li, a former Magic Leap employee, joined Apple as a Senior Computer Vision Algorithm Engineer, reported Business Insider. According to Li's LinkedIn profile, he worked first as Lead 3D Engineer, then as Principal Engineer.
Apple's second hire comes from Facebook's Oculus. Yury Petrov, who worked as a research scientist at the VR firm since 2013, took an identical position at Apple in June. Petrov's LinkedIn profile says his job at Oculus entailed “psychophysical and physiological studies of visual and multisensory experience of virtual reality (VR) including user experience factors in head-mounted displays (HMD).”
Sending employees to Stanford's virtual reality lab
It's been revealed on a couple of occasions that Apple has taken interest in Stanford University's Virtual Human Interaction Lab, sending representatives to visit the facility at least three times in three months earlier in 2016. Employees were put through immersive VR experiences, including a project that aims to teach empathy through forced perspective virtual reality interventions. For example, a male subject entering the VR world might be given a female persona and exposed to prejudice.
Apple has made a number of key acquisitions in the virtual reality/augmented reality field that would further hint of developments in this realm.
Most recently, Apple purchased both Emotient - a company that builds tools for facial expression analysis -- and Flyby Media in 2016. Emotient's tools are used to capture direct emotional response from customers, and has been used in marketing/advertising. It has also been tested in the medical setting to measure pain levels. Flyby focuses on augmented reality projects.
Apple also acquired motion capture specialist Faceshift and German AR firm Metaio in 2015, as well as PrimseSense in 2013.
Apple has been mum about the Faceshift purchase, but has finally admitted to being behind the mysterious acquisition of the company but has declined to offer detail. The buyout helped Apple continue to build out its portfolio of facial recognition technologies. It’s been similarly quiet about the Metaio acquisition.
The PrimeSense acquisition sparked rumors that motion-based capabilities may be in store for Apple TV, as well an iPad app for 3D printing. PrimeSense's 3D depth technology and motion sensing capabilities were used in Microsoft Kinect's platform.
In addition to ongoing in-house research and development, Apple holds a variety of patents covering a gamut of augmented reality applications, including transparent displays, mapping solutions and iPhone-powered virtual displays.
In November 2016, Apple obtained a patent detailing an augmented reality mapping system that harnesses iPhone hardware to overlay visual enhancements onto live video, lending credence to recent rumors suggesting the company plans to implement an iOS-based AR strategy in the near future.
Apple's U.S. Patent No. 9,488,488 for "Augmented reality maps" describes a mapping app capable of tapping into iPhone's advanced sensor suite to present users with real-time augmented views of their surrounding environment.
Apple wasalso granted a patent detailing a method of device localization -- mapping -- using computer vision and inertial measurement sensors, one of the first inventions to be reassigned from this year's acquisition of AR reality startup Flyby Media.
EPGL & Apple parntership for AR iOS apps
Medical supply company EPGL in conjunction with Apple is utilizing its intellectual property to develop iOS apps to project an image on a contact lens oriented around the perimeters of a contact lens for use in AR applications.
The app requires low power, can be adjusted quickly, and can be incorporated into the elastic material of a contact lens. The lenses may utilize a prism to redirect the image onto the retina, potentially aiding those with vision cuts, where part of the users’s vision is absent or restricted due to stroke or another malady.
This AR tech avoid the stigma of bulkier apparatuses like Google Glass, which was banned in some places because of the possibility of covert surveillance by a wearer.
Why an Apple virtual headset is unlikely
Although patents may hint otherwise, an Apple virtual headset is unlikely. They're too niche. Instead, Apple will more likely create a platform where developers can tap into its hardware/software to create VR experiences. This could mean simple apps or connected headsets, etc.
Piper Jaffray analyst Gene Munster believes an iOS ecosystem support might be ripe for launch as soon as 2018 due to the aforementioned acquisitions, hires and serious assets earmarked specifically for AR/VR research and development. There's a natural progression from current cutting-edge personal technology — smartphones — to AR/VR devices, which could see mass adoption as wearable devices priced in line with modern handsets. Munster believes Apple is currently looking at VR like it does the Apple Watch, which is to say a peripheral for iPhone. However, he doesn't see the company releasing its own hardware at least not in the near term.
Modern virtual reality and augmented reality technologies aren't perfect. The major source of user physical illness in the technology is illness induced by input lag. Apple's tight integration of software and hardware down to the iPhone's casing size can do a great deal to eliminate problems inherent with both AR and VR technology. Input lag can be minimized by leveraging Apple's strict control over the sensors used in a device, as well as managing the communication between the sensors and development SDKs — much like Xcode does now for iOS.
Much of the work that Apple needs to do is simply refinement of existing technologies. If Apple should utilize the open source nature of the HTC Vive for positional tracking in a future full-VR implementation, both the Apple VR and Windows-based VR ecosystem can flourish.
While Apple was the first to market with a PC, it didn't set the standard — IBM did that in 1981. Apple wasn't the first to release a MP3 player, but it did it better, and won the market in the end. Samsung released its smart watch a year before the Apple Watch came out, and in every regard, the Apple Watch is the superior product, with Samsung floundering with multiple models and operating systems.