Apple invention details AR mapping system for iPhone, head-mounted displays

An augmented reality invention, Apple's patent application for a "Method for representing points of interest in a view of a real environment on a mobile device and mobile device therefor" deals with interactions between computer generated objects and representations of real environments. In particular, the filing describes techniques by which images of the real world are enhanced by point of interest (POI) information.

Landmarks, buildings and other popular objects are commonly referred to as POIs in mapping applications, including Apple's own Maps. The company already incorporates POI data into first-party features, such as Maps searches for nearby restaurants or gas stations. Today's IP takes the idea beyond two-dimensional maps by adding AR into the mix.

In practice, a device first captures images or video of a user's environment with an onboard camera and displays those images onscreen. Using geolocation information gleaned from GPS, compass and other sensor data, the device determines its relative position and heading relative to its surrounding environment.

In some embodiments, information of nearby POIs, or those that are determined relevant or visible in the recorded real-world view, is downloaded from offsite servers. Using latitude, longitude and altitude data, the system can compute the relative location of target POIs and display them correctly onscreen.

After determining POI location, graphical indicators with interactive annotations are overlaid on top of the captured image. Sensors, including a depth-sensing camera, might be used to place the markers in their correct spot. Importantly, indicators are anchored to their respective real world counterparts.

For example, when a user moves their device to the left, a building shown onscreen will appear to shift to the right of the display. When such movements occur, POI markers shift accordingly, resulting in perceived object persistence.

The invention further details more intuitive methods of interacting with the POI system than has been previously proposed. For example, iPhones are rarely held in a vertical position when navigating or performing operations other than picture taking. A more natural pose, with the top of the device angled slightly toward the ground, could push POI indicia offscreen.

To compensate, the system intelligently draws lines or provides a "balloon" GUI that clearly connects POI annotations to real world objects. In such cases, users can easily tap on an annotation to bring up additional information while still holding the device in a natural manner.

Similar access methods are disclosed for POIs that appear in the middle or upper portion of a display. If the image position of the POI is far away from a user's thumb, for example, the system might generate POI annotations in screen real estate that is more easily accessible.

Other embodiments describe a user interface that switches from a real world view to a vanilla POI list view — with compass-based directional indicators — when a user device is tilted beyond a predetermined threshold. This particular presentation method would be ideal for at-a-glance navigation.

Apple's invention goes on to describe deployment on alternative devices like a head-mounted display with semi-transparent screen.

With an HMD, users can interact with POI overlays by reaching their hand out in front of their face to overlap onscreen annotations. Positioning sensors or an onboard camera might determine finger location relative to onscreen GUI representations, triggering a preconfigured app action like loading additional POI information. In such scenarios, a balloon GUI presented at a lower onscreen position would likely be a more comfortable mode of operation for most users.

Apple's patent application is an extension of granted IP reassigned by German AR firm Metaio, which the tech giant purchased in 2015 to further efforts in the burgeoning AR/VR market. Apple has expressed an increasingly intense interest in AR over the past year, but it wasn't until the announcement of ARKit in iOS 11 that the company's ambitions were revealed.

Whether Apple plans to integrate AR into Maps, or another first-party app, in the near future remains to be seen. So far, Apple seems content on delivering ARKit as a coding platform on developers can build their own wares. Although ARKit was just recently released as part of the iOS 11 beta, coders have already created a number of impressive demo projects showing off the solution's potential.

Apple's AR point of interest patent application was first filed for in April 2017 and credits Anton Fedosov, Stefan Misslinger and Peter Meier as its inventors.