Apple's heads-up display tech for self-driving cars uses AR to enhance driver safety

Joining together two fields of interest to Apple, the patent application for an "Adaptive vehicle augmented reality display using stereographic imagery" details an AR system that can use a pre-generated 3D model of the world in improving how it generates 3D content for its users. While this could be applied in many different ways, Apple suggests this interpretation could be used to provide users with even more information about the world around them, while on the move.

The pre-generated 3D model could include data about road routes, points of interest, and other areas useful to drivers, which could be held remotely in cloud storage. This data is collected together into smaller chunks, like 3D versions of tiles used by online mapping services, and relevant sections of the 3D model are provided to the car's AR system.

Apple suggests the onboard AR system could combine the pre-made 3D tile with data from its own sources, such as LIDAR sensors of a self-driving car, in making decisions of what information to provide to the driver and passengers. This data is then presented to the driver, with one example presented as data projected onto the vehicle's windscreen.

Crucially, this extra downloaded data would provide information about areas far beyond the range of the car's sensor arrays and the driver's own vision. This can allow for data that isn't immediately visible to be included in the displayed image, such as showing the path ahead in an AR view, including parts of the road obscured by buildings or hills in the distance.

If another nearby vehicle turns away and drives out of immediate view, like around a building, the downloaded data tile could tell the AR system of that road, and overlay a virtual car graphic on the display that shows where the vehicle could have travelled.

This view could also help in cases where visibility is poor, like fog or heavy snow, where the car's sensors may also struggle to create an accurate picture of the immediate area outside. The pre-existing cloud data could help give what would be an expected route and important notifications that the sensors are unable to create for themselves, in an emergency.

Being able to augment a self-driving car or driver assist system with extra data for the road ahead potentially makes these systems more useful and safer. Giving a self-driving car more data potentially means it will be more capable of planning maneuvers it may need to perform in the coming miles, with the advanced warning providing more time for processing these types of decisions.

Apple has a major interest in self-driving cars, with the company operating 45 vehicles in tests in California. Covered with an array of LIDAR and other sensors to monitor its surroundings while on the move, the cars are acknowledged to be part of self-driving system trials, though the ultimate aim of the technology is unknown.

Current speculation points to a possible self-driving platform for ridehailing services to use, most likely in partnership with third-party firms supplying the vehicles. There are also the long-standing rumors surrounding "Project Titan," thought to involve the creation of its own electric vehicles.

Regardless of Apple's plans, earlier patent applications certainly indicate it is still coming up with ideas relating to self-driving cars. In March, patent applications were found concerning gesture recognition in vehicles, a communication system between self-driving cars and other manually-driven vehicles, and understanding the gestures of police officers controlling traffic on roads.