As work on the "Apple Car" progresses, Apple is coming up with new ways to improve automated driving systems, including using other sensors to enhance LiDAR scans to better determine what's on the road.
Waymo's LiDAR-equipped minivan concept
Apple has been working on self-driving systems for quite some time, including testing of its developments on a fleet of vehicles in the United States. While it is unclear what the company's ultimate aim is for the system, such as whether it is going to include it as a feature of a supposed "Apple Car" or will be licensed out to other manufacturers, it is known that Apple is still looking at how it can improve the technology.
In a patent granted by the US Patent and Trademark Office on Tuesday titled "Shared sensor data across sensor processing pipelines," Apple proposes multiple processes at play within a self-driving system could be more collaborative with collected data.
A typical sensor data processing pipeline would, in its simplest terms, involve data collected by sensors that are provided to a dedicated processing system, which then can determine the situation and a relevant course of action that then sends to other systems. Generally, data from a sensor would be confined to just one pipeline, with no real influence from other systems.
In Apple's proposal, pipelines could make a perception decision based on the data it has collected, with multiple pipelines capable of offering different decisions due to using different collections of sensors. While individual perceptions would be taken into account, Apple also proposes a fused perception decision could be made based on a combination of the two determined states.
A logical block diagram illustrating shared sensor data across processing pipelines.
Furthermore, Apple also suggests the data could be more granular, with decisions made and fused at different stages of each pipeline. This data, which can be shared between pipelines and can influence other processes, can include raw sensor data, processed sensor data, and even data derived from sensor data.
By offering more data points to work with, a control system will have more information to work with when creating a course of action, and may be able to make a more informed decision.
As part of the claims, Apple mentions how the pipelines could be for connected but different types of data, such as a first pipeline being for image sensor data while a second uses data from LiDAR.
Combining such data could allow for determinations that wouldn't normally be possible with one set of data alone. For example, LiDAR data can determine distances and depths, but it cannot see color, something that may be important when trying to recognize objects on the road, and data it could acquire from an image sensor data processing pipeline.
Another logical block diagram, this time showing sharing between a LiDAR sensor pipeline and one for image processing.
Apple offers that the concept could be used with other types of sensor as well, including but not limited to infrared, radar, GPS, inertial, and angular rate sensors.
The patent lists its inventors as Xinyu Xu, Ahmad Al-Dahle, and Kshitiz Garg.
Apple files numerous patent applications on a weekly basis, but though they are in no way a guarantee that Apple is developing specific features or products, it does show areas of interest for the company's research and development efforts.
Driving self-driving forward
The patent is only the latest in a long string of self-driving filings, as well as connected Apple Car applications, that have surfaced over the years.
On the sensor side, these filings have ranged from the creation of new types of LiDAR 3D mapping systems in 2016 to one from October 2019 suggesting how said systems could be hidden from view within the bodywork of the vehicle.
For processing, one May 2019 filing for a "confidence" algorithm would allow the system to get just enough data from its sensors to process the road, cutting down on the amount of data processing required to speed the process up and to save resources for other elements.
Others that surfaced in 2018 covered how a self-driving system could give passengers options, such as where to park or to change the direction of travel, then to act upon statements or gestures. A "Traffic direction gesture recognition" system found at the same time could recognize gestures of police officers and other officials directing traffic.
A "Cognitive Load Routing Metric for Vehicle Guidance" aimed to find the best route to take for a journey, by taking into account the complexity of the route. This included the number of lanes, narrowness of roads, street lights, and pedestrian traffic data among other data points, which could influence both directions provided to drivers and the self-driving system.
There have even been patent filings suggesting a self-driving system could adjust its driving style depending on the stress levels of passengers.