Self-driving 'Apple Car' may combine LiDAR with other sensors for better decision-making

As work on the "Apple Car" progresses, Apple is coming up with new ways to improve automated driving systems, including using other sensors to enhance LiDAR scans to better determine what's on the road.

Apple has been working on self-driving systems for quite some time, including testing of its developments on a fleet of vehicles in the United States. While it is unclear what the company's ultimate aim is for the system, such as whether it is going to include it as a feature of a supposed "Apple Car" or will be licensed out to other manufacturers, it is known that Apple is still looking at how it can improve the technology.

In a patent granted by the US Patent and Trademark Office on Tuesday titled "Shared sensor data across sensor processing pipelines," Apple proposes multiple processes at play within a self-driving system could be more collaborative with collected data.

A typical sensor data processing pipeline would, in its simplest terms, involve data collected by sensors that are provided to a dedicated processing system, which then can determine the situation and a relevant course of action that then sends to other systems. Generally, data from a sensor would be confined to just one pipeline, with no real influence from other systems.

In Apple's proposal, pipelines could make a perception decision based on the data it has collected, with multiple pipelines capable of offering different decisions due to using different collections of sensors. While individual perceptions would be taken into account, Apple also proposes a fused perception decision could be made based on a combination of the two determined states.

A logical block diagram illustrating shared sensor data across processing pipelines.

Furthermore, Apple also suggests the data could be more granular, with decisions made and fused at different stages of each pipeline. This data, which can be shared between pipelines and can influence other processes, can include raw sensor data, processed sensor data, and even data derived from sensor data.

By offering more data points to work with, a control system will have more information to work with when creating a course of action, and may be able to make a more informed decision.

As part of the claims, Apple mentions how the pipelines could be for connected but different types of data, such as a first pipeline being for image sensor data while a second uses data from LiDAR.

Combining such data could allow for determinations that wouldn't normally be possible with one set of data alone. For example, LiDAR data can determine distances and depths, but it cannot see color, something that may be important when trying to recognize objects on the road, and data it could acquire from an image sensor data processing pipeline.

Another logical block diagram, this time showing sharing between a LiDAR sensor pipeline and one for image processing.

Apple offers that the concept could be used with other types of sensor as well, including but not limited to infrared, radar, GPS, inertial, and angular rate sensors.

The patent lists its inventors as Xinyu Xu, Ahmad Al-Dahle, and Kshitiz Garg.

Apple files numerous patent applications on a weekly basis, but though they are in no way a guarantee that Apple is developing specific features or products, it does show areas of interest for the company's research and development efforts.

Driving self-driving forward

The patent is only the latest in a long string of self-driving filings, as well as connected Apple Car applications, that have surfaced over the years.

On the sensor side, these filings have ranged from the creation of new types of LiDAR 3D mapping systems in 2016 to one from October 2019 suggesting how said systems could be hidden from view within the bodywork of the vehicle.

For processing, one May 2019 filing for a "confidence" algorithm would allow the system to get just enough data from its sensors to process the road, cutting down on the amount of data processing required to speed the process up and to save resources for other elements.

Others that surfaced in 2018 covered how a self-driving system could give passengers options, such as where to park or to change the direction of travel, then to act upon statements or gestures. A "Traffic direction gesture recognition" system found at the same time could recognize gestures of police officers and other officials directing traffic.

A "Cognitive Load Routing Metric for Vehicle Guidance" aimed to find the best route to take for a journey, by taking into account the complexity of the route. This included the number of lanes, narrowness of roads, street lights, and pedestrian traffic data among other data points, which could influence both directions provided to drivers and the self-driving system.

There have even been patent filings suggesting a self-driving system could adjust its driving style depending on the stress levels of passengers.

Latest Exclusives

Latest comparisons

29 Comments

Xed 4 Years · 2896 comments

About 4 years ago

I know a lot of people think this project is dead and of course I'd love for Apple to demo a car at WWDC, but I also want them to get it right.PS: Thank you for using a different car pick. 🙏 Have you considered contracting someone to create some nice, futuristic designs? From my experience, a great mockup can help draw a lot of attention to a website.

sconosciuto 4 Years · 295 comments

I still have a lot of trouble visualizing Apple selling cars. It's similar to how so many people mistakenly thought Apple would sell HDTVs, but with even more issues logistical and otherwise. It makes me think of the television business. Apple (Jobs) more or less rolled the music business, but it is still figuring out how to break into the television business. I see the auto business much the same: established players who are big, savvy AF, and extremely wary of outsiders trying to get their foot in the door.

So if they are not going to sell cars, what is the point of all of this R&D? To sell the underlying system to automakers? But they are already working on that themselves. I don't see why they would ditch all that research effort and go with Apple; they will want to own the technology.

Or would Apple go the way of Smart and have an established automaker manufacture the cars with Apple technology? However that still doesn't answer the question of why Apple would want to do this with such low profit margins.

lkrupp 19 Years · 10521 comments

In my opinion, until the human driver is completely eliminated from the equation the autonomous vehicle is a pipe dream. I watched a PBS documentary about the status of the technology. The really interesting part concerned interviews with the test vehicle drivers who reported that the super cautious nature of the software caused road rage in human drivers. They also reported that some drivers (aka assholes) intentionally tried to run them off the road or get the vehicle to crash. Humans are emotional, irrational, evil by nature. And as I have said before no self-driving car would have a chance against an 80 year old blue hair diving her 1975 Buick Electra gunboat.

tht 23 Years · 5654 comments

sconosciuto said:

I still have a lot of trouble visualizing Apple selling cars. It's similar to how so many people mistakenly thought Apple would sell HDTVs, but with even more issues logistical and otherwise. It makes me think of the television business. Apple (Jobs) more or less rolled the music business, but it is still figuring out how to break into the television business. I see the auto business much the same: established players who are big, savvy AF, and extremely wary of outsiders trying to get their foot in the door.

So if they are not going to sell cars, what is the point of all of this R&D? To sell the underlying system to automakers? But they are already working on that themselves. I don't see why they would ditch all that research effort and go with Apple; they will want to own the technology.

Or would Apple go the way of Smart and have an established automaker manufacture the cars with Apple technology? However that still doesn't answer the question of why Apple would want to do this with such low profit margins.

Apple would do it because they think they will have good profit margins. How that can be achieved is an interesting question. The existing automakers say it is a tight and capital intensive business, our existing experience says it is so, but that just means Apple would not do it that way or won't be offering a product that sets them up with tight or no margins.

I think it is very likely that Apple won't be setting themselves up like existing automakers. If there is a model they are going to follow, look at Tesla, but extrapolate even more:
1. They won't own any manufacturing or assembly plants. It will all be contracted out. The assembly line design will definitely be theirs.
2. They won't have a dealer network and won't have "inventory". Everything will be BTO.
3. They will have a branded service network of some kind, possibly like Tesla's at-home service.
4. There will be features that sell their vehicle and that provide those margins.
5. They will need to have a branded charging network of some kind.

To me, all this automated driving R&D means they are on a 10 to 15 year timeline. I think they decided they couldn't have a competitive product without it. They can build, for a lack of a better word, a "dumb" EV and have it shipping in a 3 to 4 year time frame, but I think they decided it wasn't worth it sometime in 2017 or whenever the Bob Mansfield shuffle happened. Too bad imo. I'd prefer a dumb EV, ie, one without automated driving features, as long as it has class leading efficiency. Wish they would buy out Lightyear One and do their relentless iteration with that type of idea.