The California Department of Motor Vehicles has released a letter it received from Apple concerning the "Project Titan" self-driving vehicle project's disengagement rates, with the letter advising how Apple is collecting data on incidents where a driver takes over for the automated driving system, and why the number is so high.
Apple's self-driving car testbed spotted in California.
Data compiled on Tuesday revealed Apple had what could be the worst-performing self-driving system undergoing testing in California, with its fleet of heavily modified Lexus RX450h SUVs recording an average of 871.65 disengagements for every 1,000 miles travelled, lagging far behind competing systems tested by Google's Waymo, GM Cruise, and Zoox.
The letter from Apple, published to the DMV website on Wednesday, was originally sent by Apple's Apple's senior director of Autonomous Systems Engineering Jamie Waydo on December 21, 2018. The letter explains it is the first disengagement report from Apple since receiving its autonomous vehicle testing permit, and covers a period between April 14, 2017 until November 30, 2018.
"Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations," writes Waydo. "To support this approach, our public road testing policies require drivers to proactively take manual control of the vehicle any time the system encounters a scenario beyond our currently proven abilities."
"In addition, our software self-monitors and returns control back to the driver when it encounters any errors or issues during operation" the letter continues, advising this allows Apple to "prioritize the safe operation of our vehicles above all else at all times."
For recording the disengagements, Apple puts such events into two classes: Manual Takeovers and Software Disengagements.
Manual Takeovers are times when the drivers take control from the self-driving system, adhering to "conservative operational constraints" adhered to in testing. Apple's examples include instances when there is a sign of an approaching emergency vehicle, construction zones, and unexpected objects in and around the roadway, with drivers and operators also instructed to assume control "any time they deem necessary."
Software Disengagements are when an issue is detected with inputs, outputs, or the decision-making ability of the driving system itself. It is further broken down into four subcategories, with Controls covering the actuator systems, Motion Planning being a failure to competently plan a path, Perception failing to sufficiently localize, detect, track, or classify objects in the environment, and Communications being issues where the driving system fails to send messages between processes in a timely manner or fails completely.
Starting from July, Apple started to categorize in more detail, introducing a new listing for Important Disengagements. According to the company, these are flagged for situations "that might have resulted in a safety-related event or a violation of the rules of the road."
The letter mentions an attached but unpublished report that describes the methodologies of identifying the disengagements, before and after the change in reporting. It is noted that the period until June 2018 is "over-inclusive when disclosing disengagements," whereas afterward only Important Disengagements are disclosed.
Between April 2017 and June 2018, Apple's test vehicles travelled 24,604 miles autonomously, experiencing 40,198 Manual Takeovers and 36,359 Software Disengagements during that time. For the post-July period, the cars are said to have travelled 56,135 miles autonomously, and to have experienced just 28 Important Disengagements, with two highlighted to be incidents where other motorists struck Apple's vehicles.
The figures provided in the letter certainly explain how Apple could be considered the worst-performing self-driving vehicle system on numbers alone, but taking the later period on its own, Apple's performance improves to a more credible 2,004 miles on average between Important Disengagements.
"Protecting the safety of all road users will always remain our primary focus," signs off Waydo. "As we continue to improve our capabilities and tools, we expect our future reporting will focus on Important Disengagements."