The latest Apple patent applications published by the US Patent and Trademark Office feature a number of filings relating to its self driving car efforts, including gesture recognition for both passengers inside the vehicle and for pedestrians diverting traffic on the road.
The self-explanatory "Gesture based control of autonomous vehicles" application from September 19, 2017 describes a system for a self-driving car to be given instructions by a passenger in situations where a choice needs to be made. Noting that the cars could drive between locations with minimal direction from people onboard, Apple suggests there needs to be a way to tell the vehicle what course of action to take.
According to the filing, the car's screen displays a number of options the passenger may want the car to perform, such as selecting where to park the car when the destination is reached, changing speed, or to change the direction of travel. While some options will be presented when required, the user can also bring up a list of currently-appropriate actions to perform in various ways, including a verbal prompt, with options selectable by voice or by gaze detection, among other techniques.
A notable inclusion is the system's use of gesture controls, performed within a designated space called the "interaction zone" in the car. A hand entering the space can bring up prompts on the screen, which can be selected by performing specific hand and finger poses and movements.
Analyzing body movements is also a key part of the "Traffic direction gesture recognition" application, also filed on September 19 last year, where a self-driving car can follow the directions of a person on a road directing vehicles. While technologies including real-time traffic condition monitoring and GPS-based applications do aid self-driving systems in selecting routes to follow, they do not take into account instances where a person on the road is driving vehicles away from a spot.
This can include police and other officials directing traffic at an intersection where existing traffic management systems have failed, but also other pedestrians. For example, another road user could stand in the road and purposefully stop traffic from traveling down a route in the event of an accident, or after a concert or public event.
The application details ways external sensors of a self-driving car could detect the presence of passengers on the road, work out if they are acting as a traffic director, and attempt to recognize the gesture being performed. As well as acknowledging to the director the command and acting upon it, the director's existence and commands could also be transmitted to other nearby vehicles, allowing them to make necessary route changes before being manually diverted.
The "Cognitive Load Routing Metric for Vehicle Guidance," filed on September 21 last year, determines the best route to take to a destination from multiple options, based on the complexity of the route. Affecting both driver and self-driving systems, the patent discusses how a routing mechanism can take into account things that can make the road difficult to drive.
For example, using an infrastructure database, it can take into account the number of lanes on a roadway, the narrowness of a road, the existence of traffic signals, and street lights, along with real-time vehicular and pedestrian traffic data, and details of the kind of expected weather conditions for each route.
Apple's "Vehicle Control System" application from September 18 covers how a self-driving car can interact with other self-driven and manually-driven vehicles on a highway. Using the multitude of external sensors, the car tries to determine the motion of other nearby vehicles, and either reacts to their predicted movements or perform other maneuvers it needs to accomplish in transit.
Flowcharts accompanying the application show how a self-driving system should work for changing lanes, starting from receiving the intent to change lane then moving to determine the car's kinematic state, detecting vehicles in a target lane, estimate the headway, and then to determine how feasible it is to change lanes before performing the action.
The same system also has to take into account the potential overtaking maneuvers of other vehicles, or in cases where it cannot make a move, planning out how it will perform its intended moves in the future.
Sensor array used by Apple for self-driving system testing
Apple regularly applies for patents, but there is no guarantee that ideas it files will necessarily make an appearance in a future product or service.
The company is currently operating a fleet of vehicles registered with California's Department of Motor Vehicles for self-driving system testing. It is unclear what Apple's intentions are for its self-diving technology, but current suggestions include a platform for ride-hailing services usable by third parties, and the rumored electric vehicle.
The safety of self-driving vehicles has recently been questioned, following a test-related fatality earlier this week. Uber has halted its self-driving tests in Arizona following the death, but based on video from the car's cameras, the local police chief suggests it is unlikely to have been the company's fault due to the pedestrian's actions.