Apple Car cameras may identify landmarks & objects of interest, detect broken windows

A van by Apple-acquired startup Drive.ai bearing external camera sensors

Digital imaging is an important part of any self-driving car, as it is used in concert with other systems to read the road and to spot potential hazards. Having the collection of cameras could be beneficial in other ways, which Apple has considered in a pair of related vehicular imaging patents.

Object Recognition

For the first patent, "Method and system for providing at least one image captured by a scene camera of a vehicle," Apple suggests using the onboard cameras to capture a photograph of a nearby object. While this would typically be relied upon for object recognition for driving, such as identifying bollards, traffic cones, or pedestrians, Apple's proposal is a little more left-field in purpose.

The filing reasons the driver or a passenger may be interested in identifying an object of interest for themselves, potentially to find other types of the same object. Furthermore, a user may want to purchase that object, if possible, and could be directed to a store offering it for sale, or for "hands-on checking."

It may also be impractical for a person in a vehicle to take a photograph of something outside while on the move, such as if they are pressed for time and cannot stop the car.

Apple's description of the patent basically revolves around telling the car to take a photograph of something outside the vehicle, triggering the car's outward-facing cameras to capture an image or multiple images. These photographs can then be used either as a record of a moment in a scene or for an object-recognition system.

The filing suggests that the Apple Car photo system could monitor the user's gaze to determine where they are looking at the time of a vocal request. For passengers, a gesture could be detected to perform the same task.

In the case of object recognition, the images can be passed to a system to recognize objects in a scene, determine the one of interest to the user, and then to acquire more information about that item.

The patent lists its inventors as Lejing Wang and Thomas Alt, and it was filed on June 21, 2019.

The idea of using external cameras for alternative purposes has been explored before, with the 2019 patent application of "Automated capture of image data for points of interest" used to take photographs of areas of an environment that may be worth knowing about to a user. That same application suggested that a user could ask for a photograph to be taken of a scene.

Broken Windows

The second patent, "Window defect sensing and image processing" is more straightforward, but revolves around the use of cameras inside the vehicle, rather than those facing out. This could include cameras used for gesture recognition or eye gaze detection, which may be used for other car functions.

In short, the patent suggests illuminating car windows, and using imaging sensors to capture a photograph of the window. The image is then used to determine if there are any cracks or defects in the window, which could form a safety hazard for the passenger or driver.

Apple proposes the use of edge lighting to put light through the inside of the window pane, with surrounding modules able to illuminate different sections. These light modules can transmit either typically unfiltered light or graze lighting, namely directional light at an acute angle, which may be better at highlighting cracks and other issues.

Rather than relying on one imaging sensor, Apple suggests the use of multiple sensors to take images, which can then be compared. As each sensor is located in a different known position, this can enable enhanced processing to be performed on the images, such as determining the external view of the vehicle and eliminating that from the data to leave just the window and defects.

Once an issue has been determined, the findings can be communicated to the driver or other users through a notification or alert.

The system isn't necessarily limited to just cracks and scratches, as it could also be used for more everyday features. It is suggested a window cleaning system could be produced to spot clean specific points of the window itself, rather than cleaning the entire window each time.

The patent's inventors are listed as Clarisse Mazuir, Malcolm J. Northcott, Jack E. Graves, James R. Wilson, Christopher D. Jones, and Martin Melcher. It was filed on May 15, 2018.

Apple has come up with other ways to detect cracks in windows, such as one revealed in early August that relied on an infrared light-blocking layer, ohmic heating currents, and resistive measurements to determine crack sizes and locations.

Apple files numerous patent applications on a weekly basis, but while the existence of a patent indicates areas of interest for the company's research and development efforts, they do not necessarily mean the idea will appear in a future product or service.