The U.S. Patent and Trademark Office on Tuesday granted Apple a patent for an imaging system that processes data from multiple sensors to detect and correct for various photographic anomalies, such as defective pixels, lens aberrations, and ineffective demosaicing, among others.
Source: USPTO
Apple's U.S. Patent No. 8,493,482 for a "Dual image sensor image processing system and method," describes a unique technology that can interleave imaging data from two or more imaging sensors, resulting in higher quality photos than a setup with a single sensor.
The lengthy patent notes that said device can be either mobile or non-portable, but does mention a number of Apple products by name, including the iPhone, iPod, and a few Mac models. Disregarding device form factor, the invention can be used on any suitable camera system, which can be described as a sensor, color filters, lens elements, processing circuitry, and storage. Other components specified include various device hardware not directly associated with the camera module.
According to the patent abstract, there are two main modes in which the imaging system operates. One uses a single sensor, as is the case with many digital consumer cameras, while another uses image signal processing to process data from two distinct sensors.
While the idea of having two or more sensors disposed in a device is a novel, the backend processing and control logic comprising the image processing circuitry is most impressive.
Going into highly technical detail, the document describes the various functions of the system, starting with the detection of image flaws pertaining to filters, dead pixels, luminance, chroma, and others. Responsible for this step of processing is the front-end logic, which compares raw image data against a litany of statistical predefined metrics and statistics stored in system memory.
Illustration of raw image data and applied Bayer filtering techniques.
In one embodiment, the system moves data through the ISP pipe processing logic before outputting to a display device. The image undergoes further processing in this step, which in some cases may involve the replacement of image data from one sensor with data from a second sensor, depending on the level of corruption. For example, if a sensor's image data presents a threshold that indicates a dead, "hot," "stuck," or otherwise defective pixel, the system will compensate with data from the second sensor.
The patent notes that the method can be used for both still pictures and video by using the same image processing circuitry. Key to the invention is not the per-pixel replacement of data, but more importantly the logic that can interleave the information from two separate frames into one before outputting to a display or memory.
Per-pixel motion history lookup for video.
While current rumors don't have Apple implementing a dual-lens or dual-camera system in its iOS device lineup in the near future, the technology could one day be implemented to bring a new level of image quality to compatible products.
Apple's dual image sensor patent was first filed for in 2010 and credits Guy Cote, Jeffrey E. Frederiksen, Joseph P. Bratt, Shun Wai Go, and Timothy J. Millet as its inventors.