Apple patents 'super resolution' multi-sensor cameras with cube prisms

As awarded by the U.S. Patent and Trademark Office on Tuesday, Apple's U.S. Patent Nos. 9,467,666 and 9,466,653 detail a multi-sensor camera assembly capable of achieving maximum image quality in a minimal amount of space. The system splits incoming light into at least three (red, blue and green) wavelengths using a series of prisms, directs and captures the rays with independent light sensors and combines the resulting data into a "super resolution" image via specialized software.

In one embodiment described in the '666 patent covering "Miniature camera super resolution for plural image sensor arrangements," Apple notes the use of color splitters allows for enhanced image resolution compared to single-sensor solutions as a lower portion of light is absorbed by filters.

Conventional single-sensor cameras commonly apply a Bayer pattern filter to derive color (typically through a demosaicing or software interpolation process) from CCD or CMOS sensor data. By contrast, three-sensor cameras gather and apply nearly all incoming light — in some cases three times the amount of a single-sensor array — passing through the beam splitter to a final output image.

Apple's IP refers to the beam splitter as either a Philips prism or smaller stacked cube variation, the latter described in detail in the '653 patent for a "Digital camera with light splitter," itself an extension of a patent granted in 2015. In a cube configuration, sensors collecting red, green and blue light information are embedded into ceramic substrates and positioned on three sides of the prism. The light splitter is designed to direct specific wavelengths to each sensor with minimal waste.

In addition to enhanced light gathering capabilities, three-sensor arrays offer better performance in situations calling for polarization imaging. Typically, polarizing filters shed 50 percent of incident light to increase visibility of specific targets, but the same results can be achieved with the splitting cube as it enables polarization sum and difference imaging. Infrared imaging is another area that would benefit from multiple wavelength-dedicated sensors.

As for deployment in smartphones, Apple illustrates a "folded" camera design in both of today's patents. To minimize system depth, a mirror mechanism is used to bounce incoming light through lens elements positioned orthogonal to the phone's Z-axis. For example, the objective lens can be installed as it appears on current iPhones, while the imaging sensor is positioned at some other location in the chassis with its optical axis arranged at a right angle to said lens.

This design holds advantages far beyond space saving benefits. The mirror, for example, can be mounted on a miniature motor to enable optical image stabilization, while the long connecting tube offers additional room for extended zoom capabilities.

Whether Apple plans to implement the inventions in a shipping device remains unclear, though the company already owns a number of patents covering three-sensor, three-lens systems, and cube splitter designs and related imaging technology.

The '666 patent goes on to explain super resolution algorithms, which in part use natural misalignments in the relative positions of the arrays to sample a given scene at higher spatial resolution. Super samples are also gleaned from edges detected in sub-sampled sites, as well as color information and artifacts.

Apple's super resolution camera patent was first filed for in July 2015 and credits Richard J. Topliss and Richard H. Tsai as its inventors. The cube splitter patent update was filed for in March 2015 and credits Steven Webster and Ning Y. Chan as its inventors.