Apple is working on improving the capabilities of computer vision to make it detect and identify subjects more accurately, such as in situations where both the camera and the subject are in motion, with one method suggesting the use of thermal imaging to help determine what elements in view need to be tracked.
Images from Flir One thermal imaging system for iPhone
The United States Patent and Trademark Office published an Apple patent application on Thursday titled "Method And System For Determining At Least One Property Related To At Least Part Of A Real Environment." The application specifically relates to computer vision, namely the use of cameras to analyze a scene, and to acquire useful data from what is seen.
The filing identifies an issue with existing computer vision analysis techniques in dynamic environments, in that many algorithms are constructed with a static environment in mind, but that the camera setup is the only moving object, one that is also the only thing where its pose can be definitively tracked.
If there are multiple moving objects in a scene, the accuracy of camera pose tracking is reduced, or even disabled depending on the motions of items within a scene. Also, the motion of objects could cause issues with recognition due to the occlusion of view by the camera.
While a 2D camera view can be translated to points in 3D space, the motion of objects in this space could also cause issues with 3D positioning of the camera.
One of the application's answers is to use a system of two cameras of differing types, consisting of a primary thermal imaging camera and another normal camera. While both could provide effectively the same viewpoint, the thermal imaging camera could provide heat map data to the algorithm to accompany the imaging data provided by the second camera.
The algorithm could analyze the thermal image and combine that with the imaging data to determine what items in a scene are separate from others, adding heat alongside color, contrast and other attributes typically used for analysis.
For example, if there are people playing soccer in a field wearing green uniforms, the thermal image would display heat where the players are located, on a cooler background of the grass. The algorithm would determine there are objects in the heated sections of the thermal map, and use that to single out the players in the normal image.
This could also be used to determine occlusion of a flat image filmed by a similar camera setup. This system could use thermal imaging to identify a warm hand covering part of a 2D image, with the data potentially used to alter the display, such as by hiding the element being blocked by the hand.
In cases where a building is being analyzed, such a system could be used to identify nearby pedestrians, which could then be automatically removed from a final composition of the structure, when put together with other images from the same view.
While the patent application would have uses in consumer imaging and video editing, as well as augmented reality, it may have more of a use in self-driving cars. Apple is exploring the concept with Project Titan, an operation that is trialing self-driving vehicle systems and sensors, and one where the addition of thermal cameras would be useful and relatively easy.
Apple has other patents and applications in the self-driving field, including some that affects what a self-driving car could potentially see. For example, one from early September describes using specialized headlamps to highlight areas of the road to warn the driver, but such a system could also light the road strategically to help vehicle vision systems see better.
As always, Apple files a large number of patent applications with the USPTO on a regular basis, and relatively few make their way into consumer products. Its existence in a USPTO filing is not a guarantee that Apple will actually adopt the described concept in the future, but does indicate the fields it is currently exploring.