Deep learning networks could be used to smoothly apply art styles in Photos

Apple is developing deep machine learning systems that could smoothly apply artistic styles to images or video sequences taken on an iPhone, likely as a future feature in Photos.

The Cupertino tech giant has applied for patents in the realm of machine learning before, including techniques applied to disparate systems such as GPS data or in-air gesture analysis. Now, it appears, Apple is looking at stylistic video editing.

In a patent granted to Apple on Sept. 29 titled "real-time adjustment of temporal consistency constraints for video style," the company outlines a system that can apply artistic styles extracted from one or more images to one or more target images.

The patent notes that users may want to apply more complicated and technically complex artistic styles to images or videos — effects that are beyond simple filters that use machine learning to apply a value to each pixel. More specifically, Apple is focusing on using artificial intelligence to "imbue the 'essence' of a particular artistic style" to an image or video.

Credit: Apple

However, this type of image manipulation isn't feasible on a portable device in real-time due to thermal and processing constraints, as well as errors in the style transfer process. To remedy that, Apple outlines techniques to use deep neural networks (DNN) and other machine learning techniques.

"The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity," the patent reads.

In one scenario, the patent contends that the system could analyze a device's hardware and software constraints to intelligently determine which DNN should be used to stylize an image or sequence of images. If the transfer process becomes more complicated over time, the system could smoothly switch to another DNN version to continue the process without any hiccups.

As an another example, the patent notes that the system could be trained to handle a certain type of input image independently from other images in a sequence. In practical usage, this could result in a slider that users could adjust to increase or decrease the temporal consistency.

The patent lists Francesco Rossi and Bartlomiej W. Rymkowski as inventors. Both have worked on Apple patents before, though only related to similar artistic styling and deep neutral network systems.