Apple working on tech to auto blur sensitive images in live video conferences

Apple is researching technology that would identify confidential or sensitive elements in a video frame, and automatically blur or replace them with a static image to protect privacy.

The use of video conferencing has dramatically expanded since the coronavirus lockdown and that means we've all become aware of confidentiality problems. Whether we've done it ourselves or just heard the stories, we now all know examples of Zoom conferences going wrong — and Apple wants to help.

Whether it's just that your colleagues are concentrating on the books in your office, or it's the Spanish journalist whose affair to the world, and his wife, was revealed by how an undressed colleague walked behind him, it's usually about the background.

"Per Pixel Filter" is like a specific and dynamic version of the blurred or replaced backgrounds users can now have in Zoom or Skype.

Video conferencing is immensely useful, says the patent application, and "provides a better user experience than just transmitting or receiving voice data."

"However, some users decide not to transmit video in order to avoid the risk of transmitting certain (e.g., undesirable) information," it continues. "For example, users that are located in areas with confidential information often decide not to transmit video during video calls in order to reduce the risk of leaking the confidential information."

There is also an issue that, "some users decide not to receive video" because of the issue. And "some enterprises prevent their employees from transmitting or receiving video in order to reduce the risk of leaking confidential information."

Under Apple's proposal, the video stream technology would identify different elements of every frame. Where Zoom and Skype can now distinguish the difference between the person speaking and their background, this would much more intelligently determine individual parts of the speaker's environment.

"[The] method includes obtaining an image data frame," it continues."[It means] generating a respective pixel characterization vector for each of the plurality of pixels." In other words, the system effectively connect and label the pixels that represent an object, and then it can work with that grouping.

Detail from the patent showing an object redacted

Given that an object or part of an object can be identified, "the method includes synthesizing a first modified image data frame." It creates a different pixel for each one in the grouped object, and that could be anything the user wants.

"[This] obfuscating the representation of the object includes blurring the pixels that correspond to the object," continues Apple. "Obfuscating the representation of the object includes setting the value of the pixels to a particular color (e.g., black, white, etc.). [Or] replacing the representation of the object with a representation of another object."

So for instance, Apple's system could replace an object with a blurred version of the that same original pixel. It could redact part of the image by turning an entire object black, white, or any other color.

Or it could effectively substitute the object for something else that's decided by the speaker or by his or her company. "For example, a display policy administered by an enterprise specifies that the displayed information is to be replaced by a company logo."

The application is credited to inventor Jack Greasley, who with another company has previously had a patent for manipulating the lighting effects on a bitmap of an image.