Memoji telling people to say 'cheese' could improve iPhone photos

Apple wants iPhone users to take better photos of other people, by using a virtual character to convince subjects to look into the camera.

Apple is known for using photography as a marketing point for its iPhone range. Its long-running Shot on iPhone campaign has been successful in demonstrating the kinds of images a user can take using an iPhone's camera.

It has also worked hard to improve the technology used in its cameras over the years. This includes the adoption of a DSLR-style Sensor Shift for optical image stabilization in the iPhone 12 Pro Max.

While Apple can easily change the camera and how images are processed, it has less control over what a person actually takes a photo of in the first place.

If you're taking photographs of a person or child, you generally want the subject to be looking into the camera lens. The subject could be unknowingly uncooperative by not looking at the camera for long enough, or not actually paying attention to the photographer properly during the shot.

While the photographer could indicate to the subject to look in their direction, their actions to distract could also pull their attention away from operating the camera in the first place.

In a patent granted to Apple by the US Patent and Trademark Office on Tuesday titled "Electronic device with image capture and stimulus features," Apple plans to fix this problem. In essence, Apple has patented its technological version of a photographer telling people to "Say Cheese!"

The patent suggests the use of a device with two cameras and two displays, such as an iPhone with an extra screen. One camera and screen are on each side.

As normal for a typical camera, one external-facing camera sensor and the photographer-facing screen are for conventional photography uses. The key of the patent filing is the combination of the photographer-facing second camera and a subject-facing second screen.

Apple proposes that the second camera could be used to read the facial features of the photographer to create a virtual avatar. This avatar then can be animated and displayed on the second screen, shown to the subject.

The iPhone can already animate Memoji based on the expressions of the user.

This would, on a base level, allow the photographer to directly communicate with the subject without taking their eyes off their screen. The use of an avatar would also serve as an attractive element for people to look at, effectively bringing their attention to the camera.

The avatar may not necessarily be a copy of the photographer's facial expressions. Instead, it could create a cartoony representation of the photographer, then show an animation and play a sound as a further reward to the subject once the photograph is taken.

A version of the system could also use some extra processing to further improve the arrangement, and take more of the photo-taking workload. Using gaze detection, the avatar could change state when it detects the subject is looking in its direction, again rewarding the subject for their attention.

This same gaze detection could also trigger the shutter, ensuring that the image has the subject looking in the camera's direction. A similar facial-monitoring system may go one stage more, instructing the subject to smile as well and to take the shot when that is accomplished.

Filed on February 26, 2019, the patent lists its sole inventor as Stephen B. Lynch.

Apple files numerous patent applications on a weekly basis. While the existence of a patent indicates areas of interest for Apple's research and development efforts, there's no guarantee it will appear in a future product or service.

Unusually for Apple's patent filings, it actually has most of the technology in place for the concept to become a reality. The iPhone's use of front and rear cameras, facial recognition, Memoji avatars, and onboard processing covers the majority of the patent's ideas, with it only really lacking a screen for the subject to look at.

Photography has appeared in a few previous patent filings, such as the September 2020 patent to improve the appearance of photos shot underwater. A deep learning system has also been proposed to apply art styles to photographs.

Others in its catalog relate to the technology of taking the photograph in the first place. For example, in December it proposed ways to completely hide a camera in an Apple Watch or iPhone when it isn't in use.