Apple researching HomePod-style Spatial Audio for Apple AR

Apple AR could feature Spatial Audio, as already used in the HomePod, HomePod mini, or AirPods Pro — but also recorded on compact devices.

It may still be the case that we think of AR as being a visual medium, but Apple has again been researching the audio side of the whole experience. Following on from previous work investigating making audio sound 3D, it's now been researching more convenient ways to accomplish this.

"Audio system having audio and ranging components," is a newly-granted patent that is concerned with how to mimic accurate 3D sound with a minimum of equipment.

"In existing methods of reproducing a sound image to an observer in a virtual space, the microphones that are distributed around the actual space must be spaced apart from one another in order to rely on the detected sound for source localization," says Apple.

"Accordingly, such methods do not allow for a single, compact microphone array to be used for sound source localization, and equipment and installation costs of such methods can be substantial," it continues.

There is also the issue that once a whole series of microphones have been spaced out around each other for recording, it's time-consuming to move them. What Apple wants is a simpler system that's also suitable for portable recordings and playback.

It's particular important for Apple AR because audio is crucial to making an experience feel real. "Virtual reality and augmented reality environments can include virtual sound sources, which are computer-generated sound sources in a virtual space," says Apple.

Detail from the patent showing the arrangement of a compact recording device

"The virtual space can map to an actual space," it continues. "For example, a user may wear headphones in a room and the headphones can reproduce a sound to the user as though the sound is a voice of a colleague in front of the user, even though the colleague is actually in another room."

"As the user moves in the room, e.g., as the user walks forward five paces, the reproduced sound can change," says the patent.

To visually show movement or repositioning of virtual objects, the recording must be accurately able to identify the source location. "When the location is identified, the recording can be reproduced to the user in a manner that localizes the reproduced sound as if the sound is coming from a similar location in the room that the user occupies," Apple's patent says.

There may be no escaping the need for an array of microphones, but this patent proposes that they be supplemented by what Apple calls "a depth capturing device." When the two can "detect a point cloud and a local sound field," and a processor can "reconstruct a global sound field" from their data, it can "render a virtual reality environment to a user."

In other words, whatever system is used to figure out that a virtual colleague is five feet to your left, Apple AR audio can play their speech as if it's that close, and in that direction.

The key is in this depth capturing device. The patent goes to lengths to describe every possible such device, but it concentrates on cameras.

"For example, [a] camera array may be a stereoscopic system having two or more cameras," says the patent. "Several cameras can be adjacent to each other and/or spaced apart from each other and looking at a same space."

"Each camera can detect the same space from a different vantage point, and thus, by analyzing image data from each camera, a distance from [the] camera array 106 different points and/or objects within the images can be determined," it continues.

The patent also touches on how this audio can be played back to a user, but chiefly it's about the accurate recording — or relaying — of it first.

Playing back audio to authentically create 3D spatial sound

"For example, when a sound producing object moves backward within the field of view, e.g., away from the position of the capturing devices, delay and gain of audio signal can be adjusted to render virtual sound corresponding to the produced sound as retreating from [the] user within the virtual environment," says Apple.

"Similarly, [the] user can move through the virtual reality environment to change a point of view relative to virtual sound source," it continues, "...and as [the] user moves, delay and gain of audio signal may be adjusted to render virtual sound as coming from the correct location..."

This invention is credited to two inventors, Martin E. Johnson, and Jonathan D. Sheaffer. Both are previously credited on a different patent for recording spatial audio with fewer than usual microphones.

Stay on top of all Apple news right from your HomePod. Say, "Hey, Siri, play AppleInsider," and you'll get latest AppleInsider Podcast. Or ask your HomePod mini for "AppleInsider Daily" instead and you'll hear a fast update direct from our news team. And, if you're interested in Apple-centric home automation, say "Hey, Siri, play HomeKit Insider," and you'll be listening to our newest specialized podcast in moments.