Apple AR may record full spatial sound needing fewer microphones

Apple's drive to have Apple AR hardware with high quality audio has led to research on how to create full spatial sound without developers having to record with large numbers of microphones.

Apple has persistently pursued the idea that Apple AR requires excellent audio as well as video in "Apple Glass," or other devices. That drive has previously included developing a new audio format, but now Apple is looking further at how such audio can be recorded.

"Processing of audio signals for spatial audio," is a newly-granted patent that is concerned with making high-quality audio recordings that work in a 3D space, without necessarily needing a great deal of equipment.

"3D audio rendering can be described as the processing of an audio signal (such as a microphone signal or other recorded audio content)," says Apple, "so as to yield sound produced by stereo speakers, surround-sound speakers, speaker arrays, or headphones that is perceived by the listener as coming from a particular direction or all around the listener in three-dimensional space."

"For example," continues the patent, "one or more of such virtual sound sources can be generated in a sound program that will be perceived by a listener to be behind, above or below the listener, or panned from one side of the listener to another."

The benefit of 3D audio is that it makes Apple AR a more enveloping, realistic experience. However, Apple says that the way that this audio is captured can be done in two ways, which it refers to as linear or parametric.

At its heart, the linear method means recording multiple microphones that are actually arranged in a 3D physical space. The parametric method means calculating how to play back audio so that it appears to have been recorded in such an environment.

"[In linear systems a] high number of microphones may be needed to achieve a quality spatial resolution," says the patent. "The number of microphones that are present in devices, however, can be limited by both physical factors (e.g. space and power consumption) and economic factors (e.g. cost and device complexity)."

Linear has the benefit that it places "a light computational load on a processor," because the spatial element comes from where the audio was recorded in relation to all of the microphones.

"In contrast," claims Apple, "parametric spatial sound processing can offer considerably improved spatial resolution for a relatively low microphone count, which is achieved by making some underlying assumptions on the sound field being recorded."

Detail from the patent showing one processing workflow for taking recorded sound and replaying it in a 3D space

"For example, one such assumption is that, at each point in time and at each frequency (i.e. in each processed sub-band) there is only one active sound source," continues the patent. "

That's not necessarily true, though, and when "this assumption is violated, under moderately reverberant, highly reverberant or diffuse conditions, the resulting output audio can become contaminated with unwanted processing artifacts."

Consequently, the patent is about methods of processing audio to "increase the spatial resolution achievable from a microphone array with a low count of microphones, while reducing the likelihood of processing artifacts and reducing processing cost or complexity."

Apple's patent goes into detail about how any 3D audio may be processed in order to be played back through any system, not specifically AR. "The system (which can take the form of a device or article of manufacture) can be, for example, a laptop computer, a desktop computer, a mobile phone, a smart phone, a tablet computer, a smart speaker, or an infotainment system for an automobile or other vehicle," it says.

We already have the 16-inch MacBook Pro which claims to have studio-quality speakers and improved speakers versus prior models in its chassis. Applying Apple's new patent could mean that future models will be able to record 3D audio without an array of external microphones.

This patent is credited to eight inventors, including Juha O. Merimaa, Martin E. Johnson, and Joshua D. Atkins. These three have previously been listed on related applications to do with making accurate 3D audio for AR.