Apple Glass could make AR believable with accurate 3D audio

Apple's AR headset or "Apple Glass" smart glasses may offer realistic sound effects to go with its digital imagery laid over a real-world view, by accurately reproducing a sound in such a way that it matches where the sound would come from in 3D space.

Gaming and virtual reality systems all offer some level of audio feedback to the user to accompany visuals. In the case of augmented reality, the experience of seeing a digitally-created item in the real-world view could easily be accentuated by a sound effect when it is interacted with in some way.

While audio can help sell the illusion of video, there are some limitations. A typical stereo headphone setup may provide some limited left and right differentiation at a basic level, while virtual sound systems are able to fake 3D positions by altering the properties of the sound effect to mimic distortions based on its positioning.

For example, a sound played behind the user's head could be muffled, while one from the front could be much clearer. However, while this can be useful, the effect isn't truly believable as differences in physiological elements like ears, the head, and body can alter what a person expects to hear from a noise in a specific position.

In a patent granted to Apple on Tuesday titled "Head related transfer function selection for binaural sound reproduction," Apple believes it can make the effect of virtual 3D sound positioning better by having what it refers to as the Head related transfer function (HRTF). These are effectively a pair of acoustic filters for each ear that characterize the transmission of sound from one place in an environment to the ear.

Each HRTF varies on a number of factors, so as to match differences between people in terms of spatial hearing expectations. While it is relatively time-consuming or impractical to create a HRTF for an individual user using consumer electronics with any real level of accuracy, there is the possibility of using either a generic "average" HRTF, or selecting a HRTF that is closest to the user's needs.

An example ratings matrix for a group of users experiencing different HRTF settings.

The HRTF is then used as part of a binaural simulation digital signal processing algorithm to reproduce an audio recording as binaural sound, which in turn is played through headphones.

The patent centers around the idea of selecting the nearest HRTF to what is needed, by whittling down the database to a select few. The pruning process can include taking away options that are far from basic characteristics of the user, such as height, gender, race, and age.

Cutting down the database to a smaller number increases the chance of the most-correct HRTF being used, the patent advises. After the initial settings, the system could then narrow down its choice to the most generic from the reduced list, to select the most appropriate HRTF for use.

To help the system more, it is suggested that a binaural measurement could be taken by the audio device for the user. If the device is being used by only one user, the measurement would only need to be performed one time, but it would offer potentially other information about which HRTF would be best.

The system would search for the best match for a listener's HRTF, not their specific settings.

It is also suggested that, by grouping users, the mass usage of different HRTFs could be monitored to help improve selection. For example, knowing that users aged 20-25 prefer HRTFs with certain properties will mean the system would err towards those particular filters if it encounters someone matching those settings in the future.

The filing lists its inventors as Darius A. Satongar, Jonathan D. Sheaffer, Martin E. Johnson, and Peter V. Jupin.

Apple files numerous patent applications on a weekly basis. While the existence of a filing indicates areas of interest for the company's research and development efforts, they do not guarantee the concept will be used in a future product or service.

Apple's audio processing has resulted in products like the HomePod, as well as its pass-through noise cancellation feature in the AirPods Pro, but it is still continuing to develop more sound-related ideas.

In December 2019, it was granted a patent for a "System to move a virtual sound away from a listener using a crosstalk canceller," which performs a similar spatial audio trick in making a sound appear to emanate from a different place than the device's speakers.

One suitable place the patent could be applied is in "Apple Glass," Apple's long-rumored AR glasses. As it is thought to use AR, and would apply digital items on a real-world environment for the user to see and interact with, perfecting the audio playback could help make the non-existent item more believable to the wearer.

A related patent called "Display Devices with Multimodal Audio" suggests a headset could have earpieces that adjust to suit the type of audio being presented.

2 Comments

dewme 5738 comments · 10 Years

About 4 years ago

I wonder if Apple Glass AR can be used to locate missing devices that have Apple AirTags attached to them, say like an Apple AirPower charging dock, 14" MacBook Pro, 32" iMac, and mini-LED iPad Pros?

All these rumors of mythical Apple products are starting to pile up stories around them to the point that future generations scraping the web archives will think that some of these products actually existed even if they never shipped. Or Apple can start shipping these products and the authors of these articles will look like geniuses. Or should I say, super geniuses for those who've already attained the mere genius rank....

MacPro 19838 comments · 18 Years

About 4 years ago

AppleInsider said:

Apple's AR headset or "Apple Glass" smart glasses may offer realistic sound effects to go with its digital imagery laid over a real-world view, by accurately reproducing a sound in such a way that it matches where the sound would come from in 3D space.

Spatial audio is offered in the AirPods Pro.

Gaming and virtual reality systems all offer some level of audio feedback to the user to accompany visuals. In the case of augmented reality, the experience of seeing a digitally-created item in the real-world view could easily be accentuated by a sound effect when it is interacted with in some way.

While audio can help sell the illusion of video, there are some limitations. A typical stereo headphone setup may provide some limited left and right differentiation at a basic level, while virtual sound systems are able to fake 3D positions by altering the properties of the sound effect to mimic distortions based on its positioning.

For example, a sound played behind the user's head could be muffled, while one from the front could be much clearer. However, while this can be useful, the effect isn't truly believable as differences in physiological elements like ears, the head, and body can alter what a person expects to hear from a noise in a specific position.

In a patent granted to Apple on Tuesday titled "Head related transfer function selection for binaural sound reproduction," Apple believes it can make the effect of virtual 3D sound positioning better by having what it refers to as the Head related transfer function (HRTF). These are effectively a pair of acoustic filters for each ear that characterize the transmission of sound from one place in an environment to the ear.

Each HRTF varies on a number of factors, so as to match differences between people in terms of spatial hearing expectations. While it is relatively time-consuming or impractical to create a HRTF for an individual user using consumer electronics with any real level of accuracy, there is the possibility of using either a generic "average" HRTF, or selecting a HRTF that is closest to the user's needs.

An example ratings matrix for a group of users experiencing different HRTF settings.

The HRTF is then used as part of a binaural simulation digital signal processing algorithm to reproduce an audio recording as binaural sound, which in turn is played through headphones.

The patent centers around the idea of selecting the nearest HRTF to what is needed, by whittling down the database to a select few. The pruning process can include taking away options that are far from basic characteristics of the user, such as height, gender, race, and age.

Cutting down the database to a smaller number increases the chance of the most-correct HRTF being used, the patent advises. After the initial settings, the system could then narrow down its choice to the most generic from the reduced list, to select the most appropriate HRTF for use.

To help the system more, it is suggested that a binaural measurement could be taken by the audio device for the user. If the device is being used by only one user, the measurement would only need to be performed one time, but it would offer potentially other information about which HRTF would be best.

The system would search for the best match for a listener's HRTF, not their specific settings.

It is also suggested that, by grouping users, the mass usage of different HRTFs could be monitored to help improve selection. For example, knowing that users aged 20-25 prefer HRTFs with certain properties will mean the system would err towards those particular filters if it encounters someone matching those settings in the future.

The filing lists its inventors as Darius A. Satongar, Jonathan D. Sheaffer, Martin E. Johnson, and Peter V. Jupin.

Apple files numerous patent applications on a weekly basis. While the existence of a filing indicates areas of interest for the company's research and development efforts, they do not guarantee the concept will be used in a future product or service.

Apple's audio processing has resulted in products like the HomePod, as well as its pass-through noise cancellation feature in the AirPods Pro, but it is still continuing to develop more sound-related ideas.

In December 2019, it was granted a patent for a "System to move a virtual sound away from a listener using a crosstalk canceller," which performs a similar spatial audio trick in making a sound appear to emanate from a different place than the device's speakers.

One suitable place the patent could be applied is in "Apple Glass," Apple's long-rumored AR glasses. As it is thought to use AR, and would apply digital items on a real-world environment for the user to see and interact with, perfecting the audio playback could help make the non-existent item more believable to the wearer.

A related patent called "Display Devices with Multimodal Audio" suggests a headset could have earpieces that adjust to suit the type of audio being presented.

I predict this will all be argued to be obvious rendering it ineligible for patenting by Google lawyers when Google rips off all Apple's technology in this area,.