Apple plans to make 'Apple Glass' comfortable for extended immersive experiences

Right from when Steve Jobs insisted the Mac had to have dialog boxes with rounded corners instead of straight rectangles, Apple has been known for sweating the details. This time, it's continuing to do so with a pair of patents including one that, following minute adjustments, has now been granted for a second time.

"Feedback coordination for a virtual interaction," the patent granted in September 2020, concentrates on something you've not realised in real life but would notice if an AR or VR environment got it wrong. When you're at a concert with a friend, the two of you are hearing subtly different sounds, just because of where you're sitting.

It's not that your pal is behind a pillar in the cheap seats, and you're in the front row. The same thing happens when you stand up or walk by everyone's knees on your way out. The music sounds different — infinitesimally so, but different — depending on where you are.

"Music processing systems... should ideally strive to improve temporal precision in musical performances in order to enhance music quality and user experience," says Apple. "This task, known as quantization, may involve presenting playback of sound at a time different from a time of performance of the sound..."

So you see the violin bow move, or you see a guitar riff start, but under Apple's proposal, you only hear the music at exactly the time you should — according to the artist or where you're standing. "[Quantization requires] determining that such a modified presentation is more aligned with intentions of the performer and/or the structure of a piece of music."

As well a your physical location relative to the real or virtual sounds, there is an issue of the equipment being used. "A feedback device (such as speaker/headphones, a haptics engine, or the like) often has a predetermined latency between initiation of associated feedback (such as audio, haptics, or the like) and user perception of the associated feedback due to, for example, hardware and/or transmission delays," continues Apple.

"In turn, feedback associated with a virtual interaction in a CGR environment should ideally strive for life-like coordination (or synchronization) between perception of the feedback and occurrence of the virtual interaction itself," continues the patent.

The details are extraordinarily fine, but the aim is simple. To enjoy something like a concert, or any experience with audio, for more than a short time, it has to sound real.

"However, existing CGR delivery systems continue to face challenges when it comes to effective and timely coordination (or synchronization) between the feedback and the virtual interaction itself," says Apple.

Apple's proposed solutions are all for head-mounted displays (HMDs), and are concerned equally with the audio itself, and with the physical position of the wearer. Not only does the patent describe detecting the HMD wearer's position in order to alter the sound, it even includes when the sound is being created by that wearer.

"Some existing CGR systems enable a user to play a virtual musical instrument by translating the real-world body pose of the user and trajectory information... into CGR interactions between a CGR avatar/representation associated with the user and the virtual instrument," says Apple.

Overheating HMDs

It's not the 1980s, so a user wearing a HMD and playing a virtual instrument can't wear a sweatband and be taken seriously. Yet all of this equipment requires processing power, and that means heat.

The patent that was first granted in 2020, and has now been granted in a finely modified form, is called "Structural thermal solutions for display devices." It's concerned with keeping wearers cool.

"Certain display panels, such as OLED and uOLED panels, for example, can operate at high temperatures," says Apple in this patent. "In known systems, such as wearable HMDs, for example, display panels are usually supported by a carrier, bezel, or other such structure, which is often formed from a lightweight material (e.g., plastic) to reduce the overall weight of the systems."

"These structures, however, offer little benefit in terms of thermal regulation," it continues. "Without proper cooling and heat dissipation, the panels (which are expensive and difficult to replace) can degrade over time, often resulting in irreparable damage or system failure."

As well as thinking of the wearer's comfort, then, this patent is also looking to the long-term wear on such devices. Apple points out that existing systems "will often incorporate additional components, such as heat sinks," to do this, but they come with problems.

"These additional components, however, create design challenges and add to the overall weight of the systems," says Apple. Its solution "addresses these challenges by providing a display system that integrates thermal solutions into structural components."

"This combined functionality not only reduces the overall number of components, complexity, and weight of the display system," continues Apple, "but increases thermal conductivity and improves thermal management to decrease operating temperatures and extend the system's usable life."

It's not that Apple has any particularly new method of dissipating or reducing heat. Rather, the company aims to leverage the hardware that already has to be present for a HMD to work.

These include the "display panel" plus "a carrier supporting the first side of the display panel." This carrier, or frame, is a "thermal solution to facilitate heat transfer away from the display panel."

There are further elements to do with including a heat sink where it can function but not adversely affect the wearer. And there are solutions that mean parts will move in order to "generate airflow."

This thermal patent is credited to five inventors, including Laura M. Campo, and Sivesh Selvakumar. Their previous related work includes a patent on regulating thermal temperatures for a HMD.

The patent regarding precision audio control is by Christopher T. Eubank, and Daniel P. Patterson. The former holds many related patents including ones to do with spatial audio for "Apple Glass" and other devices.