Apple researching how to make televised sports more interactive through AR

Apple is researching how to enhance live sports with viewers able to choose their camera angle, and single out elements they want more information on.

You might watch basketball games for the athleticism, the competition, or just to cheer on your local team, but Apple thinks you might want more. If you've ever found yourself watching a player and thinking he has a nice shoe, Apple believes you should be able to learn more about it — and that AR will help you.

A new patent application stops short of saying you'd be able to buy that shoe, but this isn't really about footwear appreciation, it is about selling. "Method, Device, and System for Generating Affordances Linked to a Representation of an Item," US patent application 20200175275, also suggests other situations where you might want to examine different aspects of something you're watching, but it does keep on coming back to basketball.

"For example, when a user is watching a basketball game on a television, the user may see a basketball player wearing a particular shoe that the user is interested in," it says. "Additionally, when a user watches a tutorial (e.g., a live video feed or a pre-recorded video feed), the user may not be able to see how to perform a technique on an item in the tutorial due to obstructions in the tutorial or the angles presented in the tutorial."

You might think this just means the tutorial has been done poorly, and you might think that focusing on a shoe means the footwear has more grip in it than the game. But Apple wants you to be able to say you want more information about something on the screen, or that you want a different angle — and preferably do so on one device.

Supplementing what the user is viewing

"When a user watches a video stream that includes items, the user is unable to glean further information or perspectives of those items without using an auxiliary device," it continues. "[The] user would use a separate device to access more information about the particular shoe that the user sees on the television."

The perspectives element is key as this would mean the user being able to control the camera angle they watch, at least to some degree. However, what the patent says we really want are "methods for displaying, an indication of a real-world interaction with a second item as a simulated reality (SR) overlay on the first item."

Most of the patent concerns the display being a head-mounted device, as this is another case of Apple investigating a form of Augmented Reality. In also covering every possible base, Apple's proposal suggests this could also be done with "one or more displays," including an iPhone.

So you could have the video you're watching on one screen, or shown on your headset, and have secondary information on another screen, or another part of the headset display. "[The system would display] an indication of a real-world interaction with a second item as an overlay," says the patent.

Detail from the patent showing how one element of video can be examined in a second screen

As well as being shown the basketball game or the tutorial, the system needs a way for the user to be able to say what element he or she wants to focus on. One clear possibility is that the user might be able to just tap on a touch-sensitive main display.

However, Apple also proposes that it should be possible for the user to interact with a tutorial's demonstration, or the player's shoe, in a combination of SR and AR. "For example, the overlay of the SR representation of the basketball shoe is overlaid directly on the representation of the live feed," it says, "such that the representation of the basketball player appears to be wearing the SR representation of the basketball shoe."

So the real-life player's real-life shoe could be shown instead as an AR item that a viewer might touch, turn, and examine. "When the device switches from the representation of the live video feed to the SR environment, [it] may pause or bookmark the representation of the live video feed in order to minimize interruption to the user," explains Apple.

More prosaically, though, that shoe could be shown in a more familiar Picture-in-Picture display. Less prosaically, the "virtual objects may be projected, for example, on a physical surface or as a holograph, so that an individual, using the system, observes the virtual objects superimposed over the physical setting."

Apple's plans for AR and sports

The application is credited to inventor Alexander Oser, who as yet does not appear to have any prior patents. However, Apple itself has very many AR patents and patent applications from Apple, which include specifically related ones.

"Adaptive Coding and Streaming of Multi-Directional Video, " US Patent application 20200177927, concentrates on how to offer a viewer the choice of multiple video angles.

That might be through presenting a 360-degree view to a headset. "For example," says this application, "a viewer may employ a head mounted display and change the orientation of the display to identify a portion of the multi-directional image in which the viewer is interested."

Detail from the separate patent application about how a user could choose from 360 degrees of video

Similarly, there is a related patent application to do with how a user could select these views, or point at the basketball player's shoes. "Computer Systems With Finger Devices For Sampling Object Attributes," US Patent application 20200174583 details how a user could get more precise control through wearing sensors on one or more fingers.

The multi-directional video patent is credited to nine inventors including Xiaohua Yang, whose prior work includes patents on aspects of video coding.

The finger-device patent application is credited solely to the prolific Paul X. Wang, whose many related patents include ones to do with fingertip tools for AR and VR, and touch-based devices with haptic feedback.

While this all adds to Apple's ever widening range of AR patents, it may also now fit in with the company's plans for sports on Apple TV+. Most recently, Apple has been revealed to be in negotiations to add live sports to the service.

Those talks are described as being preliminary, but seemingly Apple has ramped up its work in the area by poaching an Amazon sports content executive.