Apple wants to track every user movement for health, AR, and more

Apple already has technology such as accelerometers in the Apple Watch that mean it's able to automatically detect you're walking, even though you forgot to turn on the Outdoor Walk workout. It also recently applied for a patent for a camera on the Apple Watch which, as well as for photography, was about detecting movement.

Now two separate patent applications have been revealed, which despite being applied for a year apart, both aim to detect every movement — and use it.

"Posture Transition Detection And Classification Using Linked Biomechanical Model," was filed in September 2021, and only revealed now. Its 17 pages of detail — about 8,000 words — are all very specifically about detecting a change in posture.

"Existing mobile devices include motion sensors that provide motion data, such as acceleration and rotation rate of the mobile device," says Apple in this patent application. "For example, a digital pedometer can use acceleration data to count steps to determine that the user is walking or running."

"Although these broad activity classifications are useful for many mobile applications," it continues, "some mobile applications would benefit from knowing if a user's physical posture has changed, such as standing from a seated position or vice versa."

As ever with a patent application, the focus is chiefly on how something may be achieved rather what it might be used for. Nonetheless, there are clear use cases alluded to in the text, such as for when a user is wearing a headset and navigating both a real and virtual environment.

When the devices you are wearing can calculate the position of your entire body, and predict where movement is going to take you, it can alert you to real or virtual obstacles. This ability could presumably be used in the same way to help visually-impaired people walking through the real world.

Apple says little about use cases, but does have enough specifics to show that this posture detection is not just so the company can nag you about not doing your sit-ups.

"In a spatial audio embodiment, motion data from [a] headset and source device motion are fused in the source device," says Apple, "such that relative position and attitude can be calculated by the source device for use in head pose tracking for certain tracking scenarios to ensure that the spatial audio sound bed is properly centered."

"In particular, head pose tracking using relative motion is useful for determining whether the user's head is turning while their torso is stationary or whether the user's torso has moved," it continues. "Both types of motion would be sensed by the headset and would be indistinguishable."

That's a recurring point in the patent application, that it may already be possible to detect motion, but to not be able to correctly or fully interpret it.

Apple repeatedly says that its proposal is about using combinations of devices to determine what it needs. "Generally, the use of motion data from both the headset and companion device allows more complex user posture transitions to be detected," it says.

Full body pose detection

That combination of devices is also central to the second newly-revealed patent. "Full Body Pose Estimation Through Feature Extraction From Multiple Wearable Devices," filed in 2022, is concerned with detecting everything possible — using every device conceivable.

"Wearable devices are now prevalent in society," says Apple. "Many consumers wear both smartwatches and ear devices on a daily basis... [and many of these wearable devices include inertial sensors that can sense the motion of the user."

"For example, inertial sensors embedded in a smartwatch can capture a user's arm motion which can then be used by a fitness application to compute various fitness metrics, such as the amount of calories burned by the user during exercise," it continues. "Some ear devices (e.g., earbuds) include inertial sensors that control audio playback, provide microphone orientation data used in beamforming to reduce background noise, and head tracking for anchoring a spatial audio sound field played through the ear devices."

"[These can together] provide a more accurate full body skeletal model that can provide detailed insights into a user's movements," says Apple, "allowing improved progress tracking for various fitness activities or health monitoring."

"An additional advantage is that existing devices already owned by the user" continues the patent application, "can be used in combination with inexpensive compact sensor modules (inertial sensors, altimeter and camera) that can easily attached to different locations of the user's body or clothing to create a distributed sensing system."

"Using multiple cameras and inertial sensors placed at different locations of the user's body, combined with depth data for the full body allows for a more accurate three-dimensional skeletal model to be generated," says Apple.

"[This data] can be used in a variety of applications," it continues, "such as applications that detect if a person has fallen down or is sick, applications that autonomously teach proper workout regimes, sport techniques and dance activities..."

Patent applications always describe as broad a range of applications as they can, and in this case that goes from fitness to how such systems could "understand full-body sign language. (e.g., Airport runway signals, traflic policemen signals, etc.)."

These two patent applications are unusual in that they are concerned with creating a way to connect what already exists. But they're both right that we have the devices that can detect this data, so a system that correctly calculate full body poses and then exploit that for our health and fitness, is a good thing.

Or at least it is as long as this data stays with us.

Each of the two patent applications uses what looks like it's now a standard section about privacy. Rather than commenting on any specific privacy issues that the patent applications raise, the standard text assumes that "entities responsible" for such data collection "will comply with well-established privacy policies."

The posture patent is credited to four inventors, including Aditya Sarathy, whose previous work for Apple includes a patent application for "Posture And Motion Monitoring Using Mobile Devices."

Three inventors worked on the full body patent application, including Wesley W. Zuber. His prior work includes a patent for touch-based feedback, such as a haptic system in the Apple Pencil.