Future HomeKit control could involve pointing iPhones at devices

Apple could enable an iPhone's owner to determine what items should be controlled with gesture detection and determination on-device of what device the user is pointing at.

The universal remote control is a well-known and long-standing feature of the living room, which enables a single controller to be used with various home entertainment devices, such as a television, DVD or Blu-ray player, or a satellite or cable TV set-top box.

While the universal remote is a simple idea for managing many items that use a similar interface and are stored within the same room, it does have its limitations. For example, users have to press the right button on the remote to switch between controlling devices, which could add friction to the user experience if they intend to turn on a DVD player and instead turn off the television if they forget to make the switch.

In a patent granted to Apple by the U.S. Patent and Trademark Office on Tuesday, the filing for "Controlling electronic devices based on wireless ranging" offers a solution to inter-device switching issues, by handling the change on behalf of the user. At its simplest, it is proposed the remote control will switch based on what device it is pointed at.

The patent suggests there is a remote control that connects wirelessly, and within a specified range of the intended device. Using sensors including a compass, an accelerometer, a gyroscope, proximity by ultrasonic audio chips and radio signal strength, the remote has the opportunity to determine its position of itself in relation to devices it can control within an environment, as well as its orientation and direction.

Apple's overview of the proposed system

If the remote is capable of knowing what device it is pointed at, it can set itself up to communicate with a particular piece of hardware. As orientation and height could also feasibly be determined, it is also plausible for the remote to know what device it should control in a stack of devices, as is typical of a home entertainment setup.

The patent also explains that there could be an intent gesture to trigger the detection, such as a quickly-stopping sweeping movement. The movement gesture could also be tied to a command that could be transmitted to the required device, such as turning on the appliance when the remote is pointed at it rotated face-side down, though more complex gestures could also be made.

While the immediate application of the concept could be applied to nearby items that could save the user from getting up and manually changing a device's settings, it is also proposed that it could be used for other purposes.

There is the suggestion that there could be a virtual object set up for the remote that in fact transmits commands for an item or device that is further away than within the room. By pointing at the virtual object's "position," which in the real world could be represented by hardware controlled by the remote object or a token talisman, similar commands could be sent to the far-off item and altered.

Detecting the distance and the intent gesture could help tell the remote what instruction to pass on.

An example of this could be the main light in a room, controlled by a main central controller in a building. The light's position in the ceiling could be designated the virtual object's real-world position, and pointing the remote at it could send an instruction to the building's controller to turn the light on or off, or by rotating the remote while pointing, dimming the light.

Apple files numerous patent applications with the USPTO on a weekly basis, but while there is no guarantee that the concepts described will make it into a future product or service, it does however indicate areas of interest for Apple's research and development efforts.

It is arguable that Apple already has most of the components in place to enable such a system, in the form of HomeKit. Apple's smart home platform allows for the remote control of devices and appliance via the Home app, accessible via an iPhone or iPad.

There is already some functional geofencing capabilities within HomeKit, with scenes able to be enabled based on the user arriving home, while a patent application in January suggested it could use a hyper-local geofencing system within a home to provide more proximity-based functions.

The iPhone is also arguably an ideal device for such a system, as it is already able to be used as a remote control for some devices via apps, as well as managing a HomeKit installation. Given it includes a compass, gyroscope, accelerometer, audio reception, and radio communications systems, an iPhone contains all of the hardware the patent needs for it to function.

For it to become a reality, Apple would have to come up with a way for the iPhone to be fed the location of other hardware it could control, something which could use a form of ARKit application with object identification capabilities for the initial registration. It would also require manufacturers of HomeKit-enabled items to be more receptive to commands via the smart home framework.