Apple's patent for a remote collaboration system uses digital pens and 'large digital surfaces'

The U.S. Patent and Trademark Office on Tuesday granted Apple a patent that improves on large displays, like Microsoft's original Surface project, by adding digital pens and the internet, to give users a powerful, cost-effective collaboration tool that can be used to hold interactive meetings almost anywhere.

Apple's U.S. Patent No. 8,402,391, simply titled "Collaboration system," describes an enhanced user interface that leverages "large digital surfaces" and digital pens to create an environment in which a number of users can collaborate on various projects.

The invention largely centers on digital pens, mentioning consumer products like IOGear's Mobile Digital Scribe and tools from Anoto by name, as a means of inputting data into the collaboration system. These specialized pens have built-in sensors, either visual or motion-based, that capture user movements in real time, thus allowing for the digital reproduction of whatever is written on paper.

From the patent summary:

In one embodiment, users can add content to a digital system by using a pen that streams coordinates so that input to the digital system may be based on conventional pen and paper handwriting.

Apple gives the example of a user streaming pen coordinates to a collaboration server. From there, the system can share the information with other collaborators who are connected to the session, with the data showing up on an electronic whiteboard or other large interactive display. Here the invention notes that a user's mobile device, such as an iPhone, can be used as a type of bridge between the pen and a wireless network. Collaborators can be in the same room, or situated remotely, and the writer can choose to stream their notes or upload them at chosen intervals.

By utilizing digital pens instead of dedicated table PCs, the invention looks to cut down on costs while at the same time enhancing the collaboration process. Since the design doesn't call for multiple large displays, any number of users can effectively participate around a central collaboration station.

Moving on to these "large digital surfaces," the patent recognizes that some systems suffer from occlusions, either in screen blockage or shadows cast by a user's hand. The occlusions may pose as a hinderance to interacting with the on-screen menus, or otherwise negatively affect performance.

In one embodiment of Apple's invention, the menus are drawn to avoid the occlusion areas, adapting to a user depending on their "handedness," orientation and position on the tabletop or in front of the electronic whiteboard.

Instead of traditional drop down menus, the patent proposes an interface that can register clicks, drags and other UI actions oriented to a user's hands. The motions or gestures can also be recorded in the occluded area despite not having any visual feedback.

When a user wants to invoke a menu, they can use a special gesture with the digital pen, or press a dedicated button, which then triggers a specialized menu to appear around the pen's tip. The "pie menu" can contain a point and click area and a gesture area, each of which holds a plurality of icons or other visual assets that help the user navigate their options.

Further embodiments include gesture recognition, selective omission of errant touches by elbows and the like, and various descriptions of pie menu functionality.

Apple's '391 patent was first filed for in 2009 and credits Bernard Doray, Paul To, Michael Haller, Peter Brandl and Thomas J. Seifried as its inventors.