Apple details 'Siri for Mac' desktop virtual assistant in new patent application

While Apple's Siri has been an iOS staple since 2011, the tech has yet to make the jump to OS X. An exhaustive new patent filing, however, details a very similar desktop version of the virtual assistant that can perform dictation, high-level system commands and even act as a "third hand" for Mac users.

As published by the U.S. patent and Trademark Office on Thursday, Apple's massive 92-page patent application for an "Intelligent digital assistant in a desktop environment" describes a technology that goes far beyond Mac's current voice dictation feature, and can easily be considered a "Siri for Mac."

Like the current iteration of Siri, limited to iOS, Apple's desktop version is able to process natural speech and text input to perform actions like completing tasks, inputting and retrieving data, conducting searches and more. Further, the filing points out that commands are to be taken in context based on deduced user intent. In other words, Siri for desktop will use speech recognition to decipher and remembering contextual clues.

Source: USPTO

Driven in cooperation with an off-site server, Siri for Mac has an identical backend to Siri for iOS. The user-facing Siri asset can be integrated into a standalone app invoked by a special mouse or keyboard gesture, or may be tasked to run in the background, listening for a special activation phrase like the "Hey, Siri" implementation in Apple's forthcoming iOS 8.

In some embodiments, an icon of the virtual assistant can reside in the dock, notifying the user that it is active and ready for service. This and other modes allow the user to speak commands and start dictations from anywhere in the operating system.

Voice input is of specific importance to the patent's disclosures, as the technology is looking to augment keyboard and mouse input, or in some cases replace the physical tools altogether. The usual answer/response method seen with Siri for iOS is applied to the desktop variant, though more advanced operations can be performed given the extra computing power afforded by a proper computer.

Apple's desktop Siri version uses "focus selector," or mouse cursor, to help users apply context to a command. For example, the assistant can perform a copy operation for an asset under a hovering cursor, or may "hold" a batch of highlighted files for later retrieval. In essence, the focus selector plays the same role as an iOS device user's finger, except with substantially more options for control thanks to the Mac's GUI.

With mouse access, users can drag and drop files, folders and other assets onto the Siri dock icon for processing. For example, a folder full of pictures can be sorted by telling the assistant, "Sort by date," or "Sort by location." Following completion of the operation, a secondary command may be issued, or the icon may throw up a list of extra options in text or voice form.

Multiple assets can be dragged and dropped onto the assistant for advanced collation, merging and advanced comparison operations. A number of other operations are also described in the full patent document.

A particularly interesting function describes "Using Digital Assistant as a Third Hand." Apple notes that current graphical user interfaces are somewhat limiting in areas like multitasking. For example, while multiple applications may be running in a desktop environment, users are usually restricted to interacting with only one window or asset at a time.

In some cases, a user may be performing one task and find that a second or third action is required to facilitate unbroken focus on the primary operation. For example, while typing in a text editor, a user may want to access a picture or piece of information from the Web. Invoking a digital assistant by voice to perform this secondary task saves mouse clicks and allows continuous focus be paid to the text editing task at hand.

Alternative embodiments include creating folders while having multiple files selected on the desktop, asking the assistant to resize pictures, sending emails and changing speech into text, among many other functions. In theory, Siri for Mac would be able to perform any operation otherwise available to the user.

It is unclear whether Apple plans to roll out a version of "Siri for Mac" in the near future, though the latest OS X 10.10 Yosemite betas do not have such functionality built in. Rumors have long said Apple is planning to release a desktop virtual assistant — going as far back as Siri's introduction on iPhone — but there has yet to be any hard evidence of a working OS X solution.

Apple's Siri for Mac patent application was first filed for in 2013 and credits Julian K. Missig, Jeffrey Traer Bernstein, Avi E. Cieplinski, May-Li Khoe, David J. Hart, Bianca C. Costanzo, Nicholas Zambetti and Matthew I. Brown as its inventors.