Apple details 'Siri for Mac' desktop virtual assistant in new patent application

As published by the U.S. patent and Trademark Office on Thursday, Apple's massive 92-page patent application for an "Intelligent digital assistant in a desktop environment" describes a technology that goes far beyond Mac's current voice dictation feature, and can easily be considered a "Siri for Mac."

Like the current iteration of Siri, limited to iOS, Apple's desktop version is able to process natural speech and text input to perform actions like completing tasks, inputting and retrieving data, conducting searches and more. Further, the filing points out that commands are to be taken in context based on deduced user intent. In other words, Siri for desktop will use speech recognition to decipher and remembering contextual clues.

Driven in cooperation with an off-site server, Siri for Mac has an identical backend to Siri for iOS. The user-facing Siri asset can be integrated into a standalone app invoked by a special mouse or keyboard gesture, or may be tasked to run in the background, listening for a special activation phrase like the "Hey, Siri" implementation in Apple's forthcoming iOS 8.

In some embodiments, an icon of the virtual assistant can reside in the dock, notifying the user that it is active and ready for service. This and other modes allow the user to speak commands and start dictations from anywhere in the operating system.

Voice input is of specific importance to the patent's disclosures, as the technology is looking to augment keyboard and mouse input, or in some cases replace the physical tools altogether. The usual answer/response method seen with Siri for iOS is applied to the desktop variant, though more advanced operations can be performed given the extra computing power afforded by a proper computer.

Apple's desktop Siri version uses "focus selector," or mouse cursor, to help users apply context to a command. For example, the assistant can perform a copy operation for an asset under a hovering cursor, or may "hold" a batch of highlighted files for later retrieval. In essence, the focus selector plays the same role as an iOS device user's finger, except with substantially more options for control thanks to the Mac's GUI.

With mouse access, users can drag and drop files, folders and other assets onto the Siri dock icon for processing. For example, a folder full of pictures can be sorted by telling the assistant, "Sort by date," or "Sort by location." Following completion of the operation, a secondary command may be issued, or the icon may throw up a list of extra options in text or voice form.

Multiple assets can be dragged and dropped onto the assistant for advanced collation, merging and advanced comparison operations. A number of other operations are also described in the full patent document.

A particularly interesting function describes "Using Digital Assistant as a Third Hand." Apple notes that current graphical user interfaces are somewhat limiting in areas like multitasking. For example, while multiple applications may be running in a desktop environment, users are usually restricted to interacting with only one window or asset at a time.

In some cases, a user may be performing one task and find that a second or third action is required to facilitate unbroken focus on the primary operation. For example, while typing in a text editor, a user may want to access a picture or piece of information from the Web. Invoking a digital assistant by voice to perform this secondary task saves mouse clicks and allows continuous focus be paid to the text editing task at hand.

Alternative embodiments include creating folders while having multiple files selected on the desktop, asking the assistant to resize pictures, sending emails and changing speech into text, among many other functions. In theory, Siri for Mac would be able to perform any operation otherwise available to the user.

It is unclear whether Apple plans to roll out a version of "Siri for Mac" in the near future, though the latest OS X 10.10 Yosemite betas do not have such functionality built in. Rumors have long said Apple is planning to release a desktop virtual assistant — going as far back as Siri's introduction on iPhone — but there has yet to be any hard evidence of a working OS X solution.

Apple's Siri for Mac patent application was first filed for in 2013 and credits Julian K. Missig, Jeffrey Traer Bernstein, Avi E. Cieplinski, May-Li Khoe, David J. Hart, Bianca C. Costanzo, Nicholas Zambetti and Matthew I. Brown as its inventors.