First Look: Using iPhone 4S with Siri voice assistant (with videos)

Most of the new features in iOS 5 simply run faster on the iPhone 4S, but one major new feature is exclusive to the new model: Siri voice assistant. Here's a look at how it works in action, responding to a variety of requests.

Apart from iPhone 4S, other iOS devices upgraded to run the free new iOS 5 continue to supply Voice Control, which responds to a variety of voice commands such as "what time is it?" or "play songs by MGMT," or "call Tim Cook."

The new Siri dramatically expands upon Voice Control, offering a much wider, more powerful vocabulary for creating Notes, composing Emails and Messages, navigating and changing Calendar events, setting new Reminders, and taking dictation within apps virtually anywhere you could type.

Siri also plugs into external services, integrating with smart artificial intelligence servers hosted by Apple to decipher not just basic commands but natural language questions. It then provides answers through Google or Bing, Wolfram Alpha, Wikipedia, Yelp, and Yahoo weather and stock market information.

Voice recognition

Siri offloads voice recognition to Apple's servers, which apparently run software licensed from Nuance. The text to speech component of Siri is very good, far easier to begin using and more accurate than other systems I've used.

More importantly, however, Siri isn't just voice recognition and dictation. The heart of Siri is a new AI system that works to understand what you mean, not just transcribe the words you say. So while it's nice to be able to talk rather than type, the real functionality of Siri is the ability to respond to what you want to do.

This isn't always perfect, but it's very usable. Generally, when Siri fails to understand what you ask, it's because of a lack of context. You can start over and the same words will be correctly recognized.

As a test of Siri's core voice recognition, I read back the previous lines into the iPhone 4S, and quickly got the following text back as a result. I wasn't speaking with any special diction and was using a normal speaking speed. The result isn't perfect, but is very good, especially considering that you can touch to select problems and correct them. Siri didn't drop in some of the punctuation (I was barely stopping for commas), and got a few words wrong (marked in bold below):

Siri offloads voice recognition to Apple servers which apparently run software license from Nuance. The text-to-speech component of Siri is very good far easier to begin using a more accurate than other systems I've used. More importantly however serious and just voice recognition and dictation. The heart of Siri is a new AI system that works to understand what you mean not just transcribe the words you say. so what's nice to be old to talk rather than type the real functionality of Siri's ability to respond to what you want to do. This isn't always perfect but very usable generally when Siri fails to understand what you ask is because a lot of context you can start over in the same words of me correctly recognized.

Idea recognition

It's apparent when using Siri that you don't have to say everything perfectly, or mouth just the right words in the just the right order, or even have what you say be correctly recognized, because Siri looks at the context of your conversation with it to provide a useful answer more often than not.

If the response you get isn't what you were looking for, simply repeating or correcting your request generally delivers what you want. You don't have to say things according to a specific set of commands.

This use of context to interpret your commands means that Siri is most likely to get confused if you suddenly change what you're talking about without signaling that you want to start something new. You can always request to "start over."

Voice navigation

Some of the most useful things Siri does is perform jobs that would otherwise require typing in a bunch of text. Asking for a map of an area, or directions to a destination, is a key example. Or asking for weather, stock prices, definitions of terms and so on. You can also type the microphone icon in the virtual keyboard to enter text via your voice in most apps.

To create new Notes or to address and compose emails, you don't even need to hit a dictation button. You just tell Siri what you want to do. In the video below, I ask for directions, pull up an answer to a conversion question, ask Siri to compose a new Note, and ask to "call my mom," which prompts Siri to ask what my mom's name is, cross reference her Contact, and present three phone numbers I have associated with her. It will remember who I have identified as my mom, sister, brother or father next time I ask.

Also shown in the video is how Siri handles interruptions from multitasking notifications. The alarm that goes off while Siri is transcribing a Note for me has no impact on the user interface or the accuracy of the Note.

On page 2 of 3: Conversational context

Conversational context

In a second video, I ask about a weekday, a question that isn't correctly recognized; asking it again works however. An answer about stock prices and weather work as expected.

A general knowledge question and a request for a city's population get quick answers from Wolfram Alpha. Siri's use of context in answering questions is indicated when I ask about breakfast in San Francisco's Cow Hollow neighborhood. At first, while correctly recognizing the text of my question, Siri doesn't provide a useful answer. But when I ask again, despite failing to recognize the words I used, the context of the previous request resulted in Siri offering useful information.

I immediately changed the topic to getting gas in the Mission District, and Siri responds with breakfast options there instead. But when I repeat

that I'm looking for gas, Siri provides the right information.

Siri also helps you pull up information saved in your Notes (including those you've just dictated), offering a powerful way to delegate things you're supposed to remember to your smartphone. This feels addictive and easy.

Siri also helps you schedule and change your calendar, although in my example, it failed to immediately do what I asked. However, with a little persistence the system not only got it right, but also presented helpful information about potential calendar conflicts. After I thought I was finished with the calendar, I asked an unrelated question, which Siri stumbled on as it continued to talk about my calendar.

After appearing to double book the appointment I had been working on, I simply asked Siri to review what it had done to tomorrow's schedule. Everything was fine, no duplications and no missing events due to the change not being finished.

I tried asking about changing Pesos into US dollars, but Siri seemed stuck in my Calendar and unable to figure out what I was trying to do, despite catching the words correctly. However, after asking to "start over," Siri seemed able to get back on track and provide a useful answer.

On page 3 of 3: Siri outside of iPhone 4S

Siri outside of iPhone 4S

It's not clear how much of Siri's exclusive availability on the iPhone 4S is due to its far more powerful processor, and how much is simply Apple reserving the feature as an incentive to upgrade.

In previous iPhone hardware releases, Apple similarly reserved video capture as an exclusive of the iPhone 3GS, and made FaceTime exclusive to iPhone 4. While those features have been hacked to work on earlier models, they don't run nearly as well on the less powerful hardware.

Were Apple to make Siri available on the iPhone 4, it would likely work half as fast and half as well, likely tainting the experience of the majority of those able to immediately try it out. On iPhone 4S, the new feature works brilliantly, although not always flawlessly.

Apart from being far more sophisticated in what it can do, Siri also has a significant limitation over the Voice Control it replaces on the 4S: it only works if you have a network connection. In Airplane Mode, Siri won't work at all. The old Voice Control will, but its commands (and accuracy) are so limited that it isn't likely to get used that much.

Siri promises to change how often people use voice command by making it far more likely to work as expected, and to do functions that are valuable enough to remember how to use it. At some point, Apple will likely bring Siri to the Mac along with iOS 5's new iMessage chat, just as it did over the last year with FaceTime in iOS 4.

Siri is also likely to make its way to iPad 2, which is certainly powerful enough to run the software. It's less certain that Siri will be adapted to last year's iPhone 4 or the current crop of iPod touch models, none of which use the same speedy A5 chip that power the iPad 2 and iPhone 4S.