Street test measures Siri comprehension at 83%, accuracy at 62%

While the convenience of Apple's Siri comes with searching by voice, a new test has found that Apple's personal assistant is not yet a reliable enough search engine to entirely replace text input with Google.

The closer look at Siri and its capabilities was conducted by analyst Gene Munster of Piper Jaffray. To test Siri, Munster and his team gave the software 800 queries in a quiet room, with no headphones, and 800 queries on a busy Minneapolis street through Skull Candy microphone headphones.

The test measured Siri comprehension on the street at 83 percent, with accuracy at 62 percent. In the quiet room, Siri fared slightly better, comprehending 89 percent of queries and accurately answering 68 percent of answers.

Compared to searching Google, with text queries manually keyed in, Siri didn't fare as well. When Google was asked the same queries, the search engine provided an estimated 86 percent answer accuracy.

"We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app," Munster explained of his methodology to AppleInsider.

Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:

• "What team does Peyton Manning play for?" Siri would respond with the answer to the previous query.
• "Where is Elvis buried?" Siri looked for a person named "Elvis Buried."
• "Where am I?" Sometimes a map pin would be dropped in the wrong place.
• "When did the movie Cinderella come out?" Siri would attempt to find current theater movie times via Yelp.
• "How do I get from Boston to New York?" Siri responded: "I can only give directions from your current location. I can't give you directions to a place you are not in."

In Munster's test, he gave Google an A+ for comprehension and B+ for accuracy, while Apple's Siri was graded with a B for comprehension and a D for accuracy.

His testing also found that Siri relies on Google for 60 percent of its answers in iOS 5. But this fall, with the launch of iOS 6, Munster projects that Siri's reliance on Google will drop to 48 percent, replacing the search giant with Apple's own solutions for mapping and sports scores.

"While Siri is two-plus years behind Google today, we are optimistic about its future," Munster wrote. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."