Malcolm Owen
Improvements to the way Siri recognizes names of small businesses and local points of interest are due to the use of language models designed for specific locations, Apple's Machine Learning Journal reveals, helping the virtual assistant understand local names for nearby places.
Virtual assistants like Siri are easily capable of understanding the name of prominent businesses and chains, such as supermarkets and restaurant franchises, writes the Siri Speech Recognition Team, with queries concerning lesser-known or regional businesses tending to provide less accurate results. In automatic speech recognition systems (ASR), the team notes this is a "known performance bottleneck" for accuracy, with those further along the long tail of a frequency distribution less likely to be correctly identified.
Apple attempted to improve this for Siri by taking into account the user's location in queries. There would also be two different types of model being used, with a general language model (LM) working alongside a geolocation-based language model (Geo-LM), with the latter becoming more useful if the user is in its coverage area.
ASR systems typically comprise of two components, consisting of an acoustic model that analyzes the properties of speech alongside the language model that analyzes the word usage. Apple noted that the system didn't adequately represent words and names for local points of interest and how they were pronounced, with the more obscure names and combinations also appearing at a very low frequency in the LM training data.
The low frequency means that, in a general LM, the local business name is less likely to be picked up compared to another location, word, or phrase.
In Apple's solution, it defined a number of geographic regions covering most of the United States, producing a Geo-LM for each area. These local versions are used depending on the user's location, though if the user is outside all defined regions or Location Services are disabled, the general LM is used instead.
There are 169 Geo-LM areas for the U.S, based on combined statistical areas defined by the U.S. Census Bureau, covering approximately 80 percent of the population. Each area consists of "adjacent metropolitan areas that are economically and socially linked," measured by commuting patterns.
In Apple's testing, there was no real change in accuracy for general queries, but there was a relative error reduction of 18.7 percent for point of interest-based searches between general LM and Geo-LM usage. In point of interest tests in eight U.S. metropolitan regions, the relative error reduction between general LM and Geo-LM increased, with the localized version performing better by between 41.9 percent and 48.4 percent.
Apple suggests that, because of the limited impact on system speed, the regional coverage of Geo-LM still has room for improvement, but a general language model will be here to stay. "It is essential to continue providing a global Geo-LM in addition to regional LMs," writes Apple, "so that ASR can handle long-distance queries and cases with users located outside supported regions."
International expansion of the program could also occur to languages other than U.S. English, with Apple noting "The method and system proposed here are language independent."
Apple still has some way to go to catch up to Google's level of accuracy for virtual assistants. A July group test revealed Siri has improved its accuracy considerably over the last year to 78.5 percent, as well as increasing its comprehension of queries to close to 100 percent, but under the same test, the Google Assistant achieved an accuracy of 85.5 percent.
William Gallagher
Christine McKee
AppleInsider Staff
Chip Loder
19 Comments
Apple needs to improve the international functions of this and that of Maps. The biggest difference in performance between Google’s software products and Apple’s now are that Google’s are far better covered internationally than Apple’s. I know that international rollouts take some time, but Apple has to do better if we’re going to stop seeing the heavy criticism of Apple over this.
about Siri in general, there’s a reason why Siri lags. There is a difference in the way Apple, Google, Amazon and Microsoft are doing this. The others are using the technique called Deep Learning for this. Apple is using a technique, the name of which just popped away from me, that works very differently.
deep learning, which I think is misnamed, uses an AI technology that requires that vast amounts of information are force fed to it. The AI then figures out links, both mechanically and logically related, and can do some minor reasoning. The AI itself doesn’t have much depth to it. It also requires that the work be done on remote server farms, the way Siri worked in the beginning. Deep learning is being questioned as a l9ng term technology, with experts stating that it won’t be able to advance much further. An article recently stated that the end was pretty close. Google is attempting to modify the was it works for them.
apple isn’t working that way. Much of what Siri does now is done one the device itself. This is why Apple has been working on the on-chip AI so hard. Apple’s method doesn’t rely on massive amounts of data, particularly personal data. The AI is actually deeper than than that in deep eating. But, it’s a much harder technology to master. Because of that, Siri lags. But as we can see, Siri is improving faster. In another two or three years, it should begin to regain the lead. It’s been said that Apple’s ongoing persistence with privacy is holding Siri back, and it’s true. But Apple’s work is far less dependent on personal data, and if they can pull this off, then they’ll have a big advantage.
For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly.
Clearly there’s still work to be done.