Siri on HomePod Asked 800 Questions and Answered 74% Correctly vs. Just 52% Earlier This Year

Apple analyst Gene Munster of Loup Ventures recently tested the accuracy of digital assistants on four smart speakers by asking Alexa, Siri, Google Assistant, and Cortana a series of 800 questions each on the Amazon Echo, HomePod, Google Home Mini, and Harmon Kardon Invoke respectively.


The results indicate that Siri on the HomePod correctly answered 74.6 percent of the questions, a dramatic improvement over the speaker's 52.3 percent success rate when Loup Ventures asked it a similar 782 questions in December 2017.


Siri on the HomePod remained less accurate than Google Assistant on the Google Home, which correctly answered 87.9 percent of questions in the test. Meanwhile, Alexa on the Echo and Cortana on the Invoke trailed Siri on the HomePod, correctly answering 72.5 percent and 63.4 percent of questions in the test.


Munster attributed the HomePod's improved accuracy to "the enabling of more domains in the past year," as a series of software updates in recent months have enabled the speaker to make and receive phone calls, schedule calendar events, set multiple timers, search for songs by lyrics, and more.

Methodology


Loup Ventures says it asked each smart speaker the same 800 questions, and they were graded on two metrics: whether the query was understood and whether a correct response was provided. The question set was designed to "comprehensively test a smart speaker's ability and utility" based on five categories:
  • Local – Where is the nearest coffee shop?
  • Commerce – Can you order me more paper towels?
  • Navigation – How do I get to uptown on the bus?
  • Information – Who do the Twins play tonight?
  • Command – Remind me to call Steve at 2 p.m. today.
The venture capital firm said it continues to modify its question set in order to reflect the changing abilities of digital assistants. "As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive," said Munster.

Results by Category



Google Assistant on the Google Home correctly answered the most questions in four out of the five categories in the test, but fell short of Siri on the HomePod in the "command" category, according to Loup Ventures:
HomePod's lead in this category may come from the fact that the HomePod will pass on full SiriKit requests like those regarding messaging, lists, and basically anything other than music to the iOS device paired to the speaker. Siri on iPhone has deep integration with email, calendar, messaging, and other areas of focus in our Command category. Our question set also contains a fair amount of music-related queries, which HomePod specializes in.
Loup Ventures found the HomePod and Google Home stood "head and shoulders above the others" in both the "local" and "navigation" categories due to propriety data in Apple Maps and Google Maps respectively. "This data is a potential long-term comparative advantage for Siri and Google Assistant," said Munster.

What's Next


Siri's shortcomings were brutally exposed on the HomePod, which relies almost entirely on the assistant to function, so anecdotal evidence of improvement is a good sign, but Apple still has significant ground to make up against its competitors.

Loup Ventures acknowledged that Siri remains limited on the HomePod compared to its expanded capabilities on the iPhone. "This is partially due to Apple's apparent positioning of HomePod not as a 'smart speaker,' but as a home speaker you can interact with using your voice with Siri onboard," said Munster.

The venture capital firm said it will continue to compare the HomePod to other smart speakers over time amid rapid advancements in artificial intelligence.

Related Roundup: HomePod
Buyer's Guide: HomePod (Neutral)

Discuss this article in our forums

Gene Munster Shares WWDC Predictions: Beats Product With Siri Integration, Improvements to AI and AR

Ahead of next week's Worldwide Developers Conference, Loup Ventures analyst Gene Munster today shared his predictions for the features and services that Apple will unveil during the event.

Munster expects Apple to debut new Siri, AR, AI, and Digital Health functionality, including a Beats-branded accessory (presumably a speaker) that includes Siri integration, much like the HomePod. Some of Munster's predictions have been previously covered in rumors shared by Bloomberg's Mark Gurman, but Siri integration in a lower-cost Beats product is a new prediction.

We expect Monday's keynote to be highlighted by extending the reach of Siri (most likely adding new domains, opening HomePod to more capabilities, and integrating Spotlight), along with additional AI tools (new Core ML extensions).

We also anticipate new features around digital health (privacy and device management) and ARKit (development tools).

Expect Siri integration with Beats.

Collectively, these announcements advance the ease of use and intelligence of Apple's mobile and desktop experiences.
According to Munster, Apple may be planning introduce a $250 Beats-branded product that will offer Siri integration similar to the HomePod, allowing Apple to "advance its digital assistant ambitions" with a more affordable option. Apple currently sells a Beats Pill+ speaker for $179.95, and the device has not been updated in some time.

Apple is going to announce a new "Decade Collection" at WWDC according to a Best Buy leak, but that collection is limited to existing headphones in new colorways and does not appear include new products or a new speaker. It's possible that Apple does have a new Beats product ready to unveil, and a recent somewhat sketchy rumor did suggest that Apple's "low-priced" HomePod would be under the Beats by Dre branding.

That rumor would make some sense if Apple is indeed planning on introducing a Beats-branded speaker product that includes Siri integration. Siri competitor Alexa is available as an option on many speakers outside of Amazon's own, and Apple could be planning to follow in Amazon's footsteps. Obviously, though, it's not clear if the rumor and/or Munster's prediction are accurate.

Munster has several other predictions for features and services coming at WWDC. Specifically, he expects Apple to introduce new Siri domains, with support for "things like navigation and email" and integrated Spotlight Search to better improve Siri's performance compared to Alexa and Cortana, the AI assistants used by Amazon and Microsoft, respectively.

Munster also believes Apple will introduce new domains for CoreML, the machine learning SDK that Apple introduced with the launch of iOS 11. Munster doesn't offer details on what the new domains might be, but at the current time, CoreML offers features for developers like real time image recognition, search ranking, text prediction, handwriting recognition, face detection, music tagging, text summarization, and more.

Previous rumors have said Apple plans to introduce support for multiplayer augmented reality games, and Munster believes Apple will also introduce "subtle new developer tools" to improve AR development and lead to more compelling AR apps.

Similarly, rumors have indicated Apple is working on a Digital Health tool that will let parents better monitor the amount of time children are spending on iOS devices. Munster says Apple could also include additional features that notify users when data is being shared with developers and new device management features aimed at curbing "screen time and digital anxiety."

Munster's full range of predictions for WWDC can be read on Loup Ventures, and our iOS 12 roundup contains all of the other iOS-related rumors that we've heard thus far.

Related Roundup: HomePod
Buyer's Guide: HomePod (Buy Now)

Discuss this article in our forums