Apple details new 'AppleBot' web crawler used by Siri and Spotlight
AppleInsider Staff
Apple has confirmed the existence of a long-rumored web crawling service — first noticed last November — Â and provided some details of its operations in a recently-updated support document.
According to Apple, AppleBot is a crawler that primarily serves Siri and Spotlight. Both services offer web search capabilities, though until now it was thought that Apple routed queries through an existing search engine such as Google or Bing.
AppleBot operates similarly to Google's Googlebot, and Apple advises developers that it will follow Googlebot instructions if no AppleBot-specific instructions are found. Website operators are urged to contact Apple's network operations center if they have questions or concerns about the crawler.
It's not clear how long AppleBot has been operational in production. The support document first appeared in February, but was only populated with content last month.
One web developer spotted Apple's crawler last November, after seeing numerous hits from a crawler in the 17.0.0.0/8 block. That block is assigned to Apple, one of just a handful of companies that control an entire A block.
Malcolm Owen
William Gallagher
David Schloss
Amber Neely
