Evidence shows Apple operating a mysterious Web crawling bot

A developer has discovered that Apple appears to be crawling HTML websites with its own automated bot, but the exact purpose of the mysterious homegrown software remains unknown.

Evidence of Apple's Web crawler, written in the Go programming language, was first discovered by developer Jan Moesen, who has found hits from the software dating back to Oct. 15. Others have discovered significant traffic to their websites from Apple's mysterious crawler, which appears to be shut off from time to time.

Requests from Apple's bot are only for HTML pages, and do not access the accompanying CSS, JavaScript or image files from a website. Moesen found that the Apple Web crawler identifies itself as "Mozilla/5.0 (compatible; Fetcher/0.1)".

The crawler was found originating from an IP address starting with "17.", which would indicate that the bot is emanating from Apple's servers.

It's unknown whether the Apple bot is an official project for the company, or if an employee built and ran the system in an unofficial capacity from a company machine.

It's possible that Apple's efforts could be related to Spotlight enhancements in both OS X Yosemite and iOS 8. Both were updated this fall with improvements that allow users to access Web search content directly, without the need to access a third-party search tool such as Google or Bing.

Though Apple doesn't compete directly with the likes of Google or Microsoft Bing, the company has been working to provide specific information directly within Siri search results. Doing so makes it unnecessary to provide potential links through the official Siri Web search partner, Bing.