Inside iCloud: Apple's new Documents & Data cloud service

In place of the simple cloud storage and web hosting services in today's MobileMe, including iDisk, iWeb and Gallery, Apple is preparing an innovative new cloud-based Documents & Data service. Here's how it works.

The road to iCloud: the host without the most

Beginning over ten years ago with iTools, Apple's online services for Mac users have morphed through a variety of incarnations, perpetually being a mixed bag of success and failure. After a false start as a variety of general purpose web apps ranging from iCards for postcards and iReview for discussing other websites, the most successful aspects of Apple's online efforts have turned out to be email and messaging services.

These were enhanced in the move to .Mac, which eventually began to offer web client access, and further improved upon during the transition to MobileMe, when Apple added push messaging features targeted toward its new audience of mobile users.

Among the least successful aspects of Apple's online web service experiments have been the web hosting and virtual disk features associated with iDisk. In part, this has been because what Apple has offered in these areas has never been competitive in price or performance with services dedicated to blogging or online file sharing.

While the company attempted to differentiate its iDisk-related hosting offerings with desktop Finder integration on the Mac, native clients for iOS devices, and easy to use publishing tools integrated into iWeb and iPhoto, few users were sold on the idea of paying for MobileMe principally or even significantly because of its online sharing aspects.

Rethinking document cloud services

In the planning process to develop its next generation of cloud services, Apple decided to radically rethink its offerings in the area of file sharing. The result has been two separate but functionally similar features of the new iCloud: Photo Stream, a way to automatically push photos taken on one mobile device to every other Mac and iOS device the user has, and Documents & Data, a parallel solution to enable applications to do something similar for the files and data they use.

Photo Stream solves real problems for users without requiring them to think about how to use the new service. All they need to do is configure their devices and computers once with their iCloud account, something they're likely to do anyway to use messaging and sync services.

Once their equipments is all tied to the same iCloud account, Apple automatically pushes new photos up to the cloud when captured, and down to the user's various photo libraries on iOS, Mac and Windows PCs. No more cables for manual syncing, no more manual media imports, and no more looking for what device is storing the photos you want to access.

Essentially, Photo Stream is push messaging for media, with the same kind of effortless management afforded by IMAP-style email as opposed to the confusion of having various computers each set up with manual POP-style email, where messages end up stuck on one system and missing from another.

iClouds' Documents & Data delivers the same kind of genius level of simplicity for every other app that uses files. Apple outlined to developers how the new system would work at its Worldwide Developer Convention this summer under strict NDA. Fortunately for the curious rest of us, word quickly leaked out about how Documents & Data would work.

How iCloud's Documents & Data feature works

Apple outlined publically in general terms what Documents & Data does: it propagates changes made to a file on one device across every other. Similar to Photo Stream, creating a new Keynote presentation on your Mac will result in a magical multiplying of the document and an immediate background delivery of the new document on your iPad, on your iPhone or iPod touch, and within the iWorks web app so you can access it remotely from any browser.

There's no need to manually save the document to something like iDisk, or to look for where it might be stored in the file system, and no need to perform any manual sync between devices. It just happens. When you edit that presentation on your iPhone, all the changes you make similarly appear in every other instance of that file across your devices, again without needing any sort of thinking or involvement on your behalf.

The concept is deceptively simple, but it involves quite a bit of thoughtful engineering on Apple's behalf. There needs to be support built into both the desktop and mobile operating system, as well as a web interface for the cloud-based storage. Third parties also have to incorporate support for iCloud's Documents & Data in their apps. Apple has demonstrated how to do with with its iWork apps, the first to incorporate support for the new feature.

Rather than just throwing up a way to copy around files automatically, Apple took an carefully considered approach that borrows from its experience in curating the iOS and Mac App Stores. This isn't just a network-synced iDisk. Instead. each app that opts in to the iCloud Documents & Data feature is accorded a secure storage space of its own, just as iOS apps each live in their own secure sandbox, inaccessible from other apps.

Using iCloud-aware apps therefore won't eat up users' free storage on iCloud, just as Photo Stream or iTunes' media, apps and iBooks use won't count against the free 5GB of storage every iCloud user gets.

More importantly, a rogue app created to prey upon Mac users won't be able to destroy or corrupt the documents or data stored by iCloud-savvy apps, adding a new level of security within the user realm of the computing model. Currently, any piece of malware (or simply a buggy app) that a user installs has the capacity to stomp all over that user's files and any network shares he or she has mounted. With iCloud, that huge vulnerability is now possible to contain in a way that hasn't been possible before in mainstream desktop computing. Under iCloud, only the app that creates the data can change the data, and must do so securely.

Apple has also indicated that it will be relatively easy for third parties to add support for iCloud Documents & Data. In fact, it appears that Apple is leveraging the existing Versions and Auto Save mechanisms new in Mac OS X Lion to also enable iCloud to efficiently deliver updates to the cloud as you make them and subsequently push those incremental changes down to all your other devices to keep them all on the same page.

Within a week of iCloud's announcement at WWDC, prominent Apple blogger John Gruber revealed how iCloud would handle file sync conflict issues, noting that it would seamlessly resolve file conflicts that occur between devices while backing up each version of the file.

iCloud will "make a decision and it will decide which one is the best," Gruber stated. "iTunes will decide which one is right and that’s it. iCloud will push that right one to any device that has this account that has a different version."

And so, essentially, iCloud Documents & Data will work something like Time Machine in the Cloud, except that rather than being focused on traveling through the past looking for lost versions, the new feature will keep users' documents up to date in the present, with push updating of changes as they happen to all devices they own. Again, think of it as push messaging for version controlled documents.

On page 2 of 2: Strength in numbers, … & Data

Strength in numbers

Apple is working to make it appealing for developers to feature iCloud integration in their apps. The company similarly encouraged developers to add support for .Mac/MobileMe sync, something that many developers did add, despite the relatively limited number of subscribers that service had (believed to be around 3 million). In contrast, the entire installed base of Mac OS X Lion and all mobile users who upgrade to iOS 5 will have free access to iCloud, a number that will quickly boost iCloud users into the tens of millions.

Activating iCloud in iOS 5 is integrated right into the new operating system's self-contained setup system. Once configured in the iOS iCloud menu within the Settings app, turning on Documents & Data is simply a matter of sliding a switch. A sub option unique to Documents & Data allows users the option of transferring updates over cellular. Because Documents & Data delivers incremental changes (similar to an undo stack) of a document as it is edited, rather than pushing around entire document files, this option won't be a huge tax on cellular connections. Conversely, backups, over the air OS updates, and features like Photo Stream are confined to WiFi connections.

In Mac OS X Lion, iCloud is configured in the new Mail Contacts & Calendar Preference Pane unlike (and replacing) MobileMe's separate configuration pane. While existing builds of iOS 5 can connect to MobileMe and iCloud simultaneously, the latest build of Lion can reportedly only connect to iCloud if MobileMe is turned off. Additionally, the new iCloud interface incorporates Back to My Mac (formerly a tab within MobileMe) and the new Find my Mac feature within the same simple settings panel.

Third party apps that tap into the features of iCloud Document & Data will "just work" once users activate the service on their system. Again, this is somewhat similar to Time Machine, where simply turning it on creates a system wide benefit that specific apps can be optimized to take special advantage of with some work by their developer. There's nothing to configure within an app that has been enhanced to use iCloud, as all the configuration is done on the operating system level in both iOS and Mac OS X Lion.

Apple's beta iCloud web client, as has been profiled by some developers and media sources already, enables users to see and download copies of the files they are using from the browser, and currently supports translation of their Pages, Numbers and Keynote documents into PDF or Microsoft Office compatible formats. It's not clear if Apple will extend its iCloud web offerings to also add access to the files of third party apps with iCloud Documents & Data support.

… & Data

However, iCloud's Documents & Data will also be useful to more than just those app that work with documents. The "& Data" portion of the feature alludes to apps that make use of data, but don't actually generate files. An example of this might be an app that records events, such as a run tracker or an instrument monitor. These apps will similarly be able to push their raw data into iCloud and keep every other instance of the app current with the same information. This could conceivably work to allow an iOS app to remotely gather data that a desktop Mac app then presents "automagically" in real time without any manual syncing by the user.

As noted by API Evangelist, iCloud can store both apps' documents and "key-value data," which could include application state (allowing apps on various devices to sync their behaviors, such as resuming the same playback point in a video player app on another device), settings and "other important information that delivers a better user experience. Although not as apparent to the end-user, key-value storage will be just as important as document storage," the site notes.

"The iCloud service handles storage searches, change notifications, version control, conflicts, and security for applications that integrate with a user’s iCloud storage account," that site reports, concluding "neither document storage or key-value storage in the cloud are anything new. But when it is implemented as part of the IOS platform, it becomes much bigger. Apple is solving everyday problems users face when using their smart phones, by storing data centrally in the cloud."

Inside iCloud: Apple's new web services for iOS and Mac OS X Lion

Inside iCloud: Apple's new Documents & Data cloud service