Apple's iCloud disparaged over Core Data sync problems

Frustrated with the limited options Apple provides for handling their issues, several app developers have taken their complaints public via blog entries, primarily focused on problems related to Core Data database sync with iCloud.

Their complaints have been picked up by journalists who often conflate these specific, limited database sync issues with Apple's entire spectrum of online services, resulting in headlines that pronounce the entire iCloud umbrella as being "broken," with no hope in sight, while castigating Apple for advertising iCloud as "automatic and effortless."

Apple's tightlipped secrecy makes it harder to reliably forecast how soon the company will precipitate fixes for specific issues, complicating developers' plans even as their own customers rain down complaints and poor reviews for either a lack of support for iCloud, or alternatively, syncing issues within an app after the developer adds support for iCloud.

"I would love for Apple to tell us they fixed everything with iCloud in iOS 7 and we can finally use it without issue two years later, but Im not getting my hopes up," one unnamed iOS developer reportedly told the Verge for an article headlined "Why doesn't iCloud 'just work'?"

The wide span of Apple's iCloud services

Further complicating things is the reality that "iCloud" is an extremely broad marketing term Apple nebulously uses to cover nearly all of its online services.

In general terms, "the cloud" refers to the Internet, often specifically to network based services, but more recently to simple "cloud storage" of files as well. Anything happening "in iCloud" is simply being done on Apple's servers, which includes Mail, Contacts and Calendar services. Developers aren't taking issue with any of these services, which despite some sporadic interruptions, have not generated unusual volumes of complaint from users.

Apple's various digital download and shopping services, including its industry leading media content in iTunes, iBooks and iOS and Mac App Store software are also roped together under the "iCloud" umbrella. This isn't the iCloud that developers are complaining about either.

The company also offers a unique app-based file storage service that saves documents "in the Cloud" for access from any one of a user's various Macs or iOS devices, and in some cases, from the web. Photo Stream, Find My iPhone, Back to My Mac, and various other data sync services (such as Safari bookmarks and Reading List) are also features associated with iCloud.

None of these features are directly related to the iCloud problems that developers are reporting, resulting in confusion for many of the 250 million users who don't see Apple's useful and generally reliable (but certainly not flawless) iCloud as being "broken."

The dark, silver lining of Apple's iCloud

The primary iCloud problems for third party app developers relate to issues with reliably syncing databases between their own apps and the cloud (and keeping them in sync across devices). This is handled by Core Data, an Apple framework designed to simplify the management and storage of data within an app for developers, allowing them to focus on the unique aspects of their software titles. Core Data itself is highly regarded among most developers.

Apple introduced Core Data back in 2005's Mac OS X Tiger, and made it available to App Store developers in iOS 3.0. Core Data enables developers to work with a user's data as objects, handing much of the complexity of data storage and retrieval for them. The Core Data framework provides options of how to store that data suited for different tasks: either as a simple XML text file or in binary data (both examples of a standard file), or within a SQLite database (a specialized file that contains organized data that is accessed and modified via specialized transactions).

When Apple introduced iCloud, it integrated a new "Documents & Data" feature of the service with the Core Data framework, to enable apps to leverage the familiar framework to save and retrieve an app's data via iCloud. For document-based data files or simple "key value pairs" (such as preference settings), iCloud seems to work pretty well.

"The core idea behind iCloud," Apple's developer documentation explains, "is to eliminate explicit synchronization between devices. A user never needs to think about syncing and your app never interacts directly with iCloud servers. When you adopt iCloud storage APIs as described in this document, changes appear automatically on all the devices attached to an iCloud account. Your users get safe, consistent, and transparent access to their personal content everywhere."

Actually implementing iCloud support in an app isn't trivial; Apple itself spent about a year adding iCloud support to various apps built into Mac OS X Mountain Lion, and it took some additional time to get iCloud-based files working between the iOS and Mac versions of its iWork apps (depicted below).

The more ambitious effort of syncing incremental changes made to an SQLite database with iCloud is even more complex. Several developers have reported considerable frustration in getting this to work at all, and their efforts are complicated by the unforeseen actions of users, who may, for example, log out of iCloud on their device and log in as a new user, throwing additional complexity into the situation.

While Apple has implemented document-based iCloud sync in its own apps, neither it nor its FileMaker subsidiary appear to be making any significant use of SQLite databases synced with iCloud. And if Apple isn't yet using it, you can bet that it isn't polished for third party developers either.

A catch-22 for innovation

The Mac and iOS platforms depend upon the architecture of Core Data (and its integration with iCloud) to present a number of innovative features. Everything from AutoSave to Versions (below) to Time Machine leverages the novel concept of incrementally saving users' work so that it can be rolled back, archived, and even synced between devices in real time. Apple has been rolling out and progressively enhancing these features (and their underlying frameworks) over the past several years.

During that time, Apple's critics have both complained about how unimpressed they are with every new bit of hardware Apple and software update the company has released, while also complaining about any changes and growing pains that have occured as a result of Apple's often aggressive plans to roll out new features.

The alternative solutions to Apple's iCloud dilemma (and more broadly, everything about OS X and iOS) would be to either greatly scale back innovation on all fronts (allowing competitors to catch up and critics to amplify their complaints that Apple 'isn't innovating enough'); to put "beta" warnings on new software features the way Google does to avoid harsh criticism of new software and features (as Apple did with Siri, although this didn't have any real effect); or to assemble larger teams to work on various projects (complicating development and increasing the staffing and operational issues the company already struggles under given its crowded campus and the difficulty of hiring qualified, experienced developers).

Real solutions don't come easy

Perfecting the only cloud synced, relational database architecture offered on any mobile development platform is a big task that takes time. Struggling to make third party apps work with such a system in its unfinished, or at least unpolished, current state is also difficult. Complaining about things is much easier, and offering simplistic solutions (Hire more developers! Make it "just work!") is almost as effortless, so far more progress has been made in those regards.

One further complication is that Apple is pushing developers to both adopt Core Data and support iCloud in their apps. For some apps, this runs the developer smack into the bleeding edge of what's possible, a situation that further results in usability complaints from end users.

Solving the issue is complex, the very reason Apple began working on Core Data in the first place; it's simply difficult to manage the sync of database transactions across multiple devices, particularly when third party applications can encounter very specialized use cases or need to accommodate various unusual circumstances of their customers. That complexity is why app developers want Apple to fix iCloud's database sync problems rather than having to roll their own custom solutions at great effort and expense.

Once app developers or their users encounter a syncing issue, it can also be difficult and expensive to resolve. On top of that, Apple doesn't currently provide any robust tools for salvaging a corrupt database or improperly synced data within iCloud. In most cases, the only option for users may be to remove existing data from iCloud and start over.

One solution: do it yourself

Developer Steve Streza wrote, "the reality of syncing data is that its tough, and network availability is not always reliable or fast (especially on mobile). You have to write a lot of nonobvious code to handle updates and problems."

Speaking of Apple's iCloud team, he added, "what theyre doing is immensely difficult, and I have great respect for that team for trying to solve a problem of this magnitude in a way amenable to every app. But sympathy is not something you can build a business on. Of course, the downside to the DIY solution is that its really, really hard."

Streza went on to discuss the advantages and disadvantages of building one's own sync infrastructure or using Apple's iCloud in greater detail. "iCloud," he concluded "while a beautiful idea, is not production-ready for most apps, will cause endless headaches, and will lock you in to forever developing apps only for Apple products."

Any sufficiently advanced technology is indistinguishable from magic

While some developers have expressed skepticism of Apple's entire approach to Core Data database syncing in iCloud, the company has a lot invested in the technology. It's also spending billions to develop massive new data centers to host iCloud, making it less likely that the company will simply abandon the project as it did with iWeb or other earlier MobileMe features, or as Google recently has with its RSS Reader, support for CalDAV calendar sync and ActiveSync push Gmail.

Additionally, there's indications that Apple's recent management realignments may help focus attention on solving iCloud's root problems with Core Data. The Core Data framework was salvaged from the work Steve Jobs' NeXT did on Enterprise Objects Framework in the early 90s. The original EOF enabled NeXT and WebObjects apps to integrate with large scale databases. Core Data was a repurposing of this technology to solve problems encountered by desktop developers.

The original EOF project was managed at NeXT by Craig Federighi, who joined Apple when it acquired NeXT in 1996 and left after the company decided to port EOF to Java in 2001, effectively destroying it.

In 2009, after Apple had completed the work of transmuting EOF into Core Data, the then head of OS X at Apple, Bertrand Serlet, recruited Federighi back to work on OS X. Two years later, in August of last year, Serlet announced his retirement and introduced Federighi as his replacement to lead Apple's OS X efforts.

Serlet said at the time, "Craig has done a great job managing the Mac OS team for the past two years. Lion is a great release and the transition should be seamless."

Three months later, Apple's chief executive Tim Cook announced another shakeup which relieved Scott Forstall of his leadership of iOS, putting both OS X and iOS under the direction of Federighi.

For developers hoping to see progress made on Core Data, the idea that the original developer of EOF is now managing iOS is at least good news. It remains to be seen what Apple's OS teams have accomplished over the past year under the direction of Federighi, but at least there's only a few months left before the company details at WWDC what it plans to do with iOS 7, OS X 10.9, and iCloud v3.