At WWDC, Apple unveiled a new App Extension architecture for both iOS 8 and OS X Yosemite. Here's how the new extensions enable a new type of widgets in Notification Center for both platforms, along with enabling third party Custom Keyboards for iOS.
Apple introduced a wide variety of new avenues for third party development under iOS 8 and OS X Yosemite that are all tied together by the common thread of App Extensions. The previous article examined Photo Editing Extensions for Apple's new Cloud Kit-savvy Photos app.
This article examines how the same underlying technology is being put to work to create "Today Extensions," which are widgets that will appear in Notification Center, as well as Custom Keyboards for use on mobile devices.
Is Apple just really slow at stealing Android features?
Home screen widgets and third party keyboards (which replace the default system keyboard) have been associated with Android for the last five years as something iOS has lacked.
Given that Apple created the concept of Desk Accessories as mini apps for the Mac desktop back in 1981, and that OS X Tiger promoted more recent Dashboard Widgets back in 2005 (before Google even acquired Android), it might make you wonder why Apple is only now introducing standalone widgets for iOS.
Custom Keyboards pose a similar quandary: iPhone presented a virtual onscreen keyboard when it debuted in 2007 as one of its primary differentiating features. Additionally, among the first iOS App Store titles in 2008 was ShapeWriter, a non-traditional keyboard app that input text via gestures using technology developed at IBM and Linköping University, similar to Swype (both of which have since been acquired by Nuance, which has already announced support for Swype as a Custom Keyboard Extension for iOS 8).
While the original ShapeWriter could record text, the user had to copy and paste what they'd entered into the app they wanted to use, making the technology far less useful than custom keyboards integrated right into apps (like the custom numeric keyboard presented in Apple's Numbers, below) or the system itself (enabling input into any installed app).
Android didn't even debut until nearly a year after ShapeWriter, in part because Google's original concept for Android was a Java Mobile button phone, just like everyone else had been selling.
Android eventually showed up without a working virtual keyboard because Google focused on user input via physical mini keyboards and an LED lit trackball, failing to grasp the value of reconfigurable keyboards that Steve Jobs had touted as a core design concept for the original iPhone in stark contrast to the rest of the industry (below).
In 2009, Google threw open the door to widgets and third party keyboards in Android 1.5 Cupcake, nearly a year before the platform began catching on in the market. That resulted in virtually all Android users being familiar with the concept of both ideas, despite Android having shown up two years late to the iPhone party in terms of virtual keyboards.
Why Apple didn't focus on widgets and third party keyboards
Given that Apple had such an early lead, why has the company resisted opening up a third party market for system-integrated custom keyboards and Home screen widgets over the past five years? In a nutshell: performance, privacy and security.
Apple's focus on performance, privacy and security in iOS have trumped Android's widgets and custom keyboard features in the market for premium phones, keeping Apple sustainably profitable as other hardware makers using Android have gone out of business or regularly lost money (including Google's own Motorola, which lost hundreds of millions of dollars in quarter after quarter).Even if Tim Cook has a time machine, he has no need to travel back to 2009 to urge iOS engineers to focus on widgets and third party keyboards
Consider that Android's best performer, Samsung, has seen its stock appreciate over the last five years by 119.3 percent while Google's stock is up 175.24 percent over the same period.
Over the same five years, Apple stock is up over 362 percent, despite the media's delusional panic throughout 2013. Even if Tim Cook has a time machine, he has no need to travel back to 2009 to urge iOS engineers to focus on widgets and third party keyboards.
That's not to say that there's no value in widgets or custom keyboards. Android's promotion of both has highlighted a variety of interesting abilities. And of course, Apple's iOS has also incorporated system-supplied widgets (like Stocks and Weather) and app-specific custom keyboards (again, as in Apple's own Numbers). What Apple hasn't done is fling iOS wide open to unrestricted hobbyist experimentation without considering the consequences.
Google's decision to do just that with Android has manifested the problems associated with rushing poorly conceived, half baked solutions to market. Widgets on Android are commonly associated with battery drain and can help contribute to the noticeable performance lag that still affects the system.
Additionally, Google's implementation of both widgets and third party keyboards expose significant security vulnerabilities and privacy issues that many end users wouldn't reasonably assume possible. It's virtually impossible for Google to police whether a third party keyboard spies on users. And it is well known that many actually do, recording users' key presses and uploading the data via the network because there's no functional security boundaries in place to stop such malicious activity from occurring.
Security on Android is largely based upon naive trust. Even apparently legitimate custom keyboard developers have been found to send users' keystrokes back to their servers for processing in clear text making everything they type open to anyone listening.
As an experienced platform vendor that has dealt with malicious security attacks decades before Google was even incorporated, Apple exercises a lot more concern about leaving its platform open to exploit. It also doesn't delegate its basic security design to end users, most of whom have no expertise in navigating the complex issues involved.
In the design of Android, Google has repeatedly bet that nothing bad will ever happen, and has regularly lost that bet at the expense of its users. But the reputation of Android has also become tarnished, particularly affecting valuable clients in government and in the corporate enterprise, where Android remains a minority overall and offers Apple's iPad virtually no competition in tablets at all.
Widgets moving from OS X Dashboard to Notification Center
In both iOS 8 and OS X Yosemite, Apple will present widgets within Notification Center, owing to the relationship between push notifications and widgets' primary role in displaying regularly updated information. For this reason, Apple commonly refers to the new widgets as "Today Extensions."
On OS X, Today Extensions will effectively replace the Dashboard widgets introduced nearly a decade ago in OS X Tiger (below). Dashboard widgets were conceived as desk accessories that could be quickly invoked and dismissed, and were implemented in HTML, CSS and JavaScript to allow for a similar level of security sandboxing as standard web pages. Web standards have been designed not to implicitly trust foreign hosts.
However, rather than being, effectively, mini web pages like Dashboard widgets, Today Extensions are native code that--like other App Extensions--use a new security model based on Apple's XPC and modern iOS-style Cocoa sandboxing. They are launched by the system and dynamically managed; if they use too much memory on iOS, the system can terminate them to maintain optimum performance.
Building the infrastructure to support such as secure, native implementation has taken years of engineering behind the scenes. XPC, which Apple describes as an "interprocess communication technology that complements App Sandbox by enabling privilege separation," was first introduced in OS X Lion in 2011. Privilege separation is key to allowing App Extensions to perform limited tasks without opening up dangerous new vulnerabilities.
In addition to native performance and a mature security infrastructure, App Extensions also need a viable market supporting them. Dashboard widgets have long languished on OS X after their initial novelty wore off, in part because of the delay involved with launching the web rendering process that animates them when users invoke Dashboard, and in part because free Dashboard widgets lacked a functional business model similar to App Store apps.
Today Extensions (as with all forms of App Extensions) are packaged with new or existing apps, making it possible for developers to market the new widgets as a valuable feature of their paid (or ad supported) apps.
Today Extensions on iOS 8 & OS X Yosemite
Today Extensions will work slightly differently on iOS and OS X; on mobile devices, the widgets appear as touchable presentations that lack text input or editing features (meaning that a stocks widget, for example, would need a companion app to edit the list of stocks it presents). On OS X, widgets can be edited in place and can support text input.
These differences are largely due to the fact that on iOS, Notification Center is a temporary pull down designed to present information at a glance. Add a keyboard and editing controls, and the value of presenting widgets would be crowded out by all of the editing UI. Mobile devices also have less processing power and memory available to devote to widgets than a Mac would.Apple's vision for Today Extensions is clearly aimed at presenting quick access to scores, stocks and similar information in the context of other notifications
On iOS, any sort of widget that demands more UI and resources than Notification Center can reasonably accommodate would make more sense to implement as a standalone app--just as Apple implemented its own Stocks and Weather "widget apps" seven years ago on the original iPhone.
In contrast to Android, Apple's vision for Today Extensions is clearly aimed at presenting quick access to scores, stocks and similar information in the context of other notifications, rather than simply padding the Home screen with a busy box of fluff to make up for a lack of significant, native apps.
Apple's Notification Center & Android widgets
Like widgets themselves, Notification Center is another feature Android users like to complain that Apple "stole" from an open source project built upon, ironically, the foundation of "copy left" ideology where there is no such thing as stealing.
However--just as with the overall concept of both widgets and a multitouch mobile device designed to launch discrete apps from a home screen of icons--Apple publicly outlined its efforts to develop a central notification system for iOS before Android was even released.
In early 2008 at the release of the App Store and "iPhone OS 2.0," Apple's then head of iOS development Scott Forstall announced the company would be setting up a centralized Push Notification Service as a mechanism for allowing apps to respond to updates from outside services without needing to remain active in the background, constantly polling for new data and eating up battery power, CPU and memory.
Apple's overview of its push notification service.
At the time, Samsung was still promoting Windows Mobile 6, which (like Android today) required users to manually manage background processes and the memory and CPU resources their apps were consuming, an implementation Apple pointed to as an example of a poor design it wanted to dramatically improve upon.
The Task Manager application in Windows Mobile 6.
However, Apple noted later that year that it had greatly underestimated the overwhelming demand for both apps and push notifications, sending the company back to the drawing board and delaying the rollout of its Push Notifications Service until iOS 3.0 in early 2009, following a stress-testing beta program involving the Associated Press and other app developers.
In late 2009, Google, a major iOS developer, filed a patent for "notification of mobile device events," describing a feature it would later add to Android. Android enthusiasts now claim Apple simply copied its Notification Center from Android rather than having laid all the groundwork for touchscreen smartphones, a functional app store and a secure, battery-efficient notifications system.
In 2010, Apple brought its push notifications to the Mac as an API, initially to support FaceTime notifications and then more broadly as a public API in 2011's OS X Lion.
Today's basic design of Notification Center (above) appeared on the Mac as a user-facing feature in 2012's OS X Mountain Lion, after first making an appearance on Apple's mobile devices in iOS 5 the previous year.
Google patents the type of patent that Google doesn't respect today
Rather than Apple's newest implementation of widgets being "slavishly copied" from Android, the reality is that Google was fully aware of Apple's public work on notifications and--similar to Samsung--sought to patent a basic notification listing concept (below) it witnessed others developing first, including prior art developed at Palm for webOS by former Apple engineers.
Among those engineers is Rich Dellinger, who worked at Apple from 1999 to 2006 before leaving for Palm, where he developed the "non-intrusive banner notification system used in webOS" (below) that debuted at the beginning of 2009.
Dellinger returned to Apple after Palm was acquired by HP in 2010. Apple also hired Peter Hajas, who had developed MobileNotifier, an app for jailbroken iOS 4 devices that presented a pull down list of notifications. Apple subsequently released its official Notification Center in iOS 5 (above).
In parallel, Google hired Palm's webOS designer Matias Duarte, who is credited with developing the UI for Android 3.0 Honeycomb, as well as the company's latest web-inspired 'Material Design' appearance for both Android L and future versions of Chrome OS. That work draws clear inspiration from Apple's richly animated UI for iPhone released seven years ago, albeit with a unique color palette.
Custom Keyboards as a system plugin
More important than the identity of the first implementation rushed to market is the superior implementation currently available. That's particularly evident among the second type of App Extension that Android users are claiming Apple "stole" from Google: third party, system wide Custom Keyboards.
After belatedly recognizing the value of onscreen keyboards, Google opened up third party access to a core Android system resource back in 2009 to allow anyone to offer custom software for text input. This has been particularly valuable to Android users because Google's default system keyboard was widely considered to not be very good. Google's first attempt at a virtual keyboard was delivered alongside third party options.
As one user noted in a question posed to Android Central years afterward, "my biggest gripe about Android as an iOS user, was that I thought all of the keyboards but the Jelly Bean one were garbage. I found text prediction to be sorely lagging behind iOS, and having to pause to select your word I thought was a bad idea. I'm a fairly rapid typer, and on the iPhone made relatively no mistakes, and never had to backspace at all."
Apple's default keyboards--including support for Chinese finger stroke input--in addition to iOS input features including spell correction and auto correction, have long held a major advantage over the basic keyboard included with Android. So much so that Samsung was found to have infringed Apple's 8,074,172 patent on keyboard auto correction in an effort to make its devices more competitive with iPhones.
At the same time, there are third party custom keyboards that many Android users find compelling, and those input systems were previously impossible to port to iOS because of Apple's abundance of caution to protect users' privacy and security.
Apple's strict rules for new Custom Keyboards
With iOS 8's new Custom Keyboard Extensions, developers can implement alternative systems, with a few restrictions. First of all, Custom Keyboards can't be used to type in secure text field objects, such as the users' passcode or other password fields. When a user attempts to enter secure text using a Custom Keyboard (like the new Swype, below), iOS 8 temporarily reverts to the system keyboard, then returns the user back to their preferred keyboard afterward.
Custom Keyboards are also prevented from entering "phone pad" input, such as the phone number field in Contacts. Apple also notes that Custom Keyboards "cannot select text or control cursor position" and "cannot offer inline autocorrection controls near the insertion point" due to the way they are implemented."Your first consideration when creating a custom keyboard must be how you will establish and maintain user trust. This trust hinges on your understanding of privacy best practices and knowing how to implement them." - Apple
Apple also outlines that "by default, a keyboard has no network access and cannot share a container with its containing app."
The company's documentation devotes an entire section to "Designing for User Trust," noting to developers, "your first consideration when creating a custom keyboard must be how you will establish and maintain user trust. This trust hinges on your understanding of privacy best practices and knowing how to implement them."
The company spells out three principles of design to earn users trust. Under "Safety of keystroke data," it states "users want their keystrokes to go to the document or text field they're typing into, and not to be archived on a server or used for purposes that are not obvious to them."
Under "Appropriate and minimized use of other user data" it adds,"If your keyboard employs other user data, such as from Location Services or the Address Book database, the burden is on you to explain and demonstrate the benefit to your users."
Finally, under "Accuracy" it notes, "accuracy in converting input events to text is not a privacy issue per se but it impacts trust: With every word typed, users see the accuracy of your code. To design for trust, first consider whether to enable network access. Although network access makes many things possible for a custom keyboard, it also increases your responsibilities."
To turn on network access, a Custom Keyboard must request permission from the user. There are a variety of reasons why a user might opt to allow this, including features such as saving a custom autocorrect dictionary; providing enhanced touch-event processing, input prediction, and speech recognition for dictation; providing quick access to "names, places, and phone numbers relevant to the user" from the user's Contacts or from network supplied data; or autocorrection of nearby place names from Location Services.
Apple outlines specific requirements for developers to implement in their Custom Keyboards when network access is granted by the user, including the instruction, "do not store received keystroke or voice data beyond the time needed to provide text back to the user or to provide features that you explain to the user."
Users who aren't interested in Custom Keyboards will still get the benefit of iOS 8's intelligent QuickType keyboard, which offers predictive text customized to both the app users are typing in and the person they are addressing, in addition to isolating common answers from the context of the conversation (below).
Apple's considered design for App Extensions
As with Photo Editing Extensions, Custom Keyboards and Today Extensions provide third party developers with new ways to extend and add value to users via Apple's platform, which strictly regulates how App Extensions can work.
Apple has also designed App Extensions to benefit developers by highlighting the value of their work. As the company notes, "when users enter your extension, your custom UI can help to show them that they're shifting into a new context. Because users can distinguish your extension from the current app, they can appreciate the unique functionality that you provide."
At the same time, Apple has also given consideration to how users manage their own environment. "Users' awareness of extensions as separate entities also means that they can identify and remove extensions that misbehave or don't perform well," Apple states. The company has also outlined how users will be able to easily specify which Extensions they want to activate or remove.
The inclusion of both Today Extensions and Custom Keyboard functionality into the design of App Extensions indicates that Apple is working to erase any perceived advantages held by alternative operating systems. However, the design of iOS and OS X aren't simply reacting to follow market trends. Apple is also establishing its own unique functionality with App Extensions in ways that are not shared with the rest of the industry, as the following segment will outline.