How Samsung & Google teamed up to steal Apple Data Detectors for Android

Apple's critics contend that it either doesn't have innovative inventions worthy of patents, or has grossly overestimated the commercial value of its patents. Apple Data Detectors is one example that proves both ideas are wrong.

Along with Slide to Unlock, Apple Data Detectors is one of the five patents Apple selected to use in its second U.S. patent trial against Samsung.

Apple seems to now prefer referring to Apple Data Detectors as "Quick Links," perhaps because, like the company's Push Notification Server, the original, descriptive name of the technology boils down to an unfortunate acronym.

Regardless of its name, Apple Data Detectors not only represents a significant technology worthy of patent protection, but is also clearly valuable enough for Google and Samsung to have coveted for their own products, and strategic enough to motivate both parties to willingly infringe one of Apple's most well-known patents in order to appropriate it for use without paying Apple much, or anything at all.

We know Google and Samsung intentionally took Apple Data Detectors because both companies documented internal communications talking about what a valuable invention it was. Those documents were revealed this week in evidence submitted during the Apple vs. Samsung trial.

Zeitgeist at the origin of Apple Data Detectors

In the early 1990s, Apple's 'Golden Age of the Macintosh' was beginning to fade. Apple was still selling profitable desktop machines, had just introduced its revolutionary "PowerBook" (which changed how every other PC maker designed their own notebooks) and was also gearing up to launch its 1993 Newton MessagePad, the world's first consumer-oriented tablet.

However, in software Apple's unique Macintosh desktop environment was being flagrantly copied by some of the very companies that initially mocked the notion of graphical computing as being nothing more than a "toy." Apple successfully sued to stop a series of infringers, leveraging its copyright claims to various original elements of the Mac environment.

However, Apple wasn't able to stop Microsoft, which after winning a favorable court decision in 1992, cleaned out the entire Mac technology portfolio that Apple had developed over the previous decade at great expense, and began reselling Apple's work as its own under the name Windows 95.

At that point, Apple recognized that it either needed to license its technology to third parties in a model like Microsoft, or protect its original technologies via patents, a legal mechanism designed to provide government-backed ownership of original inventions the same way that a title deed provides government-backed protection of real estate ownership claims.

Apple experimented in both directions. It began working out agreements to host the Mac environment on top of various UNIX machines; began directly licensing Mac and Newton software to OEM manufacturers such as Pioneer, Bandai and Motorola; and began applying for patents on a variety of technologies it could detail to the U.S. Patent and Trademark Office as being original inventions.

Apple's PARC: the Advanced Technology Group

Apple Data Detectors, one of the most famous ideas to be patented by Apple in the early 1990s, resulted from work within the company's Advanced Technology Group, a think tank formed in 1986 by Larry Tesler "to allow Apple to study questions, technologies, and user issues that were beyond the time frame or organizational scope of any individual product group," as described by James R. Miller.Apple's ATG, like the Xerox PARC a decade earlier, was given free reign and liberal funding to work on futuristic ideas that didn't necessarily have obvious immediate product potential.

Apple's ATG, like the Xerox Palo Alto Research Center a decade earlier, was given free reign and liberal funding to work on futuristic ideas that didn't necessarily have obvious immediate product potential. Like PARC, Apple's ATG originated a variety of concepts that ended up making other companies rich or at least famous.

Examples include Apple's Metadata Content Framework, which began inside the ATG before escaping to become Netscape's Resource Description Framework for push content on the web. The resulting RDF Site Summary, or RSS, was later backronymed "Really Simple Syndication" as it became the mechanism for syndicating news feeds on the web and, in full circle back to Apple another decade later, podcasting.

Other examples of ATG's technological influence on Apple and the industry include components of Hypercard, which directly influenced the development of the World Wide Web (ATG was also an original sponsor of NCSA Mosaic, the basis of both Netscape and Microsoft's Internet Explorer web browsers); PlainTalk speech recognition and MacInTalk Pro speech synthesis; Newton's neural net handwriting recognizer; digital television standards; the Gotham 3D graphics accelerator; QuickTime video and QTVR panoramic stitching of photos.

Inside Apple Data Detectors

Apple Data Detectors originated within the ATG as a result of research into collaborative, intelligent agents aimed at using smart data analytics to make static information within a document readily actionable by users.

Development of the technology had "proven to be a painfully difficult task," the group's researchers noted. Charged with turning its raw technology concepts into a marketable feature, the ATG developed a series of Apple Data Detector prototypes designed to showcase the potential and value of its work.

The first was an open, extensible architecture that enabled a Mac to detect and recognize structures within a text selection, and then pass these items (such as phone numbers, event dates, physical locations or email addresses) to a second set of user-configurable actions that could offer to extract the data for use (such as adding it to a contact record or addressing a reply message).

The ATG's task went beyond just packaging a new concept in a sellable form. The new feature had to work efficiently in order to avoid slowing down the system to the point where any benefits were overwhelmed by the additional overhead required.

Microsoft's not-so Smart Tags

It also needed to present a streamlined, uncluttered interface to avoid complicating every "detected" document with tons of links, icons and menus to suggest every possible option available to the user in a confusing mess.

Microsoft introduced such a "Smart Tag" feature for Word in 2003, and had planned to add the feature system-wide in Windows. However, its implementation was so distracting and confusing that users' complaints resulted in the feature being canceled and removed from all of the company's subsequent products.

It was not obvious to Microsoft how to implement a mechanism for dynamically detecting and tagging content similar to what Apple had designed and patented more than five years earlier.

Microsoft, the world's largest software developer with a near monopoly on the world's operating systems, a status that allowed it access to vast resources and many of the world's most talented software architects and coders, couldn't deliver a worthy competitor to Apple Data Detectors more than five years after Apple outlined its system in a patent filing and then delivered a functional software implementation.

It was not a trivial invention.

The evolution of Apple Data Detectors: LiveDoc for Classic Mac OS

Apple commercially released Apple Data Detectors for Mac OS 8 in 1997. In its initial form (above), it allowed a user to select text in a document such as web page, then invoke a contextual "right click" menu to isolate the actionable data elements detected. From there, a user could navigate sub menus of options for each item, such as selecting to add a detected date and time to the user's calendar as a new event.

Apple Data Detectors next evolved into a new form with LiveDoc, which handled the detection phase in the background, highlighting the structures it found for users to interact with directly (below). This feature advance required support from applications, which needed to interact with the Mac platform's LiveDoc Manager. ATG developers recognized that this could be a potential problem.

In documenting their work, LiveDoc developers noted that they were "concerned about requiring developers to change their applications to gain access to LiveDoc's capabilities. We know from experience that developers are justifiably reluctant to change their applications just to implement a new feature provided by the toolbox, so we experimented with some alternatives that we hoped would ease this restriction."

The first effort was building LiveDoc functionality into the Mac's builtin TextEdit library that many apps were already using. This seemed to work for the popular third party email program Eudora, but by the time Apple finished the integration, Eudora had released a new version that no longer used TextEdit to handle its text handling.

Apple also invested efforts in building LiveDoc functionality into OpenDoc, an ambitious effort to replace monolithic Mac applications with interchangeable software components that could be assembled by users as needed in a new computing model oriented around users' own documents, rather than third party apps.

However, the developers noted that "we ultimately discovered that it still was not possible to get information about the way in which text was rendered without the LiveDoc Manager consulting the application." OpenDoc not only failed to support LiveDoc, but collapsed itself as concept.

"In retrospect," LiveDoc developers noted, "it seems that what we were trying to do was to graft an object-oriented software model onto a platform that was not object-oriented."

Essentially, the primary problem for LiveDoc was that it was an advanced object oriented system that Apple was trying to shoehorn into the Classic Mac OS. The best Apple could do was tack on the feature in a way that developers would need to go out of their way to implement, an uphill battle because Microsoft had been busy working to destroy Apple's Mac platform and take away its developers for its own Windows platform.

Drop Zone for the Classic Mac OS

Connected to the Classic Mac OS's lack of true object orientation, another problem brewing in the development of Apple Data Detectors was interface complexity. Highlighting too many data structures and providing too many potential options quickly resulted in an avalanche of clutter, as Microsoft had discovered first hand.

"Ideally, these menus of actions should be kept small but relevant" the ATG team observed, further noting that, "the problem is that 'relevance' is really determined by the meaning of the context in which the user is currently working.

"For example, the name 'Apple Computer, Inc.' could be associated with such actions as, 'Find the corporate headquarters on a map,' 'Get Apple's corporate phone number,' 'Get the current trading price of Apple stock,' 'Get the people in my address book associated with Apple' and so forth. All of these actions might be useful in one situation or another, but a user working on a financial task is far more likely to be concerned with the current price of Apple stock than the location of its corporate headquarters. Hence, the effective management of LiveDoc's action menus will come only with access to, and some understanding of, the contexts in which a user might be working."

One concept created to skirt the problems of third party app integration, complexity and the Classic Mac OS's lack of object orientation was Drop Zones, which presented a series of Activities users could drag detected items to. The system would then launch an Assistant to offer appropriate actions.

Apple Data Detectors attacked as "evil," disappear for nearly a decade

As with Slide to Unlock, the novelty of Apple Data Detectors was first attacked for being "too new" before being copied by Apple's competitors who insisted it was nothing new at all.

Reminiscent of how Rob Enderle frantically tried to vilify the iPhone's multitouch unlock feature as being something that would contribute to rape by making emergency calls too difficult to make, Jaron Lanier warned the world a decade prior that intelligent agents were "both wrong and evil" and would lead to alienation and the undermining of human values.

Then, just as the ATG's Apple Data Detectors work began to develop into a usable form, severe financial and operational problems at Apple resulted in the downsizing or elimination of all projects that were not directly contributing to the company's viability. Steve Jobs personally took responsibility for dismantling the ATG in late 1997.

As Tim Bajarin observed at the time, "the group provided some important technology, but they also spawned the 'Not Invented Here' syndrome. That had actually really hurt Apple over the last few years, with money put into proprietary research and development when they could have bought technology from outside."

When Apple's new Mac OS X first appeared in 1999, there was no trace of Apple Data Detectors, in large part because the existing implementation of the technology for the Classic Mac OS had been built in a way that could not be easily integrated into the new operating system.

Data Detectors return to OS X, iOS

Apple didn't throw away the work its ATG had developed however. After recovering financially, Apple began aggressively advancing the object oriented development frameworks it had acquired from Jobs' NeXT, creating a foundation that could finally support advanced technologies like Apple Data Detectors in a flexible and powerful way.

In 2007, eight years after the USPTO granted Apple's U.S. patent 5,946,647 and a decade after it first shipped the feature for the Classic Mac OS, Apple released new support for its Data Detectors technology in OS X Leopard.

Apple began by cautiously adding support for Data Detectors into some of its own apps with the release of OS X 10.5 Leopard, primarily Mail. A variety of other apps also included the feature, but it was turned off by default. Apple also incorporated the "lost" feature in its iPhone, where the convenience of selecting data that could perform a task with a simple touch was even more obviously valuable.

By 2009, iOS 3.0 and OS X 10.6 Snow Leopard had expanded the role of Data Detectors to other apps, enabling an immediate, touchable link for any detected data capable of invoking an obvious action. Apple got it work everywhere "automagically" by integrating the technology into Core Text, the object oriented text system that the Classic Mac OS lacked a decade previously.

In OS X, however, Apple continued to bury the feature in a contextual menu that few users seemed to be aware of. That noticeably changed in OS X Mavericks, when Apple began incorporating sophisticated, intelligent Data Detector actions for selected content, including integration with its new Maps app for the Mac.

Google, Samsung decide they want Apple Data Detectors

Just months after Apple launched its iPhone in 2007, an Android manager at Google referenced the new device's support for Apple Data Detectors, noting in an email presented this week during the Samsung trial that, of the new iPhone, "one of the most powerful features is the interaction of text objects," according to a trial report by Rick Merritt for the EE Times.

Apple also introduced testimony by Todd C. Mowry, a Carnegie Mellon professor, who stated, "I think it was an invention way ahead of its time [because] it is extremely inconvenient with a cellphone to copy and paste text."

Apple developer Thomas Deniau further testified that iOS Data Detectors software includes "tens of thousands of lines of code," adding that "it's very complicated to accurately identify so many data structures in so many different languages. We support about 20 languages."

Samsung didn't just incorporate a Google copy of Apple Data Detectors within Android for use in its own products; it independently identified that it wanted and needed the feature in order to be able to compete with Apple's products.

In an internal Copy Cat document created for the first Galaxy S model in March 2010, Samsung's Software Verification Group noted in its "Relative Evaluation Report on S1, iPhone" that Apple's iPhone, "with a single tap on the URL, phone number or email in the memo, immediately moves to the corresponding web page, outgoing call window or email compositing window" and that a "long press displays a menu with detailed options like copy, save, etc."

On its own Galaxy S 1, the group observed, "link for URL, phone number or email address is not provided." Samsung concluded as a "direction for improvement" that its products "need to improve usability by providing Links for memo contents such as Web, Call and Email, that can be linked."

Apple Data Detectors was 25 percent of Samsung's product description for its new project

That August, Samsung's "UX Roadmap" for 2011-2012 described a "smart organizer" project proposal that outlined four primary features. One of these requisite, defining features was "function to add schedule by automatic recognition of contents of document (by system)."

Samsung's document even provided a graphical depiction of what it aimed to achieve, under the the bold text: "automatically applies to Calendar schedule by detecting schedule in the email contents according to user's intent."

That graphic happened to be a cropped image of the "Apple Data Detectors in action" screen shot as it appeared in online documentation describing the history of the ATG's work in developing Apple's patented technology.

The novelty of Apple Data Detectors

The technology Apple paid to develop in the 1980s and 1990s, was patented in 1999 and implemented as a desirable, differentiating commercial feature across Apple's product portfolio in the 2000s.

Yet in 2012, developers on StackOverflow were still posing the question "How does Apple find dates, times and addresses in emails?"

"It not only works for emails in English, but in other languages also. I love this feature and would like to understand how they do it," one developer asked. "The naive way to do this would be to have many regular expressions and run them all. However I this is not going to scale very well and will work for only a specific language or date format, etc. [...] Any idea how Apple is able to extract entities so quickly in its email client?"A quarter of Samsung's vision for "the next big thing" was quite literally a digital photo copy of Apple's work

Several who replied noted that it was Apple's patented technology. So it wasn't exactly a secret that the feature Samsung thought it needed came from Apple and was patented by Apple, or that a quarter of Samsung's vision for "the next big thing" was quite literally a digital photo copy of Apple's work.

It would be hard to imagine a more clear cut case of intentional theft of another company's patented technology, other than the Slide to Unlock patent Apple is also suing Samsung over.

Apple argues for injunctions on infringement

At the end of 2011, Apple successfully leveraged its Data Detectors patent to win a U.S. International Trade Commission injunction against infringing products from HTC. That win helped Apple to force an end to its patent lawsuits with HTC, resulting in a settlement where HTC paid Apple patent licensing royalties and agreed to a "no clones" clause.

However, judges in cases involving both Google's Motorola and Samsung have blocked Apple from winning sales injunctions against those companies' infringing products. In 2012, Judge Richard Posner worried that a ban on Motorola's infringing sales could have "catastrophic effects" and would be "contrary to the public interest."

Apple hadn't attempted to block all Motorola sales, but did ask Judge Posner to rule that Google must remove Apple's patented technology from Motorola's phones. A report by Reuters noted, "But Posner said it may be preferable to direct Motorola to pay Apple a compulsory royalty. Forcing Motorola to adopt inferior technology, as opposed to paying a royalty, would not benefit consumers, he said." Apple doesn't want to be forced to license its patented technologies to Google for use in Android

Apple doesn't want to be forced to license its patented technologies to Google for use in Android. An injunction on infringement, Apple's attorney argued at that trial, "means we're not competing with them where they are using our technology against us."

After Apple won a damages award for Samsung's unlawful infringement in late 2012, Judge Lucy Koh similarly ruled that she would not grant Apple an injunction against Samsung's infringing products, stating that Apple "cannot obtain a permanent injunction merely because Samsung's lawful competition impacts Apple in a way that monetary damages cannot remedy."

Samsung seeks to set low price for Apple's patents after U.S. Government forces licensing

Apple is likely concerned that after being forced to license its inventions to its Android competitors, it has little leverage in demanding fair royalties for them. Samsung has now adopted an argument that Apple's patents are essentially worthless, just like the two patents it acquired to defend itself in this second trial.

Industry observers like FOSS Patents blogger Florian Mueller have also taken to complaining that Apple's demands for commercially significant royalties from companies infringing its patents are unreasonable.

The standoff between Apple and the remaining Android licensees who refuse to stop using the company's technology ensures that legal cases will continue to drag on while the infringing companies continue to bring in revenues from sales of their infringing products.