One Apple GPU, one giant leap in graphics for iPhone 8

Apple is building its own GPU architecture, but why? Rather than being motivated by simple cost savings, evidence points to the timing of a significant technical leap forward that could be as big of an advance as iOS was ten years ago.

Not for cost savings

Apple has been reportedly working on a GPU project for at least four years. By early 2013 it was known to have hired a dozen former AMD graphics engineers working near Orlando, while posting job openings involving topics such as "modeling GPU hardware."

Why would the company launch an internal GPU development team when it already owned a stake in Imagination Technologies, the British reference platform designer that Apple had been using for its mobile GPUs since the launch of the original iPhone? Why not just buy Imagination outright, or continue to license its technology?

The answer is probably not centered on cost savings. Apple pays relatively little to license Imagination's PowerVR GPU designs. In fiscal 2016, Apple paid Imagination less than $100 million for technology used across sales of more than 250 million devices that drove over $157 billion in revenues. Developing its own GPU designs would certainly cost Apple more than $100 million of R&D per year.

The Metal GPU

To understand why Apple would consider migrating away from PowerVR GPU designs, consider a parallel graphics shift Apple introduced in 2014: its proprietary Metal API (application programming interface) for driving graphics tasks to the GPU.

Apple was already deeply invested in the general-purpose OpenGL graphics API (specifically OpenGL ES on the mobile side). The reason for introducing Metal wasn't that it saved the company money, but rather that it enabled it to tightly customize iOS (and later macOS and tvOS) development to the subset of graphics capabilities that Apple actually shipped, throwing away tons of irrelevant overhead to radically optimize how efficiently Metal code could use available GPUs.

OpenGL was designed to flexibly work across OS platforms and across GPU architectures. That flexibility grew into an issue for Apple because it limited its ability to optimize performance. Microsoft, Nvidia, AMD and other companies had already introduced alternatives to OpenGL in order to focus on their own strengths. Metal did the same for Apple's platforms.

Similar to OpenGL, Imagination's stewardship of PowerVR GPUs is aimed at solving the needs of a wide variety of licensees working on various product categories. In fact, the British chip designer has been working to lessen its dependance upon Apple for years.

Its latest major, next-generation Furian mobile GPU designs were expected to debut in the next two years. The company's description of Furian notes that it is "designed to scale efficiently to the higher performance points required for applications such as AR and VR," and touts its "leadership in its existing markets such as mobile, tablets and automotive."

Given Apple's tight secrecy on its plans for future products, it doesn't require any deep understanding of Furian to realize that Imagination's goals were not laser-focused on solving the future issues Apple faced, partly because Imagination does not even know what those goals are in any detail, and partly because Imagination has been publicly seeking to serve the needs of a broader set of customers outside of Apple.

Across the last decade, PowerVR has served Apple comparably to OpenGL. However, the general purpose design of PowerVR— and the independent direction that Imagination is heading in for its future designs— both indicate that Apple is designing its own GPU in part to ditch the extraneous GPU goals Imagination is working on with its other clients in mind. That's the same intent Apple had in introducing Metal.

The San Francisco GPU

Another example of Apple replacing an apparently functional, status quo technology with its own highly-optimized alternative approach is the San Francisco fonts it began introducing in 2014. Apple developed the new font specifically for use on Apple Watch, to enhance legibility of text on its small display.

The next year, Apple introduced a variant of San Francisco for iOS 9 and macOS El Capitan. It has since extended use of its new font to its website, advertising and its corporate branding right down to the characters printed on its keyboards.

Apple had previously been using a variety of fonts (mostly related to Helvetica) for different purposes. Converging on one custom-built font family (at least for Latin, Greek and Cyrillic alphabets) served a series of technical purposes while also creating a consistent appearance and character.

One feature of San Francisco is its built-in support for Dynamic Type, which scales text sizes while adjusting its weight, leading and tracking to remain consistently legible. The design of San Francisco considered its uses spanning from a small watch to mobile devices to desktop Macs to an Apple TV home theater television.

Apple is likely taking a similarly broad approach to designing a GPU. As with San Francisco, Apple is almost certainly aiming purely to replace PowerVR on specific devices where it's needed most: iOS devices, Apple Watch and Apple TV, although it is possible that it could eventually seek to use its GPU on the Mac as well.

It might even make more sense to eventually use a custom Apple GPU in future Macs than to shift away from Intel x86 processors to use an Apple-developed ARM-based CPU. The former would require much less work from app developers than the latter.An Apple GPU could be designed to serve specific use cases Imagination doesn't have any reason to address

Regardless of whether it will ever be used in Macs, Apple's apparent intent to replace Imagination's GPU across all of the products that currently use PowerVR explains one reason why it would want to replace PowerVR in the first place: an Apple GPU could be designed to serve specific use cases Imagination doesn't have any reason to address.

The most obvious example is Apple Watch. There is not really any addressable "smartwatch market" outside of Apple, and certainly not one that focuses on the $300 price tier for a digital watch with a fluid UI with battery constraints closely aligned to Apple's. The primary competition to Apple Watch comes from a mix of luxury mechanical watches and simpler "smart band" activity trackers.

Imagination has no reason to design GPUs that scale down for optimized use in a wearable, the same way that there's no apparent demand for a general purpose typeface that scales effectively on a small OLED screen.

Another analog to San Francisco: prior to deploying it virtually everywhere, Apple had largely been using Helvetica, a font widely used by other companies. If Apple had simply worked to adapt Helvetica, other companies could copy its approach, and consumers would have a harder time seeing an apparent difference.

Along similar lines, other companies do currently use PowerVR GPUs in their competing products. A notable example is Amazon, which sells a low-priced Fire TV box that currently uses a Chinese MediaTek chip that incorporates a PowerVR GPU.

Apple could either work with Imagination to develop GPU designs appropriate for use in TV boxes (which other companies could subsequently avail themselves of) or it can develop its own GPU that scales effectively for TV applications (and that its competitors can't benefit from).

Having seen its desktop interface appropriated by Microsoft and its Post-PC mobile user interface appropriated by Google and then brazenly stolen by Samsung, Apple has good reason to avoid giving away its work when it doesn't need to.

The APFS GPU

A third broad technology that Apple is currently deploying is its new Apple File System (APFS), first announced last year and already deployed in iOS 10.3 (and apparently also in the latest watchOS and tvOS). It is expected to reach Macs later this year.

Like OpenGL and Helvetica, Apple's previous HFS+ file system was essentially functional and could have been left in place longer. However, there were also known issues that needed to be fixed to both enhance the current platform and to enable new features desired for the future.

Like Metal and San Francisco, Apple's APFS was designed to scale, taking into consideration the needs of a small wearable like Apple Watch all the way up to a desktop Mac, while also serving iPhones, iPads and Apple TV.

While Metal focused on stripping away unnecessary layers of OpenGL overhead and San Francisco accommodated font needs unique to Apple's product line, APFS replaces the existing HFS+ with essentially a future-proofed file system designed to accommodate growth and upcoming needs.

Similarly, today's PowerVR GPUs currently allow Apple to remain competitive in areas ranging from Apple Watch to iPhone to iPad Pro to Apple TV. However, to accommodate future capabilities, Apple will want a clean foundation to build upon, similar to what APFS does on the file system level. Like HFS+, the design of PowerVR dates back into a previous era of technology

Like HFS+, the design of PowerVR dates back into a previous era of technology. Its original design dates back to the mid 90s, but even its mobile-specific variants were developed more than a decade ago, then expanded upon in successive generations. Its "tile-based deferred rendering" is somewhat unique among GPUs, but shares some of its approach with ARM's Mali and Intel's integrated HD graphics engine.

Given how dramatically mobile technology has developed since 2005, it's not hard to imagine that there may be many new, more effective approaches to accelerating graphics than what Imagination has been expanding upon over the last decade. The fact that Imagination itself is working on a speculative, next generation design for future graphics makes that obvious.

The iOS GPU

With Imagination's Furian on the horizon, it may also be the perfect timing for Apple to launch a radically new GPU design that breaks with the existing status quo to offer a major leap in graphics capabilities.

Apple realized a decade ago that technology could enable a new class of mobile "super phone" with the ability to run desktop-class applications based on a scaled down macOS, rather than using one of the decade-old mobile platforms of the day originally designed for 1990's era PDAs or pagers (like Palm OS, Windows CE, Java Mobile/Blackberry or Nokia's Symbian). The result was iOS.

In a similar fashion, Apple could apply much of what it knows about mobile devices and operating system needs to develop a super GPU for the next decade of mobile devices (and potentially other applications, ranging from Macs to VR headsets to automotive).

Some of the concepts Apple could incorporate in a new GPU design include ray tracing rendering for photorealistic images with real time shadows, reflections and translucency— a capability Imagination demoed last year but that apparently hasn't been adopted. In addition to making mobile games look as impressive as more powerful desktop GPUs, such a technology could also radically enhance the user interface with fluid animations and advanced lighting effects.

There are many other areas where a custom-designed GPU could set iOS devices apart as dramatically as iOS itself did a decade ago. These could include new optimized GPGPU functions for file and memory compression, file encryption, new realtime video effects and filters, Machine Learning, and perhaps even specific optimizations for compiling JavaScript or Swift, accelerating the web or enabling software development on iPads.

Apple has touted its camera Image Signal Processor built into its A-series chips, noting that iPhone 7 can process "100 billion operations in 25 milliseconds" to perform such tasks as focus, exposure, white balance and noise reduction. The company also incorporates other computational engines in its custom SoCs ranging from audio noice cancelation to the Secure Enclave.

It may be that some general purpose functions currently performed by the CPU could be moved to a GPU core specially designed to accommodate a wide range of GPGPU tasks, or even be added as specific new hardware accelerated co-processing features, blurring the line between the CPU, GPU and other processing cores on the SoC.

Given the advanced work Apple already does in optimizing its mobile CPU cores and much of the other components of its A-series chips, it's not hard to grasp how the company could aggressively push mobile computing ahead in new custom silicon features. In addition to graphics, Apple is also rumored to be working on power management (currently outsourced to Dialog) and could eventually move into customizing its own modem hardware (currently outsourced to Qualcomm and Intel).

Apple can ship new tech

Other companies are also working on their own future plans for mobile GPUs. In addition to Imagination's Furian, Nvidia hopes to advance its Tegra tablet and automotive chips and Qualcomm has its proprietary Adreno mobile GPU. Samsung uses ARM's Mali GPU. However, Apple has a key difference in that it isn't just speculatively developing technology it hopes to be able to sell in the future.

Unlike any other company, Apple has the unique ability to immediately roll out new technologies across hundreds of millions of devices, both via software updates to its existing installed base, and through high-volume premium device sales that are effectively guaranteed to find buyers. Apple has the unique ability to immediately roll out new technologies across hundreds of millions of devices

In 2012 it rapidly converted its mobile products to Lightning, and the next year it launched Touch ID and 64-bit apps with the A7; both are now ubiquitous. Every year it refreshes 80 percent or more of its iOS installed base with modern software, rolling out such features as the aforementioned Metal, San Francisco and APFS.

Apple's largest competitor sells roughly 300 million smartphones every year, but only around 40-45 million of those will be premium class Galaxy S8 flagship models. Many are shipped new with old versions of Android, and will never be updated.

Samsung could design a new premium GPU but would have trouble spreading it very far because most of its phones are low-end models. Even its Galaxy S8 is split between at least two different GPU designs: its own Exynos/Mali and Qualcomm's Snapdragon/Adreno. Samsung couldn't even roll out Tizen phones or Galaxy Gear watches in meaningful numbers.

Google has proven it has little ability roll out its platform software to a significant slice of its installed base every year. Even Microsoft has shown it can't roll out Windows 10 rapidly— even when it gives it away and forces updates on users. Many hardware makers have also flopped out of the gate, ranging from Amazon's Fire Phone to HP's experiments with webOS to Nvidia's attempts to sell Shield hardware and Google efforts to sell Nexus Q, Google TV and Pixel C.

Apple could introduce a radical new GPU architecture and establish it across its product line extremely rapidly, potentially as quickly as this year, with iPhone 8, Apple TV 5 and Apple Watch 3. As it has in earlier years, it could also update previous models with a new SoC, including its new GPU on them as well.

Apple is not afraid to pay more for strategic advancement

Apple has regularly invested lavishly to deliver advanced features that its competitors would have difficulty copying, whether in hardware, software or in integrated silicon. In large part, it is uniquely capable of doing this because it generates billions in revenue from consistently successful, profitable hardware releases. Apple is now spending over $10 billion annually on R&D. Compare that Nvidia spends less than $1.5 billion per year.

Source: Above Avalon

The biggest complaint currently being made about Apple targets the lack of updates for the Mac Pro, a high end PC that sells in extremely small quantities. Everyone else in the PC or mobile industry has the opposite problem: all they have are Mac Pro-like flagship products that sell in limited volumes, and high-volume commodity products that don't sell for enough to warrant significant advancement.

Look at some examples of Apple's investments. In hardware, Apple paid $356 million to acquire AuthenTec, then invested additional R&D into perfecting and hardening Touch ID for reliable, constant use by mainstream users. Motorola's 2010 Atrix had previously attempted to use AuthenTec sensors, but couldn't get them to reliably work and abandoned the feature within a year.

AuthenTec noted that apart from Apple, other vendors balked at the high component cost of its best fingerprint sensors, largely due to the fact that outside of iPhones, smartphones are either high-volume / low-cost, or low-volume / high-cost, neither of which supports the use (or development) of a speculative and expensive new component.

Beyond hardware, Apple has also invested billions of dollars into developing software, ranging from iOS frameworks to Xcode development tools to finished apps (such as the movie making portfolio of Final Cut Pro, iMovie and the new Clips). No other mobile hardware maker has successfully written its own OS and maintained it for ten years while capturing a comparable lead in profitability.

The software "innovation" of Apple's competition— particularly Samsung— is largely made up of imitative gimmickry, thin UI skins and lazy basic bundled apps rather than cutting edge, productive investments in the development of core software technologies. Samsung has now worked on two OS releases, Bada and Tizen, that have flopped as an alternative to Android despite the fact that Samsung sells around half of the world's Android phones— and would clearly prefer to ship its own OS.

Google and Microsoft have spent lots of money on their mobile OS platforms without being able to recoup their investments in sales of hardware, OS licensing or from ecosystem-related income. Other attempts to launch mobile platforms— including webOS and Ubuntu Touch— have also suffered from an inability to fund their development.

At the intersection of hardware and software, Apple has also invested billions into silicon. In 2008, Apple began work to build its own silicon design team that resulted in 2010's A4, released just as third parties were offering their own SoCs that beat the commodity chips Apple had been using from Samsung.

Texas Instruments's OMAP, Nvidia's Tegra and Qualcomm's Snapdragon chip families had all threatened to speed Android and Windows Mobile past Apple simply through superior brute computing capacity. Apple not only kept up, but by 2013 the 64-bit A7 catapulted Apple far ahead of everyone else in the mobile device industry. TI and Nvidia dropped out of the phone business, and Qualcomm was sent reeling.

The time appears ripe for similar jump on the GPU side.