Editorial: Apple's next hardware play could be in game controllers

Apple is thought to be working on Augmented Reality glasses that are perhaps two years out. But in moving the AR experience from the handheld screens of iOS devices to a fixed panel you look through, users will need a new way to navigate AR worlds. Little attention has been given to the role of game controllers in AR navigation, but Apple's patent filings and its recent announcements suggest that its next new play in hardware could be an advanced gaming controller, perhaps even a wearable one.

AirPads for wireless kinetic gameplay anywhere

Image a wireless gaming controller designed to use motion gestures, buttons, directional joysticks, touch, and haptic feedback. You don't have to imagine, because Microsoft, Nintendo, and Sony have been working to perfect the versatile home console gaming controller for decades. However, these three companies have only tied their controllers to their own, latest generation consoles. Each generation of those consoles has typically sold in the ballpark of around 100 million units. And only recently have these companies embraced Bluetooth for their wireless connectivity.

Apple has an installed base of 1.5 billion devices globally, most of them iPhones that are always wirelessly connected to a mobile data network, and they all support Bluetooth connectivity. Apple has previously floated plans for Made For iOS (MFI) game controllers, but the types of games that are popular on iOS devices and Macs have somewhat limited the demand for third-party game controllers.

However, a few things are changing. First, Apple launched gaming on its Apple TV box two generations ago. And it has increasingly made games a bigger part of its Mac and iOS App Stores. And this year, it unveiled Apple Arcade as new gaming subscription to monetize the development of better games across its platforms— games that are not trying to support themselves with annoying ads or In-App Purchases.

Apple Arcade not only hints at a new Apple TV, but also a new controller optimized for cross platform play

On the hardware side, back in 2013, Apple acquired PrimeSense, the motion-based gaming camera maker that powered Microsoft's original Xbox Kinect. Apple turned the technology into TrueDepth for iPhone X, launching new vocabulary into the world including Animoji and Memoji, along with Portraits and Portrait Lighting. It also developed all kinds of AR features including face-based graphics augmentation that was put into use by Snapchat and other third-party developers to turn participants into graphically enhanced avatars. Yet the original intent of PrimeSense involved motion-based gaming, something that Apple hasn't yet turned into a product.

Additionally, the advanced custom silicon powering Apple's mobile and home devices has made incredible leaps over just the past four years, accelerated by its own development of the new custom Apple GPU, as well as the maturing of its Metal framework for optimizing the performance and blurring the differences between its hardware platforms.

Finally, this summer Apple announced that it would be adding support in its Game Controller framework for Sony's Playstation 4 Dual Shock and Microsoft's Xbox One S and Elite 2 controllers. That required work to abstract the differences between the two standards and Apple's own MFi design so that existing games would work seamlessly with any of these third-party controllers.

Apple announced support for other game controllers iOS 13 and tvOS, but it can also make its own

What's stopping Apple from developing a game controller? In the past, the company has delegated this task to third parties— perhaps because it thought that the market was too small to be concerned with. When it shipped the first tvOS-based Apple TV in 2015, it paired it with the Jony Ive-designed Siri Remote, effectively a simplified TV remote designed to act as a basic game controller. While the Siri Remote does okay as a playback controller, it is a terrible game controller.

But the elegantly bespoke thinness of Ive's Siri Remote isn't the only work Apple has put into input controllers. In a recent patent filing the company detailed the concepts behind a wearable interaction tool in the form of a "finger-mounted device with sensors and haptics."

An Apple patent application image showing potential mounting points for finger sensor units

Add up all of the above changes, advancements, and research and it creates the picture that Apple appears likely to enter the game controller business. In part, to sell new hardware to its Apple TV and iOS buyers, but also because it now has advanced technology capable of significantly improving the game controller as a product: the custom silicon driving proprietary advances of AirPods and Beats wireless headphones.

Imagine getting a set of "AirPads" which allow you to play games on Apple TV, your Mac, and your iOS devices, but which can also interact with AirPods to play audio-based AR games. You could virtually fight your friends or other players "Mortal Combat" style, with nothing more than your handheld or wearable AirPad controllers. Or you could challenge other players to a "DDR" dance-off, or something related to "Guitar Hero," "Tetris," "Jenga," "Angry Birds," "Pokemon," or any number of other arcade titles designed to make use of wireless mobile game controllers.

The next step in the future could be adding AR glasses for visuals that make your wireless gaming even more engaging. But Apple is still working on the building blocks of ARKit and the tools to make the development of AR worlds easy, such as its new Reality Composer tool. Game controllers already have immediate use and market, and the content for using them already exists in the App Store.

Apple is still building the foundation of AR tech and content, but gaming controllers already have a use and apps to use them with

Setting up a future launch

This strategy would have a lot in common with AirPods. Apple embraced Bluetooth and BLE early on, giving it head start over Android and Windows Phone. Apple then acquired Beats, giving it a huge retail headphones business. It subsequently launched its own AirPods, leveraging the custom W1 silicon it developed to erase some of the pain-points of existing wireless headphones, facilitating easy pairing, movement of headphones between devices, improving latency, enhancing sound quality, and boosting signal resiliency.

These silicon and software advances that Apple made to the existing category of wireless headphones turned AirPods into a huge hit, one that is still raging years later. Competitors have been unable to successfully copy Apple's success, despite nearly everyone trying to do so. Many analysts kept telling us AirPods weren't going to "move the needle" until it was obvious that Wearables were indeed becoming a huge business segment for Apple, one that is growing rapidly.

With a wireless gaming controller taking advantage of the same or similar custom Apple silicon, the company could create another hit with an advanced controller that acts as a unique "anywhere" gaming system on its own, paired with a Watch or iPhone and linked up with a pair of AirPods. Apple could gamify exercise and fitness; create server-based multiplayer gaming for physical challenges such as virtual sword fights, capoeira, laser tag, or other exercises; and enable new ways to interact at meetings, presentations, and conferences.

It's not hard to imagine that the company that introduced and popularized the mouse, the trackpad, the touchwheel, and multitouch gestures could introduce a versatile new wireless gaming controller input device that would work everywhere from legacy game consoles to modern iOS devices to future AR systems. And now that Ive is no longer calling all the shots at Apple, such a game controller doesn't have to look like a super-thin rectangle with rounded corners.

This all happened before

Apple has regularly lined up the prerequisites for a secret new product in public view. As detailed above, it could be doing this again with gaming controllers. But consider how many times it has done this in the past.

Apple's pioneering support for FireWire and USB and its acquisition of what would become iTunes foreshadowed its venture into iPods, something that nobody in the Apple universe anticipated, even though the market for MP3 players was not brand new and Apple had been making the "rip, mix and burn" of music an increasingly important element on its Macs.

Years later, while the media imagined that the future would be played out by mobile phones playing MP3s— largely because this was the biggest threat they could imagine for Apple's iPod business— Apple instead aimed higher to deliver desktop-class computing on its new mobile iPhone, tethered to a data plan. Apple had developed a series of things to make this possible, ranging from mobile hardware savvy with iPods to building its own Safari web browser and sophisticated support for device data sync and digital software app sales. Yet months before iPhone was released, many still thought Apple would just license Symbian from Nokia or repeat its effort to add iTunes to a Motorola handset.

Back in 2006 I actually imagined Apple could build a full screen, touch iPod Pro in two models, one oriented to business users with messaging, email, a browser, device-synced contacts and calendar apps, and another designed for consumers that would let them take photos, videos, and voice clips and blog their experiences or listen to music and show off their photo albums. As it turned out, Apple made both devices in one, opening up the world to Instagram, iMessage, Safari, Keynote and Angry Birds on a handheld device.

Imagine an iPod playing media and running apps. Apple delivered it.

Around 2008, at a time when many thought the future of PCs were represented by cheap netbooks, Apple introduced iPad, a new tablet based on multitouch and aiming at simplicity, not just cheapness and miniaturization of yesterday's PCs. Apple had already done the work to deliver such a device, but had to wait until the market for mobile tablet devices was opened up by sales of iPhones subsidized by mobile data plans. There had been many stabs at tablet PCs by Microsoft, Samsung, and others, but the iPad arrived and took over the entire market because Apple spent time thinking about how to launch it correctly.

Conversely, while HTC Vive, Samsung Gear VR, Facebook Oculus, Google Glass and Cardboard, and Microsoft Hololens all attempted to deliver the far-off future of VR headsets, Apple instead aimed at what was really deliverable, focusing its efforts on AirPods to deliver always-there audio AR instead. This far more practical approach has turned into a multi-billion dollar business for Apple while the rest of the industry diddled way billions to deliver something nobody could use or afford, and for which there wasn't any compelling content.

Across 2016, pundits demanded to know where Apple "VR" product was, after earlier demanding a Mac netbook and a Curved or 3D iPhone.

Similarly, while Amazon, Google, Microsoft, and Samsung have worked to "outdo" Siri with expansive voice-first platforms of their own, and have tried to give away WiFi mics as a way to build a voice platform among affluent users, Apple has instead simply made Siri a feature across all of its hardware, from Macs to Apple Watch to iOS devices and Apple TV. Apple now has, far and away, the world's largest voice assistant platform, spread across the most languages and countries, and focused on valuable, affluent buyers of premium hardware.

Apple now offers HomePod as a premium home speaker with Siri features and HomeKit integration. Media critics and analysts have tried desperately to portray it as a failure, despite it selling in the millions at a premium price point and earning profits that no other voice speakers appear capable of making. Apple's HomePod is already doing business in the magnitude of Sonos, establishing itself as a significant player in the home as a base for HomeKit accessories and streaming integration with Apple Music and iOS devices.

This again suggests that Apple could broaden its move into game controllers with a stand-out new model, paired with advanced silicon that makes it more reliable, easy to pair, automatic to use between your devices, and compelling for third parties to build gaming experiences around. It could even make use of wireless charging like Apple Watch and the latest generation of AirPods, meaning it needs no exposed ports at all. It could be gloves, a handheld device or some other form of wearable.

Apple is serious about gaming software, and you don't have to look any farther than Apple Arcade and ARKit to see that. It should really think about making game controller hardware.