Apple launches HTTP Live Streaming standard in iPhone 3.0

At the March unveiling of iPhone 3.0, Apple only dropped a subtle hint about new streaming video features in the new operating system (literally limited to writing "streaming video" on the slide of other features, below), leaving out any details about how it would work and not even mentioning the feature in any detail in the presentation.

For the last decade, Apple has been selling QuickTime Streaming Server, which uses an RTSP (Real-Time Streaming Protocol) server to stream live or rebroadcast video feeds to viewers. Apple uses this technology to stream some of its own live events. However, despite offering royalty free streaming and also delivering it as an open source project, QuickTime's RTSP streaming server hasn't gained the traction it was once expected to achieve.

A large part of this is due to the fact that RTSP traffic is blocked by many firewalls, making it difficult to deliver streams reliably. The audio and video conferencing used by iChat also relies on RTSP, causing some users frustrating problems for the same reason. Getting RTSP video streaming to work on the iPhone would be even more difficult, as it routinely moves between mobile and WiFi networks.

Apple attempted to solve the RTSP problem long ago in QuickTime Streaming Server by creating an option to bundle up RTSP streaming video traffic into HTTP packets, which appear identical to standard web traffic and therefore are permitted through most firewalls. This involves a extra layer of overhead however, resulting in a greater demand for bandwidth. For the iPhone, Apple decided to pursue a different strategy, which it calls HTTP Live Streaming.

HTTP Live Streaming

The technology behind HTTP Live Streaming leaked into public knowledge in May when Apple submitted it to the Internet Engineering Task Force (IETF) as a draft standard on track to become an RFC (or Request For Comments, the memorandum used by the Internet Society to define how technologies work in order to foster cooperation and compatibility between the vendors implementing them).

Apple's HTTP Live Streaming proposed draft looks a lot like a method Microsoft began selling last year, called Smooth Streaming. The difference is that Apple's proposed IETF standard can use anybody's encoder and broadcast server, and will work with any client software designed to receive the stream. In contrast, Microsoft's Smooth Streaming is of course designed to exclusively use Microsoft Expression Encoder, Microsoft Internet Information Server with a Smooth Streaming extension, and requires Microsoft's Silverlight 2 on the client.

Essentially, Apple wants a standard for streaming video that anyone can use so that it can continue selling hardware without being either shut out of the market by proprietary software, or held captive by it; Microsoft, as a software vendor, wants to create another captive market where it has the power to shut out competitors at its whim. In parallel to Microsoft's Silverlight Smooth Streaming, Adobe also offers an equivalent Flash-based streaming server of its own.

If this is all beginning to sound familiar, it's because video streaming has followed much of the same historical trajectory as multimedia playback, making the history of streaming another chapter in the history of QuickTime.

The advent of streaming

Back in the mid 90s, Apple's pioneering advancement of software-based desktop video authoring and playback gave the company a strong lead in multimedia computing. With the arrival of the Internet however, there seemed to be a huge potential for sending efficient streams of video to users (primarily over dial-up) instead of relying on CD-ROMs for distribution of large video files or expecting users to directly download huge videos over dial-up connections.

Internet media streaming was popularized by Progressive Networks in 1995 with its proprietary RealAudio streaming format. In 1997, the company was renamed RealNetworks and launched a RealVideo service as part of RealPlayer 4.0. It also partnered with Netscape to develop what would become the RTSP standard for streaming.

Real had been founded by Microsoft millionaire Rob Glaser. Microsoft owned ten percent of the company and licensed Real's streaming formats in NetShow, its product aimed at killing Netscape's streaming server. Microsoft's NetShow incorporated Real's streaming formats for compatibility with existing content, but hoped to eventually shift Internet streaming to its own new ActiveX Streaming Format (ASF). Despite its interests in Real, Microsoft's growing ambitions resulted in the company pitting itself against RealPlayer with its own Windows Media Player in 1998, a phoenix that rose from the ashes of 1996's Active Movie/DirectShow player, which themselves were rebranded versions of the company's ill fated QuickTime competitor originally named Video For Windows.

Just as QuickTime suddenly failed to work properly under Windows 98 and Internet Explorer, Windows Media Player suddenly stopped playing Real's streaming formats, as Glaser testified in the Microsoft Monopoly trial. Real executive David Richards also testified that Microsoft was pressuring AOL to drop support for Real and use Microsoft's own streaming software instead, citing an email on the subject from AOL's CEO to Glaser which warned, "They want to kill you guys so badly, it is ugly."

Microsoft hoped to own the future of streaming and digital playback both, so it took on Real and Apple at once, pushing the idea of streaming ASF (the Real killer) via MMS (Microsoft Media Server, the new name for NetShow and not to be confused with the mobile messaging protocol) and establishing ActiveX Authoring Format (AAF) as its QuickTime killer.

In 1998 AAF was rejected by the ISO in favor of QuickTime as the basis for the new MPEG-4 media container format. By 2003 MMS, which used its own proprietary system for streaming media, had been deprecated by Microsoft in favor of its own new RTSP server, Windows Media Server 9. After the ActiveX brand was sufficiently tainted by widespread security flaws, the A in ASF and AAF was changed to stand for "Advanced." Most recently, Microsoft was forced to drop its ASF and adopt the MPEG-4 container to support Smooth Streaming.

During its streaming battle with Microsoft, and without any other revenue streams to fall back on, Real turned itself into an adware vendor that attempted to leverage its existing value in RealPlayer to inundate users with marketing partners' messages and attempts to sell them subscription music. It also filed suit against Microsoft and won an antitrust settlement of $460 million in 2005.

Apple jumps on streaming bandwagon

While still recovering from its mid 90s brush with death, Apple, unlike Real, did have other real business to keep it going. It released QuickTime 3 in 1998 with a sort of fake streaming called HTTP Progressive Download. Rather than actually streaming video in real time, it only allowed users to begin downloading a file and start watching the portion the was available.

The next year however, at NAB 1999, Apple released QuickTime 4 with QuickTime Streaming Server, which supplied real standards-based RTSP streaming. Rather than imposing a per user royalty fee for streams, Apple allowed unlimited streaming use, hoping this would enable it to catch up in the streaming business dominated by Real and demanded by Microsoft. Apple also released the software as open source as the Darwin Streaming Server.

In a press release, Steve Jobs, still acting as interim CEO, stated, "Finally, streaming live video and audio over the Internet no longer requires proprietary software and expensive servers. By including streaming as part of Mac OS X Server and introducing Darwin Streaming Server, Apple is significantly lowering the cost of streaming digital video and audio and the result should be a deluge of high quality streamed content."

Apple remained in third place in the streaming market however, and the deluge of streaming content didn't materialize as expected. Internet radio based on streaming MP3 files did begin to take off however, using the SHOUTcast protocol developed by Nullsoft. Along with the GPL Icecast server, QuickTime Streaming Server also adopted SHOUTcast audio streaming server support as a feature, and Apple also added radio streaming client support to iTunes.

Streaming stymied by technology, licensing

In 2002, Apple launched QuickTime 6 with the new QuickTime Broadcaster, which enabled users to capture live video and stream it, either unicast to another user or multicast over a local network where multiple users could view it. Multicast transmission of video is efficient, but isn't allowed over the Internet because ISPs can't decide how to bill users for the traffic, which rather than being unicast point to point like a telephone call, is instead spread out for multiple users to receive more like a television broadcast.

QuickTime Broadcaster can also send a video stream to QuickTime Streaming Server, which will then reflect the single stream to a variety of other unicast clients, or relay the signal to other servers for load balancing. A remaining problem is the RTSP firewall issue; getting around this requires a network of servers that encapsulate streams as HTTP packets and then rebroadcast them to local users inside the firewall, a solution that doesn't work outside of large corporate installations.

Another factor holding back streaming was MPEG-4 licensing. In its press release, Apple noted, "QuickTime Broadcaster supports the broadcast of MPEG-4, but as with QuickTime 6, the distribution of QuickTime Broadcaster is being delayed until MPEG-4 video licensing terms are improved. The MPEG-4 licensing terms proposed by MPEG-LA (the largest group of MPEG-4 patent holders) includes royalty payments from companies, like Apple, who ship MPEG-4 codecs, as well as royalties from content providers who use MPEG-4 to stream video. Apple agrees with paying a reasonable royalty for including MPEG-4 codecs in QuickTime, but does not believe that MPEG-4 can be successful in the marketplace if content owners must also pay royalties in order to deliver their content using MPEG-4."

The iPod's end run around streaming

One of the main purposes of streaming was to get around the issue of limited Internet bandwidth in the late 90s. At a time when most users only had dial-up, streaming audio or video could make more sense than waiting for a download to finish. By 2003, with broadband nearly ubiquitous, Apple followed up the launch of the iPod with the new iTunes Store, which in addition to streaming Internet radio now offered a library of music for progressive download. Apple also happened upon a new demand brewed by the iPod itself: podcasting.

Using an RSS feed, content creators could publish their broadcasts as progressive download files rather than streaming them in real time. With this new technology, the emphasis on streaming reverted back to the CD-ROM and Walkman climate of the early 90s, where users obtained high quality prerecorded content in advance of listening to it rather than trying to stream content in real time and having to put up with lower quality streams and needing some cost effective way to tune into these streams.

Microsoft continued following Real's efforts to market music subscriptions; in contrast, Apple quickly established itself at the top of this new business of both selling prerecorded content and delivering podcasts. iTunes served as a library for listing the podcasts of conventional broadcasters, and Apple also encouraged universities to publish their content as free podcasts in iTunes U. Podcast Producer, part of Mac OS X Server, enables organizations to create workflows that capture video from remote clients, perform automated editing to add opening videos and titles and end credits, and then produce output files ready for iTunes distribution as podcasts or streams for QuickTime Streaming Server.

iPhone 3.0 introduces mobile streaming

While iTunes users can listen to radio streams from their broadband PC, there wasn't a practical way to take streamed content with them on their iPods. That is, until the iPhone appeared with a hefty mobile data plan in tow. Suddenly, the landscape changed again, with a new demand for streaming to take advantage of the bandwidth customers had already paid for when buying the iPhone.

Among the first apps for iPhone 2.0 was AOL Radio, followed by a series of others that delivered audio or video streams proprietary to their provider: BBC, TV.com, and even users' own video with SlingPlayer. AT&T cried foul, asking that its mobile network not be used to capacity by apps on the device when other providers were making a killing selling their users little audio and video clips.

Apple didn't build RTSP support into the iPhone, leaving vendors to work out their own delivery mechanism. Navigating the problems of firewalls and roaming between networks would likely make video streaming over RTSP little more than frustrating, with a constant rebuffering of the stream to annoy users. Instead, Apple has now adopted this emerging alternative it calls HTTP Live Streaming.

Unlike progressive downloads, HTTP Live Streaming actually does stream content in real time, although there can be a latency of as much as 30 seconds. It works much simpler than RTSP; essentially, the content to be broadcast is encoded into an MPEG transport stream and chopped into segments that are around ten seconds long. Rather than getting a continuous stream of new data over RTSP, the new protocol simply asks for the first couple clips, then asks for additional clips as needed. This works great through firewalls, and doesn't require any special servers because any standard web server can deliver the chopped up video segments.

Where HTTP Live Streaming shines

The real benefit to HTTP Live Streaming is that the server can maintain multiple versions of the clips in different formats. This allows an iPhone user with a WiFi connection to negotiate a higher quality version of the video than if only EDGE were available. Even better, the phone can renegotiate a higher or lower quality dynamically if it improves or loses signal. This enables the watcher to experience the best video quality possible at the current bandwidth available, continually optimized as new segments are requested.

Unlike Microsoft's Smooth Streaming trojan horse for Silverlight, HTTP Live Streaming works with any playback client on any platform and does not involve a layer of DRM, although it does support encryption, allowing broadcasters to limit access to their content. Because support is built directly into the iPhone's embedded QuickTime player, users don't even need to download apps for every broadcaster or channel; content creators can simply publish their feeds within a standard website, and iPhone can access them just like a desktop client. Other phones can similarly support the same interoperable standard, providing a leg up for mobile platforms with less commercial attraction or no ability to run real applications, like the Palm Pre.

The embedded support for HTTP Live Streaming in the iPhone also provides a route around AT&T's App Store limitations, enabling vendors to deliver their content, unrestricted, to iPhone users at optimal quality using published standards over the web, even directly from their iPhone apps. It also standardizes video playback, allowing publishers to simply hand the iPhone content rather than having to build their own player. As Apple adds features to QuickTime, those apps will simply benefit from those enhancements, and will all look and work consistently.

The simple video streaming outlined in HTTP Live Streaming is similar to SHOUTcast in that it uses a regular HTTP server and posts content data using and extended version of the M3P (MP3 Playlist) format that supports video as well. That will make it easy for homebrew broadcasters to set up Internet TV broadcasts following the same pattern as Internet radio.

What's next? The obvious followup is to add support for HTTP Live Streaming in Apple TV, allowing for HD streams direct from broadcasters, facilitating the ability to only pay for channels you want to watch, skipping around the local cable monopoly while gaining access to content they don't carry.

The same content would also be accessible on the iPhone, a desktop PC, or any other device with the capacity to play modern video codecs. And that's why Apple is not supporting Mozilla's efforts to use the obsolete Ogg Theora on the web, which lacks silicon support for hardware acceleration on mobiles and appliances.

For examples of HTTP Live Streaming, see iphone.akamai.com. Viewing live streams requires iPhone 3.0 or Snow Leopard QuickTime X.