Affiliate Disclosure
If you buy through our links, we may get a commission. Read our ethics policy.

Ogg Theora, H.264 and the HTML 5 Browser Squabble

 

Pundits are roasting Apple over a scuffle raised by Mozilla and Opera to define the free Ogg Theora video codec as the official way to present video on the web in the new HTML 5 specification. The problem: HTML isn't supposed to define content codecs, and even if it were, Ogg Theora, commercially abandoned nearly a decade ago, doesn't have what it takes to deliver video on the increasingly mobile web.

While the tech media has largely portrayed the disagreement as either a huge roadblock for HTML 5 or a war between free software advocates and big corporations, the reality is that specifications like HTML 5 are not intended to enforce political views but rather to foster interoperability. At issue is the video format specified in the new HTML 5, a situation that is new because the HTML specification has never defined a simple system for embedding video in the same way that web developers can place ordinary graphics within their pages.

While web browsers use the "img" tag to recognize and display graphical elements in any number of formats, from GIF to JPEG to PNG, there has never been an equivalent way to simply place video or audio files within HTML. Instead, web developers have have to post their video as an embedded file, which various browsers may or may not know how to display properly.

This conspicuously missing link in seamless audio and video playback on the web has a history that parallels QuickTime and the development of the ISO MPEG video standards themselves, the same industry standards that Mozilla, Opera and some open source advocates are railing against. It also, of course, involves Microsoft.

The QuickTime Plugin

Displaying video that works properly across all browsers has been a tricky feat, as competing multimedia playback plugins usually only support specific container files (used to store the media data) and codecs (used to compress the audio and video data). Web developers originally used MJPEG (Motion JPEG), a simple video format that plays video using a series of JPEG stills and uses a basic raw WAV file for audio. That is the only format that works across browsers, but isn't very efficient in terms of quality versus download size.

Apple's QuickTime predated the web as a system for playing back video in a variety of different codecs, including MJPEG, and could support new codec components from third parties. When the web began to take off in the mid 90s, Apple provided a QuickTime plugin for Netscape and other browsers that enabled them to play audio and video directly within the browser. This initially made QuickTime the way to deliver video on the web, particularly in a world where Macs made up a disproportionate ratio of both web users and web developers.

After realizing the significance of the open web and its potential to marginalize its Windows monopoly, Microsoft targeted Netscape for death and took aim at QuickTime as well. Microsoft licensed the NCSA's SpyGlass browser code that Netscape was built upon and produced Internet Explorer, which it then bundled with Windows for free to destroy any market for Netscape's browser.

Microsoft's War on QuickTime

Copying and killing QuickTime proved to be harder. Microsoft had already failed to match QuickTime's performance with its own Video for Windows product in the early 90s, and then found itself in hot legal water after it intentionally stole Apple's QuickTime code in an effort to make VfW a viable competitor on Windows in the San Francisco Canyon scandal. After years of failing to unseat QuickTime as the standard for video playback on Mac and Windows systems, Microsoft stepped up its efforts to take over web video in the late 90s by simply making Internet Explorer ignore the QuickTime plugin users had installed.

In the Microsoft Monopoly trial that centered around the company's abuse of Netscape, Apple executives testified that while Netscape worked fine with the QuickTime plugin, successive versions of Internet Explorer increasingly failed to properly send video to QuickTime, creating the impression for consumers that QuickTime didn't work correctly. In the trial, Apple presented a chart of 22 media file types, all of which were passed to QuickTime by Netscape 4 on Windows 95. In Internet Explorer 3 however, only 15 were passed to the QuickTime plugin; in IE 4.0 only 11 were passed, and under Windows 98, only 4. Microsoft's lawyers only claimed that nobody could prove the company was breaking compatibility with QuickTime on purpose and maliciously.

At the same time however, Microsoft representatives were threatening to destroy Apple's business within video authoring unless the company agreed to scuttle QuickTime playback on Windows and grant Microsoft competition-free access and control over video playback on Windows. Unlike other companies that played along with Microsoft's threats, Apple turned down Microsoft's famous demand to "knife the baby" and took on the company head to head in video playback.

Vigorous competition against Microsoft resulted in the release of QuickTime 3 against a variety of technology announcements by Microsoft that largely failed to materialize. Microsoft's promised, cross-platform Active Movie never arrived, and the intent to deliver a new Windows Media container file also never gained real traction. Instead, in 1998 the ISO adopted QuickTime's container as the basis for the forthcoming MPEG 4 suite of multimedia standards, a high profile rejection of Microsoft's plans.

Microsoft pushes Flash against Adobe

While fighting Apple on the QuickTime front in video, Microsoft was also embroiled in a fight against efforts by Adobe and Sun to promote PGML, based on Adobe's PostScript, as an interoperable, open standard for presenting vector graphics on the web. Microsoft teamed up with Macromedia to submit its own competing VML standard. The W3C standards body eventually worked out a compromise that drew from both, creating a new standard called SVG (Scalable Vector Graphics).

Rather than supporting the new SVG standard, Microsoft continued to push VML in Internet Explorer, resulting in market confusion and no real adoption of SVG. On top of that, in order to help scuttle any adoption of Adobe's free SGV web plugin, Microsoft began bundling Macromedia's proprietary Flash player with Internet Explorer 5 in 1999, giving the Flash vector graphics plugin the same broad audience over the Windows monopoly that it had leveraged in reverse to kill QuickTime by breaking compatibility with certain movie formats.

Microsoft's anti-compeitive suppression of QuickTime subsequently resulted in a new application for Flash: video presentation. Because the plugin was widely installed, it could be used to package up proprietary video for opaque playback in the browser. Rather than linking to a standard video file for browser playback using a video plugin, web developers began to package their videos up as a closed Flash movie file for playback by the Flash player plugin Microsoft was widely distributing with Internet Explorer.

Anyone who didn't use Internet Explorer could download the Macromedia Flash plugin, just like downloading the QuickTime plugin. The difference was that because web video was now locked up in the Flash format, it couldn't be played back using alternative players in the way that QuickTime-compatible movies can be played back by alternative media software. Additionally, Macromedia's Flash player really only worked well on Windows, effectively tying Flash content to Internet Explorer.

Apple takes back media with the iPod

Having failed to take over web video itself, Microsoft had at least managed to attach web video to a format that worked best on Internet Explorer. The company then turned its attention to taking over standard video formats used in publishing music and video. Microsoft hoped to replace DVD's ISO MPEG-2 standard with a high definition, heavily DRMed Windows Media format on the same disc, called iHD. Microsoft also pushed into the music business to replace MPEG MP3 with Windows Media Audio, enticing the studios with promises of pirate-proof DRM there as well.

However, Microsoft failed to deliver its promised Windows Media DRM technologies on time, resulting in an opening for Apple's iPod and the iTunes music store. Apple adopted the ISO's standard MPEG AAC (advanced audio codec), which enabled sophisticated data compression and supported studio-demanded DRM. The success of the iPod trampled Microsoft's WMA as a commercial audio distribution format, and iTunes for Windows even managed to make AAC the new standard on both Macs and Windows, quite the coup for company that had been written off as a 2% also ran in the desktop wars.

Apple then did the same thing in video, promoting the state of the art MPEG H.264 standard for video compression in iTunes video with hardware video decoding support on the iPod for efficient playback. Having failed to gain any real traction in pushing WMV playback into DVDs as a quasi-standard, Microsoft managed to rubber stamp its own WMV codec through its SMTPE pet standards body in the US, resulting in it being renamed VC-1. It then got VC-1 licensing mandated as part of both the Blu-Ray and HD-DVD specifications, although only the ill fated HD-DVD ever really used it. Meanwhile, iTunes continued to popularize H.264 in the same way Apple had helped launch AAC among consumers.

Apple takes on WMV and Flash

Not content with simply encouraging the use of interoperable, open standards, Apple also took up the task of removing dependancies upon non-standard codecs designed to tie content to Windows exclusively. Microsoft had refused to support the latest version of WMA and WMV on the Mac, and Macromedia had done little to make its Flash plugin work well on Macs, Linux, or in other browsers besides Internet Explorer.

To ensure that commercial content could continue to play back everywhere, including its own Mac and iPod hardware, Apple refused to support Microsoft's proprietary WMA and WMV playback on the iPod, channeling iPod demand into creating a music market that only rewarded interoperable formats from raw WAV audio to published standards such as MP3 and AAC. It did the same with iPod video, supporting only MPEG-4/H.264 formats at the expense of WMV, Real Video, and other proprietary formats.

With the iPhone, Apple did something parallel to Flash: it similarly refused to support video playback of Flash movies, officially citing that this was only because Flash, now owned by Adobe, didn't work well enough to include it. While that was true, the real reason for killing Flash on the iPhone was to push web developers back toward publishing standard video formats rather that wrapping their video content up in a proprietary Flash binary.

Somewhat ironically, Microsoft itself has now taken aim at Adobe's Flash by introducing its own copy originally called Sparkle, along with the Silverlight runtime. With Internet Explorer now facing increased competition and plummeting market share, Microsoft's ability to assassinate competitors' plugins as it did with QuickTime and its power to coronate others, as it did with the Flash plugin, have weakened.

HTML 5 and the Ogg Theora option

Meanwhile, the browsers taking away Microsoft's control of the market, including Opera, Mozilla's Firefox, Apple's Safari, and Google's Chrome, are working together to solve the problem of having no standardized way to present video. HTML 5 defines simple ways to embed video files so that the client browser can display them. Microsoft's strategy seems to be to wait and see how those efforts develop, and perhaps attempt to derail them before they obsolete any real need for Flash or Microsoft's own anti-interoperable, web-standards hostile, belated clone of Flash in Silverlight.

Apple already convinced Google to serve up its YouTube videos as H.264 in addition to Flash binaries, initially to support playback on the iPhone. Google has also demonstrated a version of its YouTube website delivering video using HTML 5's native support for publishing H.264 video without Flash.

The remaining problem is that two browser companies that depend upon free distribution of their software are opposed to licensing the H.264 codec. Instead, they wanted the W3C to designate Ogg Theora as the official video codec of HTML 5. Ogg Theora is based upon VP3 video technology originally proprietary to On2, and subsequently abandoned after it became obsolete. Flash video is based on On2's VP6, and Skype's proprietary video conferencing uses VP7. In the face of interoperable standards published by the ISO, On2's proprietary codecs are increasingly in the same boat as Microsoft's WMV/VC-1, without Microsoft's remaining clout.

The open source community has embraced the obsolete VP3, abandoned by On2 back in 2001, as a viable alternative to today's H.264. For software developers, using Ogg Theora allows them to avoid paying royalties in order to share in the patent pool contributed by MPEG's member companies. This has also resulted in adoption of Ogg Theora by video game developers and organizations like Wikipedia.

Ogg Theora vs. H.264

However, hardware makers like Apple and Nokia are opposed to HTML 5 being tied to Ogg Theora for a number of more practical reasons. First, the cost of licensing H.264 isn't significant to these companies in the way it is to smaller software developers, and particularly the browser vendors who hope to distribute their software for free. While Apple also distributes Safari for free, most of its users are buying Macs or iPhones to run it. In contrast, Mozilla's entire Firefox business model revolves around Google paying it around $50 million a year to direct search queries its way. Larger companies like Google similarly have no cost-related preference for using Ogg Theora.

Second, since companies like Apple, Google and Nokia are already invested in H.264, the prospect of having to support an old obsolete codec is not at all desirable. Google has noted that there's no way the company could serve up YouTube's billions of streams using the much less efficient Ogg Theora codec, saying it would consume the world's Internet bandwidth due to its less sophisticated compression. Ogg Theora also lacks the hardware acceleration available for H.264, making it completely unattractive for use in mobile devices from netbooks to smartphones.

Opera and Mozilla don't make mobile hardware, so they don't care about this. Mozilla doesn't even have a viable mobile browser. Opera's mobile business largely centers on Opera Mini, which isn't really a web browser but actually an applet that displays pre-rendered web pages served by Opera's web proxy servers to less powerful phones that can't run a real browser. All of the advanced new mobile devices use a WebKit-based browser, from Safari on the iPhone and iPod touch to the Palm Pre, BlackBerry Storm, and Google Android devices.

Apple has also voiced concerns that Ogg Theora may be encumbered by unknown patents. That risk isn't significant to Mozilla and Opera, both of whom could simply abandon the format for something else in the same way that the web temporarily abandoned commercial support for GIF after Unisys tried to sue everyone for using it. However, for a company like Apple that has built a business that requires selling media content and supporting hardware acceleration in mobile devices, it's not possible to randomly drop a codec technology when a submarine patent threat appears. Apple also has a bankroll to attract patent trolls that Mozilla and Opera both lack.

A problem that lacks any need for a solution

The real problem with the squabble over the future of HTML 5 is that the standard doesn't need to specify an official codec. There's no official codec for graphics; web developers can use JPEG, GIF, PNG, or any other format. If users can't see the image, they might need to load a helper plugin. There is no problem related to lacking an official graphics format.

The same is true with video. In fact, despite all the outrage being trumpeted by Mozilla, there's no reason the browser needs to license support for a video codec anyway. If users are smart enough to download Firefox, they can also download QuickTime and watch H.264 video on Apple's licensing dime, or they can download one of the free H.264 capable codecs available for FOSS platforms, including the GPLed x264, the engine used by VLC, FFmpeg, and HandBrake.

Further, there's no real reason web developers need to only serve one codec. It's easy in HTML 5 to offer up video as H.264, Ogg Theora, and even as a Flash binary so that anyone can watch it. As far as saving organizations like Wikipedia from having to license MPEG, there's no real problem for users to download a package of open source codec components for QuickTime or their media system playback of choice that enable Ogg Theora playback on non-mobile systems.

This whole imagined war over the official video codec of HTML 5 simply a non-issue. What is an issue is HTML 5 adoption. In addition to promoting interoperable video, HTML 5 also enables rich application support including client side databases for fat client sophistication and offline support. HTML 5 is designed to replace the need for a proprietary plugin like Flash or Silverlight just to present web content with sophisticated client-side interoperability.

The only question now is who will deliver the best HTML 5 support, and how quickly the foot-draggers seeking to hold onto the past decade's proprietary technologies will be left behind.