Road to Mac OS X Leopard: QuickTime, iTunes, and Media Features
In addition to the new searching, browsing and back up features for office documents (Finder, Dock, Quick Look, Cover Flow, Spotlight, Time Machine) and new support for collaborative information sharing (Mail, iChat, iCal, Server), Mac OS X Leopard also advances support for audio and video media in QuickTime, iTunes, Photo Booth, Front Row, and other applications. Here's what's new in Leopard and where the ideas behind rich media on the desktop came from.
The Origins of QuickTime
After Steve Jobs left Apple in 1986 to begin work on an advanced workstation system that would provide the "NeXT" advances in computing, a significant number of engineers and designers from Apple followed him, hoping to start work on something truly new and exciting. NeXT focused much of its attention on solving the issues the Mac operating system was about to run into: limited hardware portability, a weak underlying operating system foundation, and the complications it posed for software developers.
While NeXT worked to solve many of the underlying problems of the Mac platform, Apple continued its own plans to extend the range of the Mac's application-level software. In 1987, Apple's Color QuickDraw replaced the Mac's original drawing routines with an improved architecture. While being backwardly compatible with most existing software, it added support for color graphics and video.
Color QuickDraw initially only supported 8-bit pixel depth (a palette of 256 colors), because all color graphics hardware was prohibitively expensive at the time. The high price of 8-bit color came from its requirement for at least eight times the video RAM compared to the original Mac's 1-bit black and white graphics; the additional demands of moving those extra bits around also required dedicated computational processing power similar to or greater than the Mac's own main CPU.
The 1987 Macintosh II (above) cost $4000 - $5500 and supported 256 colors (from a palette of 16 million) on a 640 x 480 display, using a video card that was sold separately for a couple thousand bucks. However, that was at a time when a $3000 DOS PCs only supported EGA, with 16 colors (from a palette of only 64) at 640x350 (below, in Windows 2.0, 1987). IBM's new PS/2 line introduced the new VGA PC standard that year, which offered a 640 Ã 480 display with 16 colors or 320 Ã 200 with 256 colors, but those high end desktop PCs cost $6000 - $11,000. Macs weren't expensive; expensive hardware was.
Analysts with selective memory frequently forget that Macs didn't actually cost more back then, they just delivered a lot more in terms of hardware, large related to graphics. Similarly, NeXT's $6500 Cube wasn't comparable to $6,500 PCs of the day; there simply wasn't anything that even approached it. Color graphics cost so much that NeXT initially released its first computers using 4-bit color, providing 16 shades of grey. This gave it a distinctive monochrome display, and allowed it to dedicate its video RAM to provide a larger display resolution, commonly 1120 x 832. Jobs announced he wasn't going to do color until the technology was available to do it right.
Hardware Sales Push Sophisticated Software
Apple's profitable hardware sales to high end creative users enabled it to invest heavily in new software development. in 1989, Apple released 32-bit QuickDraw, which delivered modern 32-bit (16 million color) graphics. 24 bits were devoted to color information, and 8 bits were reserved for an alpha channel, which defined translucency. Using 'alpha bits,' graphics could spell out compositions that layered independent images over each other without fully obscuring the lower layers.
Apple's 32-bit QuickDraw also standardized the mess of incompatible graphics hardware being offered at the time; it simply drove incompatible graphics cards off the Mac market. The PC market continued to labor under incompatible graphics hardware, application software, and limited operating system support. This was one aspect of why it took Microsoft ten years to duplicate the Mac desktop on the PC; it had to account for such oddities as non-square pixels and incompatibilities between vendors all trying to sell the cheapest hardware possible.
In 1991, just as Microsoft was making headway with its first commercially viable version of a graphical desktop with Windows 3.0, Apple's Bruce Leak demonstrated QuickTime 1.0 as an upcoming new component for Mac System 6, raising the bar tremendously. QuickTime wowed the audience by playing back the "1984" commercial on a Mac at a time when Windows PC makers struggled to standardize on simple audio playback. Later that year, Microsoft announced the Multimedia PC initiative to deliver a standardized PC platform and shipped its Multimedia Extensions for Windows 3.0 to provide support for standard sound cards and CD-ROM players.
Critics were quick to deride QuickTime as a 'jerky postage stamp video', but the technology behind QuickTime was more than just a cutting edge movie player running on hardware that wasn't quite to the task; it was an architecture for playing anything time related. Quicktime acted as an operating system for time-based media. It provided:
- a container file for holding independent tracks of audio, video, text, metadata, timecodes, and other temporal information.
- system routines for displaying, capturing and editing media tracks while keeping them all in sync.
- a codec plugin architecture for adding support for new file formats.
- tools to magically translate content between codec formats.
- a development environment for creating audio and video productions, controlling the playback interface, and orchestrating interaction between different types of media.
Desktop Video Publishing
The original Macintosh had found its niche in print publishing. Its graphical interface allowed it to do things commodity DOS PCs simply couldn't. As Microsoft began selling Windows 3.0 in 1990, it signaled its intention to not only move its own Mac Office applications to Windows, but also influence Adobe and other Mac developers to migrate their print publishing software to Windows, leaving Apple without any differentiation in the market place, and stuck only with the extreme high end of the market. At the same time, Microsoft told DOS developers to port their apps to OS/2, ensuring that the majority of first generation apps on Windows would be transplants from the Mac, and primarily Microsoft's, as noted in How Microsoft Got Its Office Monopoly.
Apple had invested in QuickTime in order to retain its competitive advantage in the future. Like Adobe's PostScript for print, QuickTime unlocked the potential for an entirely new class of multimedia applications. Apple released QuickTime 1.0 (below) for Mac System 6 in December of 1991; in 1992, Apple announced the development of QuickTime for Windows to extend its reach beyond the Macintosh. On Windows, QuickTime would be playback only, maintaining media development on the Mac platform.
It delivered QuickTime 2.0 (below) first on the Mac and then on Windows in 1994. Since QuickTime was such an integrated component of the Mac System Software, the Windows version had to include a direct port of a lot of the internal Macintosh Toolbox code. It consequently largely bypassed Windows to talk to the video hardware directly, delivering impressive playback performance on both platforms.
The QuickTime Canyon Scandal
Concerned about Apple's encroachment upon the PC market it intended to control itself, Microsoft released a competing product called Video for Windows at the end of the following year. It couldn't match QuickTime's performance because Windows had never been designed to work with media; VfW was also constrained by the incompatibilities and inconsistencies of PC hardware, just as GDI— the Windows drawing routines modeled after the Mac's QuickDraw— had been over the previous half decade.
When Microsoft requested a free license for QuickTime for Windows in 1993, Apple refused; the two companies were still locked in a court battle over Microsoft's appropriation of the Mac desktop, and Apple didn't want to get its second wind knocked out of it.
Intel, hoping to accelerate the development of media work on commodity PCs— and fully aware that Apple had contracted with San Francisco Canyon to port portions of its QuickTime technology to Windows— approached Canyon to develop a video driver that would provide Microsoft's VfW with similar performance to QuickTime. While Intel knew that Canyon possessed Apple's code, it did not specify that Canyon needed to do clean room development, and gave the company an unrealistically short timeframe to develop the new code. Canyon simply delivered Apple's code to Intel, which then licensed it to Microsoft.
When Video for Windows suddenly improved in 1994, Apple investigated and found that Microsoft had simply stolen code from QuickTime in order to compete with QuickTime. Apple sued Microsoft and won an injunction that stopped it from distributing portions of the stolen code. The case was eventually resolved as part of the 1997 agreement between the two companies, detailed in Mac Office, $150 Million, and the Story Nobody Covered.
On page 2 of 4: Expansion of the QuickTime Parachute; QuickTime Reborn; Microsoft Goes On the Offensive; and The Tables Turn for QuickTime.
As part of its cross platform strategy for QuickTime, Apple continued porting parts of the Mac Toolbox used by QuickTime to run on other hardware architectures, including Silicon Graphic's MIPS-based IRIX graphics workstations. The result of this portability effort came to be referred to as the QuickTime Media Layer, a portfolio of technology Apple could depend upon if it had to jump off the Mac platform.
Among the additions to QuickTime Apple made in the mid 90s was QuickTime VR, for creating immersive, interactive object and panoramic videos. While Apple struggled to maintain its QuickTime portfolio as its refuge against the increasing tidal wave of commodity PCs running Windows, Microsoft announced additional plans of its own for washing over Apple's remaining territory. In mid 1996, it floated Active Movie as a cross platform solution identical to QuickTime in features and announced Surround Video as a competitor to QuickTime VR.
Fortunately for Apple, Surround Video never went anywhere and Active Movie was never delivered cross platform. Instead, it was relegated to being another Windows movie playback system to replace VfW, and was later folded into the Direct X package under the name Direct Show. Apple had problems of its own, however. The fiasco of Copland and related development failures had contributed toward an exodus of engineering talent. Bruce Leak, who pioneered QuickTime development at Apple, left in 1995 to help start WebTV, which was made up largely of employees from Apple and its General Magic spin off.
As Apple bottomed out in 1996, it began talks with NeXT. Among other features of the NeXTSTEP operating system, Jobs demonstrated NEXTIME (below), its own media development tool. Like everything else at NeXT, it benefitted from sitting upon a modern Unix foundation and being tightly integrated into the advanced AppKit development frameworks. This was particularly compelling for Apple because it has been centering its future strategies around QuickTime; migrating the two companies together seemed like the perfect fit because NeXT supplied strong replacements for all the pieces that were weak on the Mac, and Apple could provide a market for NeXT's largely overlooked software technology.
As Apple filled out its strategy for merging NeXTSTEP with its own portfolio of assets, including QuickTime, it restarted media development efforts that had fallen off into limbo. Among them were QuickTime 3.0, which had acquired the baggage of QuickTime Interactive, an extensive hypermedia development system based upon HyperCard.
Jobs canceled a wide variety of projects at Apple that fell out of the scope of its newly refined mission to get back on track and begin innovating again. Among them were QTi. Instead, efforts were devoted to pare down QuickTime 3.0 (below, as included in the Rhapsody DR2 release) into a practical, shippable product that would throw Apple back into relevancy in the media development race.
While earlier versions of QuickTime had tried to nurture a market for multimedia creations deployed on CD-ROM— something QTi and HyperCard were oriented toward— by 1997 it was obvious that the real market for media development was the Internet. Macromedia's newly acquired Flash and RealNetworks' RealAudio and RealVideo streaming products were being aimed at that online market. Having been late to identify the importance of the web, Microsoft was also now gunning toward dominating Internet media, too.
Microsoft Goes On the Offensive
In early 1997, Microsoft acquired WebTV and began retooling the company and its products to fit its Windows-centric strategy, as noted in the article Windows XP Media Center Edition vs Apple TV. Microsoft also notified Apple that it would have to cancel the development of QuickTime playback on Windows. Microsoft's Christopher Phillips famously told QuickTime manager Peter Hoddie, "we want you to knife the baby."
After Apple repeatedly refused to drop QuickTime for Windows, Microsoft made it clear that if Apple did not hand over media playback to Microsoft, the company would throw its weight into developing its own authoring tools and simply obliterate Apple. Deposition testimony in the Microsoft Monopoly Trial reported that Eric Engstrom, Microsoft's manager of multimedia technology, threatened that âif necessary, Microsoft would assign 150 engineers to an authoring development project in order to displace Apple from that market.â
At that time, Apple's entire QuickTime group only had around 100 engineers. Engstrom reported that Bill Gates was not interested in an authoring program because the market for was too small. However, he assured Apple that âif Microsoft needed to make an investment in providing authoring tools in order to push Apple out of the playback market, then the company would devote all the necessary resources to accomplish that goal.â
âWe're going to compete fiercely on multimedia playback, and we won't let anybody [else] have playback in Windows. We consider that part of the operating system, so you're going to have to give up multimedia playback on Windows,â Engstrom repeated in a phone conversation, as noted in Microsoft's Plot to Kill QuickTime.
Microsoft subsequently jumped on Avid Cinema, a new consumer video editing product based on QuickTime that long time Apple partner Avid was preparing to release. The company told Avid engineers, "You need to rip QuickTime out of your product if you want to be in this channel." It had deepened its involvement with Avid in a 1998 deal that resulted in Microsoft owning 9.1% stake in the company. A few months later, Avid announced its partnership with Microsoft in the company's new Advanced Authoring Format for multimedia authoring. That signaled a direct assault on Apple's QuickTime, as promised.
Microsoft intended AAF (authoring) and ASF (streaming) to become the center of its new Windows Media architecture, and simply obliterate QuickTime by buying up the most visible user of QuickTime next to Adobe's Premier video editing product. That strategy ended up backfiring.
The Tables Turn for QuickTime
In early 1998, the ISO's Motion Pictures Expert Group announced its choice to base the container format for the upcoming MPEG-4 standard upon Apple's QuickTime rather than Microsoft's ASF, citing the maturity of Apple's technology over Microsoft's brand new and unproven architecture.
Further, while Adobe was making Avid-like moves to distance itself from QuickTime in Premier, Randy Ubillos and the development team behind Premier had left the company for Macromedia in 1995, hoping to deliver a new professional level video editor based on QuickTime that went beyond what Adobe had in mind for Premiere. However, by 1998 Macromedia had focused its attention on Flash, and was ready to get rid of Ubillos' Key Grip project, which had been renamed Final Cut.
Microsoft's efforts to strip QuickTime support from Avid had put Apple on the defensive; it agreed to buy Final Cut from Macromedia, initially just to keep the project alive. It subsequently realized the best way to make sure that significant QuickTime applications remained intact was to maintain them itself, as noted in How Microsoft Pushed QuickTime's Final Cut.
That year, Apple also released QuickTime 3.0, which delivered full authoring support for Windows as a direct challenge to Microsoft's AAF and Windows Media initiatives.
On page 3 of 4: A New Trajectory for QuickTime; New Applications for QuickTime; New QuickTime Hardware; and Mac OS X 10.4 Tiger & QuickTime 7.
The next year in 1999, Apple delivered Final Cut Pro (previous page) running on the new PowerMac G3 with FireWire DV support. The new product served as a low cost demonstration of the power of QuickTime, and offered to do a lot of the work that previously required access to an expensive Avid studio. Apple also released the new iMovie as a consumer version of its QuickTime editor to replace Avid Cinema.
By the end of the year, it had also released QuickTime 4.0— which added support for Internet streaming using standard, open protocols and even offered its QuickTime Streaming Server as an open source project. QuickTime 4.0's new player (below) departed from the standard Mac user interface guidelines in an attempt to mimic a real device, with a rotary volume control, multiple slide out panels— similar to the drawers that would later show up in Mac OS X— and the brushed metal appearance that debuted in Final Cut Pro. It gained vocal critics for doing so.
In 2000, Apple presented Mac OS X Developer Preview 3, which incorporated a brushed metal appearance within the new Aqua for QuickTime Player (below). Compared to the bright white stripes on standard windows, brushed metal looked restrained and sophisticated. Other user interface gadgetry of the Mac OS 9 version were left behind.
Starting with the 2000 Mac OS X Public Beta, the revised Aqua-Metal QuickTime Player (below) incorporated QuickTime TV channels for accessing content from Apple's partners in an effort to leverage QuickTime's popularity for movie playback to enter the streaming market. Phil Schiller, who Apple had brought in from Macromedia to serve as its VP of Worldwide Product Marketing, called QuickTime TV "the fastest growing network of live and on-demand streaming audio and video."
New Applications for QuickTime
The work Apple had invested in porting the QuickTime Media Layer to other platforms also paid off, as Apple was able to use it as the foundation for porting the System 7 Mac APIs to run on its new Unix-based operating system from NeXT. Called Carbon, it allowed developers to get their existing applications prepared to run on Mac OS X with much less effort than they'd feared.
As Apple struggled to convince third party developers that the Mac platform would be worth investing in, the outstanding success of Final Cut Pro demonstrated that the company would be well served by acquiring and build its own applications. Apple subsequently bought DVDirector, its development team, and the related DVD technology portfolio from Astarte GmBH in 2000, and released the product the next year as DVD Studio Pro to support an emerging new market of pro, freelance, and small corporate film studios producing their own DVDs.
In 2002, Apple acquired Nothing Real for Shake, its high end video compositing software, and then bought Emagic, a German software developer of the Logic professional level music studio tools. It also began rapidly assembling a consumer suite of QuickTime-based applications, adding iTunes and iDVD to iMovie in 2001, and iPhoto the following year.
By the release of Mac OS X 10.2 Jaguar, Apple had professional and consumer suites of QuickTime-based applications to show off the advantages of the new Mac platform. It continued with GarageBand, Motion, and Aperture, and delivered regular updates to its existing applications over the same period, as detailed in Why Apple Bounced Back.
New QuickTime Hardware
It also worked to rapidly evolve QuickTime to support new hardware products. QuickTime 5.0 shipped in 2001, just three months after Apple introduced iTunes 1.0 (below, for Mac OS 9). Apple had acquired the popular SoundJam MP and designed an entirely new interface to create iTunes, using the brushed metal appearance of iMovie and QuickTime Player.
In October of that year, Apple unveiled the first iPod along with support provided by iTunes 2.0 for both Mac OS 9 and Mac OS X 10.1. The new version added features including an equalizer and crossfade, but retained an identical interface.
The next year, Apple incorporated support for MPEG-4 part 2 and AAC audio in QuickTime 6.0, and in 2003, iTunes 4.0 supplied pioneering support for AAC encoding and playback, which enabled the new iTunes Music Store. That subsequently pushed the rapid adoption of the new AAC standard and maintained the availability of commercial music for the Mac at a time when Sony and Microsoft were promoting digital downloads that only worked with Windows, as described in the article Rise of the iTunes Killers Myth. Six months later, iTunes for Windows and the new Mac OS X 10.3 Panther were released in October.
Mac OS X 10.4 Tiger & QuickTime 7
The popularity of the iPod, fueled in part by the underlying power of QuickTime in iTunes, helped to fund further investment in software that resulted in Mac OS X 10.4 Tiger's QuickTime 7 in 2005. It delivered support for MPEG-4 part 10 (H.264, or AVC), which would subsequently enable hardware-based video decoding in the new video iPods offered later that year and launch Apple into the TV and Movie business with the iTunes Store.
Because it dropped support for the Classic Mac OS, QuickTime 7 could also begin using the native Quartz engine (Core Graphics) for screen drawing, which allowed for live resizing of windows during playback. Video was painted on the screen as an OpenGL surface. QuickTime 7 also introduced Quartz Composer animations and could support Core Image filters on live video via Core Video. The old Mac Sound Manager was also replaced with the new Core Audio, for high resolution, 24-bit sound. Apple also introduced the QTKit, a Cocoa framework for QuickTime development.
After the release of Tiger, iTunes gained video and podcasting support, and was at version 6 by the end of the year, skipping thorough just four months of 5.x after two years of 4.x. Version 5.0 had introduced the new unified look similar to Leopard, which has only been moderately tweaked since, two major version numbers later. It appears iTunes was the Guinea Pig for experimenting with interface elements; iTunes 7.0 added Cover Flow last year, for example. It also debuted iPod Games, which while having little to do with Leopard, appears to similarly be testing out the model for secure distribution of software on mobile devices, and may likely be used as the official way to install software on the iPhone next spring, as the article Steve Jobs Ends iPhone SDK Panic noted.
On page 4 of 4: New QuickTime Features in Leopard; Quicktime Applications in Leopard; QuickTime Streaming Server; and HD-DVD and Blu-Ray.
Apple earlier announced that Mac OS X Leopard would support the Open GL 2.1 specification and provide "a dramatic increase in OpenGL performance by offloading CPU-based processing onto another thread which can then run on a separate CPU core feeding the GPU. This can increase, or in some cases, even double the performance of OpenGL-based applications." This directly benefits QuickTime, because everything it draws is an Open GL surface.
Leopard also offers significant enhancements in H.264 encoding and provides support for alpha layer transparency in H.264 video playback. The new QTKit Capture features makes it easy for developers to add video recording support to their applications using an attached iSight, USB (VDC), or FireWire (IIDC) device such as a DV camcorder. Apple uses this itself in the new Picture Taker Panel and in Photo Booth, which not only takes snapshots with effects (and four shot multi-strip photos, below), but also now captures video, which it can save as a typical movie or export as an animated GIF for use as a buddy icon. Photo Booth also does the same background effects of iChat, described in Road to Mac OS X Leopard: iChat 4.0.
iChat Theatre— used to perform document sharing from iPhoto or Keynote as part of a video conference— is also a feature powered by QuickTime. Third party applications can integrate with iChat using the Instant Messaging framework and Core Video and Core Audio APIs; the Instant Messaging framework compress their content using H.264 and send it across the network as video.
While not a part of QuickTime itself, Leopard's Image Kit is a new Cocoa framework powered by Core Image and Core Animation. It provides powerful imaging services for developers for finding, browsing, and viewing images in a collection, previewing and setting Core Image filters on an image, as well as rearranging and rotating images in a group view. An example of this functionality is provided in Preview, detailed in Road to Mac OS X Leopard: an extensive look at Preview 4.0.
Quicktime Applications in Leopard
The new QuickTime player tones down the brushed metal using a standard Leopard window appearance, but retains the rounded panel chin of its predecessors. However, Apple has positioned iTunes as the general purpose media playback client for the system, and has abandoned the idea of QuickTime TV to instead focus on the Internet radio, podcast, and video store support within iTunes. In addition to iTunes, the new Front Row provides an alternative media player interface, which is now identical in layout to the Apple TV, apart from not providing any special support for playing YouTube videos.
And of course, videos can also be played directly in the Finder within Cover Flow (above), by mousing over the preview and clicking the play icon that appears, or by hitting the spacebar to bring up Quick Look. Related features of the Finder were detailed in Road to Mac OS X Leopard: Finder 10.5.
QuickTime Streaming Server
In Leopard Server, QuickTime Streaming Server is now a 64-bit application, and provides support for Icecast-compatible MP3 audio streaming, native MPEG-4 and 3GPP (mobile device) streaming, support for Instant-On video streaming that lets users start watching faster without all the buffering, and even scrub back and forward within an on-demand stream and still resume immediate playback updates.
It also now supports "streaming thousands of simultaneous movies encoded at 1280x720, 24p, 5mbps from a single Xserve," streaming live or on-demand video using H.264 compression, use of server-side playlists for streaming a set of media files as if it were a live broadcast, and relay support for high volume streaming using multiple servers.
HD-DVD and Blu-Ray
Users looking for HD disc DRM in Leopard might have to make do with Vista; it appears Leopard provides no built in support for either the leading Blu-Ray or the struggling HD-DVD format, at least not for commercial DVDs such as movies. The overhauled DVD Player in Leopard can play back non-encrypted HD-DVD discs created in Apple's own DVD Studio Pro, but that's largely because it uses the same file structure as regular DVDs, and hardly a feature for would-be HD movie watchers (and not new; Tiger's DVD Player can as well). The key sticky bit is a lack of DRM support for the new disc formats, which is understandable given Apple's strategy for online video through iTunes, and the fact that no Macs ship with HD disc players yet.
Macs already support both HD-DVD and Blu-Ray drive hardware; third party software could be used to play back DRM encrypted HD discs, and in the case of HD-DVD, unpack movies compressed with Microsoft's proprietary VC-1 codec. The barrier seems to be a lack of interest.
As noted in the article Origins of the Blu-ray vs HD-DVD War, only around 150,000 standalone HD disc players have sold on either side of the isle, which makes the Zune look like a brilliant success in comparison. Narrow down that market to users with Macs, and its obvious why no third parties have jumped at the opportunity to license the HD-DVD and Blu-Ray DRM and write a software player for the Mac; there's no market. Anyone with a 50" HDTV screen worthy of playing Blu-Ray can probably afford to buy a PlayStation 3 to watch their movies.
That should leave Mac users hoping for expanded options in iTunes, but there's yet no official word on HD downloads or movie rental options. Of course, those options are all unrelated to the launch of Leopard, as little or nothing related to iTunes ever appears in advanced developer releases before being launched publicly.
Check out earlier installments from AppleInsider's soon to conclude Road to Leopard Series: System Preferences, Parental Controls and Directory Services, What's new in Mac OS X Leopard Server, Dashboard, Spotlight and the Desktop, Safari 3.0, iCal 3.0, iChat 4.0, Mail 3.0, Time Machine; Spaces, Dock 1.6, Finder 10.5, Dictionary 2.0, and Preview 4.0.