Road to Mac OS X Snow Leopard: 64-bit security

Over the last eight years, Mac OS X has enjoyed a tranquil existence in stark contrast to the high profile security threats and attacks suffered by mainstream Windows users. Microsoft's monopoly over the PC world has long meant that anyone interested in creating extensive, easily spread damage through software exploits would focus their efforts on Windows.

Microsoft's Security Efforts

Since Microsoft's OS was originally developed primarily for business computers sitting together in a trusted LAN environment, it was not only easy to exploit software flaws in the system but also trivial to find ways to fool the system into forwarding viral payloads to other systems. Once exposed to the open Internet, Windows didn't stand a chance.

Reinforcing the Windows PC to survive the onslaught of malicious exploits saturating the Internet would be a complex and expensive task, one Microsoft did not immediately recognize as a priority. However, once Windows started gaining a reputation for lax security after falling victim to a series of famous exploits in the late 90s and into the beginning of the current decade, Microsoft began reevaluating its priorities.

Longhorn, which was intended as a close successor to 2001's Windows XP, ended up being pushed off as the company was forced to initiate a major new effort to solve the outstanding security issues in XP. Toward the end of 2004, Microsoft shipped XP SP2, the product of extensive work within the company using code scanning, auditing, testing, and fundamental feature and architectural reviews, in addition to external source code auditing and penetration testing.

In addition to Microsoft's efforts to identify and patch flaws and vulnerabilities in its software, the company also initiated measures to make unknown vulnerabilities more difficult for attackers to find and exploit. This included having a firewall installed by default and requiring that RPC servers authenticate communications, so that remote attackers would need to present valid credentials before ever being given access to anything that might be attacked to allow entry.

Microsoft is now very public about its security efforts, and takes every opportunity to tout its recent security work as a defense against any criticisms of its past mistakes in taking a less than serious approach to security.

Apple's Security Efforts

In contrast, Apple has never experienced a security crisis related to Mac OS X. Virus writers have nearly zero financial motivation to create new attacks from scratch that target Macs. The theoretical potential of "hackers" attacking Macs for fame and glory, as imagined by pundits with a bias against Apple, has simply failed to materialize over the last half decade, despite their insistence that the threat is so alarmingly close as to be palpable.

Even in cases where exploits have been found or artificial attack installers have been designed, viral outbreaks haven't occurred because installations of Macs aren't ubiquitous enough to sustain the critical mass required for an acute network infection. Add in the fact that Mac OS X wasn't dragging along the same legacy of promiscuous LAN origins as Windows, and you have a series of factors that combined to give Apple a pass from focusing on security retrofitting in crisis mode.

Instead, Apple has had the luxury of planning Mac OS X releases to roll out security features incrementally. As with its other plans for feature enhancements in Mac OS X, the company has remained tight lipped on many of its security efforts. There's evidence the company has performed code security scanning, as simple buffer overflows have been cleaned out of many system libraries, according to a security expert familiar with the history of the OS.

Mac OS X 10.4 Tiger eliminated most of the easy local buffer overflows, while 10.5 Leopard has expanded upon that to remove many of them from remotely accessible network services. Leopard also incorporates stack protection, library randomization, a non-executable

stack, and sandboxing for some system processes. These features are incremental improvements in security that will be expanded upon in Snow Leopard.

Mac OS X's sandboxing is provided by the Mandatory Access Control (MAC) framework, an implementation of the MAC framework from TrustedBSD. Sandboxing imposes permission controls on processes that can, for example, limit them from connecting to a network, from writing any files, or from writing any files outside of specific directories. While sandboxing doesn't prevent a process from being attacked, it does limit the amount of damage malicious attackers can cause once they gain control of a sandboxed application.

On the iPhone, sandboxing is used to restrict each application from accessing anything outside of its own data files and preferences. Even apps that have access to the public networking APIs are restricted from direct access to the communications or networking hardware.

On page 2 of 2: Security in 64-bit Snow Leopard; and Security before it's needed.

Security in 64-bit Snow Leopard

In addition to expanded sandboxing, the move to 64-bit computing will provide a series of other benefits related to security. Apple's 64-bit binaries set all writable memory as non-executable by default, including thread stacks, the heap, and any other writable data segments.

This is already present to an extent in today's Leopard Server, which runs some services, such as the Apache web server, as 64-bit processes. Using the vmmap command reveals that no memory allocated by these 64-bit apps is both writable and executable. On 32-bit Intel systems, while no memory is marked as both writable and executable, the legacy x86 processor design does not enforce the permissions bits, but 64-bit CPUs do. This feature prevents exploits from injecting malicious executable code into memory and tricking the app to run it as it if were its own instructions.

Another security weakness in the x86 architecture solved in the move to 64-bits is the use of registers for function call arguments. This makes exploits using return-into-libc techniques much more difficult. On 32-bit x86, function arguments are passed directly on the stack, so when an attacker has overwritten the stack segment, they can completely control the arguments passed to a function that they cause the compromised program to "return into," according to a security researcher.

The move to 64-bits also greatly enhances the Address Space Layout Randomization (ASLR) techniques used to secure Leopard. Currently, 32-bit binaries are restricted to a relatively small 4GB allocation, making it easier to predict useful addresses for malicious code to target. Additionally, Leopard keeps dyld, Mac OS X's dynamic loader (responsible for loading all of the frameworks, dylibs, and bundles needed by a process) in the same known location, making it relatively trivial to bypass the existing ASLR.

With the much larger address space available to 64-bit binaries, Snow Leopard's ASLR will make it possible to hide the location of loaded code like a needle in a haystack, thwarting the efforts of malicious attackers to maintain predictable targets for controlling the code and data loaded into memory. Without knowing what addresses to target, the "vast majority of these exploits will fail," the security expert explained.

Security before it's needed

Apple's sheltered existence in isolation from regular malware attacks puts it in the enviable position of being able to focus on building security features proactively, rather than in response to ongoing, embarrassing exploits. For Mac users, that means the window of opportunity for malware exploits is being closed off before circumstances change enough for the platform to become a viable target.

The company is being relatively quiet about its security efforts because it doesn't want to be directly compared against Microsoft, which is ahead in some security areas, at least in its latest software releases. However, Microsoft's installed base of the billion PCs running Windows worldwide is not protected by advancements in the latest releases because relatively few users have upgraded to the latest releases.

That give Apple a strong position in maintaining its security halo because the Windows PC world is so rife with low hanging fruit for malicious attackers that the Mac platform remains an undesirable target. That leaves disgruntled pundits with nothing to complain about outside of misleading vulnerability counts. So while PC users contend with the constant din of security issues and performance sapping layers of security software, Mac users are free to just enjoy the silence.

Road to Mac OS X Snow Leopard: 64-bit security is the fifth installment in AppleInsider's ongoing Road to Mac OS Snow Leopard series. Previous installments are listed below in the order they were published.

Road to Mac OS X Snow Leopard: 64-Bits

Road to Mac OS X Snow Leopard: 64-bits, Santa Rosa, and more

Road to Snow Leopard: twice the RAM, half the price, 64-bits

Road to Mac OS X Snow Leopard: the future of 64-bit apps