Secunia issues contradictory vulnerability report assailing Apple

Danish security firm Secunia has issued a report graphically assailing Apple as having the "most security vulnerabilities" despite noting in its research that the "statistics provided should NOT be used to compare the overall security of products against one another."

Secunia's security vulnerability counts are frequently cited as proof of Apple's lax efforts in maintaining the security of its products, but as the company notes in its detailed statistics pages, "it is IMPORTANT to understand what the below comments mean when using the statistics, especially when using the statistics to compare the vulnerability aspects of different products."

This comparison is not a comparison

The company threw its warnings out the window when publishing its second half year report for 2010, including a graph (below) that tracks Apple against ten other vendors of software products that offer little or nothing in common with the Mac maker, including network gear vendor Cisco, printer maker and PC assembler HP, and web-centric developer Google, in terms of documented software vulnerabilities.

Secuina explicitly stated in its report that the "graph is not an indication of the individual vendors’ security, as it is not possible to compare the vendors based on number of vulnerabilities alone," but that didn't stop the report from generating such sensational headlines as "Apple the new world leader in software insecurity" and "Apple wins software insecurity world cup."

A closer look at Secunia's numbers shows a bizarre calculation of vulnerability numbers that appears intentionally designed to mislead, despite the company's stern warnings not to misuse the data.

How Secunia counts vulnerabilities

Not all vulnerabilities are equal: Secunia outlines five levels of criticality ranging from minor "not critical" issues to "extremely critical" problems that can result in remote exploits without any interaction from the user, and for which active exploits are already known to exist. Yet Secunia's vulnerability report totals throw all these various types of flaws together into sums that are frequently used for meaningless comparison purposes.

Such vulnerability totals grant "weaknesses with a very limited security impact" the same weight as severe issues that actually pose a meaningful and readily exploitable risk. Secunia notes that the vast majority of the vulnerabilities it catalogs refer to moderate or low risk problems, while less than 0.2% of its issues are rated as extremely critical.

That makes any discussion about the volume of vulnerabilities disingenuous, as it is really only the very few serious problems that pose an immediate threat, while the scores of other potential weaknesses that exist have very little real impact on users, according to Secunia's own research.

There's also no real comparison of the number of reported flaws that have been patched compared to the live threats still in existence. In fact, many of Secunia's advisories pertain to flaws that vendors announce after having found and release a fix, making any count of these patched vulnerabilities irrelevant to anyone who updates their software regularly.

Vulnerabilities by product name, not code base

Secunia's vulnerability counts reset when Microsoft changes the name of its product, but continue to accumulate for Apple because the company hasn't rebranded Mac OS X since 2003, when Secunia began keeping track. Browsing Secunia's database, it appears Mac OS X has suffered from hundreds of vulnerabilities while Microsoft's Windows has racked up far fewer, but that's only because Microsoft's regular rebranding efforts reset Secunia's clocks.

At the same time, Secunia does not break up Apple's vulnerability counts by each reference release of Mac OS X, so its current vulnerability listings date back through Jaguar, Panther, Tiger, and Leopard, as well as the currently installed base of Snow Leopard.

How Secunia arrives at its totals are also puzzling, as according to its own statistics Apple's Mac OS X was affected by 6 "advisories" in 2010, only one of which has not yet been patched. That issue is rated as "not critical" and can only be exploited by local users.

In contrast, Secunia outlines 17 advisories for Windows 7 in 2010, with one that has not yet been patched. That flaw is listed as slightly more serious, but is still described as "less critical."

Last year, Secunia issued 12 advisories for Mac OS X, two of which are listed as unpatched. One is the 2010 vulnerability, the other wasn't carried forward for some reason. Secunia's advisories may include multiple vulnerabilities, making it difficult to track outstanding problems that have not yet been patched among those that have.

For 2009, Windows 7 got just 4 advisories, but Microsoft was only selling it for a few months. For most of the year, it was selling Windows Vista, which had 28 advisories. Vista also contributed 21 additional advisories in 2010, bringing the combined Windows Vista/7 count to 36 vs Apple's 6 for all versions of Mac OS X. An additional 30 new advisories were also attached to Windows XP in 2010.

But again, these numbers are almost completely meaningless because Secunia's "advisories" are in almost every case only reporting that the vendor has reported a patch. Secunia catalogs Apple's security updates and then counts the vulnerabilities its addresses after they've been fixed, in addition to a very small number of reported but not patched flaws. But Secunia isn't comparing outstanding, unlatched flaws; it's primarily calling attention to the flaws Apple has already publicly fixed.

That means Apple would have a better security profile, according to Secunia, if it either failed to do anything to secure its products or if it lied about the number of flaws it was actually patching, and didn't credit security researchers with having discovered the flaws they have. Apple is also penalized for using open source software for which its easier to find new flaws.

Vulnerabilities in bundled software

Yet even after establishing Apple as the leader in vulnerability counts, Secunia offers contradictory data in specific markets. For example, among web browsers, Secunia says Mozilla's Firefox, Google's Chrome, and Microsoft's Internet Explorer were ahead of Apple's Safari in terms of vulnerability "events," ("the number of administrative actions needed to assess or maintain software," or the number of different update patches the vendor delivered that users will need to apply).

Firefox also ranked first in terms of CVE ("Common Vulnerabilities and Exposures," or uniquely identified vulnerabilities) with 96 flaws reported between June 2009 and June 2010. There were 84 CVEs reported for Safari, 70 for Chrome, and 49 for Internet Explorer, according to Secunia. Again, these reports are almost entirely culled from the patches vendors release themselves, rather than being outstanding known flaws that need some sort of fix.

These flaw numbers are also not qualified by criticality, nor is there any consideration given of what version real users actually have installed. The majority of web users are still using outdated versions of Internet Explorer, while web stats show Apple's Safari users are more likely to be using the latest and most adequately patched version. That reality is all lost in simplistic vulnerability counting.

It is indisputable that the largest number of security problems affecting users is related to Windows users with inadequately updated software. Yet according to Secunia's numbers, one would get the impression that Apple's users are under greater threat from real security exploits because Secunia has counted up more vulnerabilities from combing through Apple's security updates and cataloging flaws that have already been fixed.

Plugin vulnerabilities

Also important to consider is the fact that the core software is rarely where the most or most dangerous exploits actually occur. Security expert Charlie Miller, when asked at CanSecWest which browser is safest, replied, "there probably isn’t enough difference between the browsers to get worked up about. The main thing is not to install [Adobe] Flash!"

According to Secunia's data, Adobe's Flash Player plugin suffered 51 CVEs, while the company's Acrobat and PDF Reader each added another 69. Oracle's Sun Java JRE added another 70 CVEs, making Java and Flash plugins together far greater in documented security flaws than the entire web browser, regardless of the browser used. Yet according to Miller, installing and using Flash was far more dangerous than simply brewing the web itself, despite its having fewer documented CVEs.

However, once again Secunia is not classifying these issues as patched or not, or as serious or not. Secunia is simply reporting which flaws have been reported and cataloged during the past year. There's also no way to know if any these products have serious flaws that are yet undiscovered and therefore hiding serious flaws that greatly outweigh the known minor issues, most of which have already been patched and pose no issue to users if they keep their systems up to date.

That alone makes Apple's refusal to add Java and Flash support to iOS a no brainer. Conversely however, Apple is dinged by Secunia for bundling Adobe and Oracle's software in Mac OS X. In fact, a large number of the vulnerabilities attributed to Mac OS X are actually related to patches Apple distributes to fix know issues in Flash and Java.

From its database, 17% of the advisories its issued for Mac OS X in 2010 were related to Java. None of the advisories listed for Windows 7 address issues in Java or Flash, apparently only because Microsoft doesn't distribute those patches itself.