Brian Mastenbrook: A Guide to Finding Security Problems on OS X

Note: this is recapped from a conversation I had with Landon Fuller about how to find security vulnerabilities on OS X.

I use Mac OS X, Windows, and Linux on various different laptop and desktop computers, but most of my time is spent on Mac OS X owing mostly to some Mac-specific programs I frequently use. This doesn't bother me as OS X is a very pleasant environment to spend my time in. However, I am distressed by the security weaknesses of OS X as a desktop computing environment, especially with how poorly Apple seems to handle these issues.

I've reported a number of issues to Apple so far. Generally I can find an issue within a few hours of searching for a problem, and most of the problems I've found are design flaws which should have been caught with even a cursory security-focused review. However, I've very little time to spend on this. While I'd prefer that Apple allocate more resources in this direction, it would certainly help if more people were searching for issues to report.

With that in mind, here are some focus areas for those who would be interested in finding issues in desktop OS X. I'm sure there are hot spots in OS X Server as well, but I'm not generally too interested in that area as OS X is still very uncommon as a server operating system and many of the issues would be shared with other UNIX-like operating systems. Because the common open source server applications have already received a great amount of attention, there's a lot less low-hanging fruit than in the area of desktop security.

There are two major areas of desktop OS X security which I believe deserve major focus. One is network-based attacks that can make use of applications like Safari to execute arbitrary code. The other is in escalation-of-privileges vulnerabilities which can be combined with network-based attacks to turn a trojan or worm into an automatically-spreading virus.

In the area of network-based attacks, here are some focus areas I think should be investigated:

• Problems that allow an arbitrary file on the victim's computer to be opened via the Finder, combined with a file type containing executable content which is not specifically excluded by the quarantine blacklist. The quarantine file type blacklist is far from complete and will never be complete so long as it's an explicit exclusion list. You can find the list of excluded types and extensions at /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/System. Arbitrary files can be placed in predictable locations on the victim's file system using either disk images or remote filesystem URLs (afp, ftp, smb).
• URL scheme problems. Safari opens other applications via URL schemes without applications, and most of the system applications haven't received anything resembling a thorough scrubbing. When issues are fixed, they tend not to get fixed all the way. I've reported issues in the Help Viewer application three times already, and it's still not completely fixed. To see the list of registered URL handlers, install RCDefaultApp.
• Any problem involving a "safe" file type that Safari will automatically open after downloading. The list of "safe" file types is in the same file as the quarantine blacklist.
• Problems where non-SSL content from apple.com or other domains is assumed to be trusted, which can lead to a variety of problems. There's no such thing as a trusted network, especially given how common unsecured WiFi access points are. Even on a supposedly secure network, if a virus manages to gain root through an escalation-of-privileges problem, it can easily set itself up as a DNS server and router to intercept requests from other systems on the network. This area is particularly ill-explored in my opinion.

Escalation-of-privileges vulnerabilities haven't been an area of focus to me, mostly because it would likely take more than a few hours to find an issue here. Nonetheless I think there are some promising areas for these kinds of issues:

• Privileges problems. Historically, "donating" files to another user was not allowed on UNIX-like operating systems, but OS X users may create files and directories owned by another user via disk image mounting, and these files may be created in any folder that the user owns via union mounts. The system ignores devices in mounted disk images and the suid/setgid bits on files, but I'm still convinced there's at least one issue lurking here. One approach would be to find a way to make a process running as root move or copy a file outside the union mount, which would allow the suid bit to work on the root-owned file.
• Suid binaries shipped with the system have likely not been thoroughly validated. The ARDAgent problem demonstrates that there are large gaps in the process for reviewing what suid-root binaries are allowed to do. Finding a problem here would be tedious, but should be straightforward. For reference, the list of suid binaries installed can be obtained with find / -perm +4000 -print.
• Authorization Services. It's complicated and is designed to allow ordinary users to make settings changes like choosing the active WiFi SSID without having to authenticate multiple times. I already know that the authentication prompts can be hijacked silently by another process running as a user, but I'm convinced there's at least one issue lurking in one of the programs that uses Authorization Services that would allow a user process to gain root without waiting for a prompt (System Preferences being a likely culprit).

If you're going to search for issues, please practice responsible disclosure, either by reporting issues directly to Apple or through VeriSign's iDefense VCP or 3com's Zero Day Initiative.

Happy hunting!