BSDCan 2005 Remarks

I’ve just returned from the 2nd annual BSDCan conference in Ottawa, Ontario, organized by the very capable Dan Langille. In addition to being a super nice guy, Dan is also the founder of the FreeBSD Diary, FreshPorts, FreshSource and is involved with the BSD Certification project. Let nobody say that Dan hasn’t given enough to the BSD community!

In the spirit and style of the USENIX conference summaries published in ;login: magazine, I’m going to summarize (and pollute with my own personal remarks) the sessions that I attended.

Debugging Kernel Problems
Greg Lehey

Slides

This was a full-day session that I attended on Thursday 12 May, conducted by the very capable Greg Lehey. Greg started out by showing userland tools for investigation of kernel problems, such as iostat, vmstat, and the like. He then moved onto a review of hardware data structures, a topic that I haven’t paid much attention to since university (which wasn’t that long ago, so a lot of this was still fresh in my head).

Greg then demonstrated a number of techniques for live kernel debugging and inspection, including the use of DDB and how to invoke GDB over a FireWire connection to a remote DDB session. In his view this is much better than serial debugging, since the maximum speed of a serial line is 115,200 bps, and a remote serial session only works reliably at speeds of 9,600 bps or lower. Firewire also lets you inspect the target processor’s memory without the assistance of the target processor, a feature that, ironically (and correctly), Poul-Henning Kamp pointed out as a security flaw.

After lunch Greg demonstrated techniques for examining kernel dump files (and how to generate them, of course). I must admit that I was somewhat bored by this phase of the discussion; after thinking about it for a while I concluded that it would have been much more instructive had this section been more interactive, for instance, if Greg had supplied us with a known-faulty kernel built with debugging symbols, and a core file from said kernel.

The book that Greg is writing on this subject is likely to be very instructive, and the draft so far is quite good. However, a lot of the hacking about is still very much black magic and narrowing down your decision tree about what path to take while investigating a kernel problem is still likely to require a lot of background knowledge about the manner in which the FreeBSD kernel operates. In addition to reading the source code, Kirk McCusick and George Neville-Neil’s book (The Design and Implementation of the FreeBSD Operating System) probably provides as good an introduction as any from the theoretical perspective.

A new security issue
Colin Percival

PDF

Colin Percival used BSDCan as the forum for the initial public disclosure of a security issue affecting the HyperThreading (simultaneous multithreading) technology in certain Intel processors, although theoretically the problem can affect any processor permitting multiple threads to share the same cache.

In his presentation, Colin illustrated how cache access timing differences between a case when a cache line needs to be flushed versus a case where it does not can be used as a covert channel to communicate information between two threads. Specifically, if one thread A can determine the pattern in which another thread B is causing cache flushes, it may be possible to deduce the data that thread B is using. An critical example (selected for admitted shock value) is if thread B is OpenSSL, where the timing induced by a cache miss, for example, can be deduced to indicate a lookup table query. The conclusion is that it is possible to deduce approximately 310 bits of a 512-bit RSA exponent using the tactic. Colin went on to demonstrate that this is enough information leakage in order to factor the RSA modulus in polynomial time.

Colin proposed several solutions (aside from the obvious workaround which is to disable simultaneous multithreading, the subject of a FreeBSD security advisory). The solution he advocates as being the most feasible at this time, although none are trivial, is to create a cryptographic library whose execution path will be independent of the data passed in. This would render all such timing-based snooping attacks irrelevant. However, he readily admits that this will be a very difficult task (and in particular, he believes that retrofitting OpenSSL will be near-impossible), and is currently seeking sponsors that would allow him to embark on this new implementation.

FreeBSD Wireless Networking Support
Sam Leffler

Sam spoke at length about the work being added to FreeBSD’s wireless infrastructure in order to support such features as the WPA security protocol. Sam is one of the major contributors to Net80211, which is a device-independent 802.11a/b/g library being used by all the BSD systems for wireless networking.

One of the most interesting applications resulting from Sam’s work is the notion of virtualized wireless LAN devices sitting upon a physical device. This would permit the creation of “virtual access points” with different SSIDs. For example, the physical device ath0 (an Atheros-based wireless adapter) could have several virtual interfaces, say, wlan0 and wlan1, the former of which could be reserved for paying customers, and the latter of which could be for free customers. Each of these virtual interfaces could have its own 802.11 properties, including encryption methodology, port speed, or even firewall rules (outside the scope of Net80211 but trivially possible using packet-filtering).

One area of ambiguity I did not manage to resolve in my head is how the system would be able to resolve the MAC address clashes, i.e. each of these virtual interfaces would have the same MAC address as the physical device. If anyone can shed light on that, please feel free to comment on these remarks!

(Sam’s work has necessitated adding a lot of extra options to ifconfig(8), which caused Poul-Henning Kamp to later joke that Sam is on a mission to add the entire English dictionary to ifconfig. I still think it’s a better idea to overload ifconfig than to create dozens of userland tools for managing the network interfaces, like Linux does.)

The FreeBSD Package Cluster
Kris Kennaway

Kris talked about the software and hardware infrastructure needed to make a continuous build environment for compilation of FreeBSD ports, and creation of binary packages from successful compiles. Although Kris has some pretty serious hardware at his disposal (including a 12-CPU Sun E4500 server), he still uses a number of optimizations, many of them homegrown, to ensure that the compilation of 12,000+ ports completes within a reasonable amount of time.

A highlight of his talk was his admission that although the Package Cluster produces binary packages for FreeBSD 4.x, 5.x and 6.x, the base operating system is one flavour (5-STABLE at present). Kris creates chrooted environments for each port being compiled, and populates it with the userland for the target FreeBSD OS revision. However, the host OS never changes. Although he says that this violates “what we tell our users to do” (i.e. never run a different kernel from userland), this mostly works; he has to copy in proper binaries for kernel-dependent userland utilities like ps, mount, etc.

At the present time, the compilation of all 12,000+ packages takes a little over 24 hours. The package build system is integrated into or its output consumed by many other tools like fenner’s portsurvey.

SEBSD: Port of SELinux FLASK/Type Enforcement to FreeBSD
Chris Vance

Chris spent the first part of this talk describing the work already done by the TrustedBSD project to implement a Mandatory Access Control (MAC) framework,
permitting MAC implementations to be installed as kernel modules. One such implementation is the SEBSD port of NSA’s FLASK/SELinux over to this framework.

I’m not a security guru nor an expert on this kind of technology so I’m afraid to say that much of this talk went over my head. I see much of this work to be good, however, in making BSD an acceptable choice for organizations, such as the NSA itself, who have very stringent userland requirements that aren’t satisfied by the standard UNIX discretionary access control mechanisms that currently exist today.

Work In Progress Session
Robert Watson – chair

Robert Watson chaired this series of five-minute presentations about various works in progress centering around the BSD kernels, or using BSDs for various applications. Some of the ones I found more interesting:

• der Mouse (a/k/a Mike Parker) is writing an SSH implementation completely from scratch.
• Rob Krten is working on porting a QNX-based telephone answering/call recording/TV display (!!!) system to BSD.
• The TrustedBSD mac_trustedexec module was discussed. This should be able to prevent common attacks like LD_PRELOAD hijacking of shared libraries, among many other things, by using the extended attributes capabilities of TrustedBSD to store a checksum of binaries along with their inode. Trusted execution dependency trees are also possible.
• Scott Long talked about UFS journalling, which is work that is just beginning.

OpenBGPD
Henning Brauer

WWW

OpenBGPD is a full implementation of the Border Gateway Protocol, version 4, and grew out of dissatisfaction with existing open source software for doing BGP, namely GNU Zebra (which is now dead due to the fork to a commercial product) and Quagga, Zebra’s successor.

Just from the presentation, OpenBGPD appears to be very solid and well-written. It also includes many features that I have not seen elsewhere (particularly in Zebra) such as MD5-signing of BGP updates and IPSec with both static and dynamic keying.

This being an OpenBSD product, there was the usual ideological posturing and slagging of other operating systems and vendors (Henning referred to FreeBSD as “PartlyFreeBSD” and insulted Cisco at least ten times during his presentation). An audience member asked further about support for OpenBGP on FreeBSD and was told something along the lines of “maybe it works, but I don’t care if it does.” I continue to find it unfortunate that many OpenBSD developers’ Standard Operating Practice is to berate people — and then have the audacity to show slides asking for donations. That aside, I think the technical merits of this BGP implementation can outweigh the hardline ideological attitudes, which in my view are completely irrelevant to the technology and should not be expressed in a presentation of this nature.

ioctl is just sooooo 1980ies
Poul-Henning Kamp

Poul-Henning is one of my heroes: he can take a complicated concept and make it simple and amusing to boot. It took me at least three tries to explain the crux of this talk to my girlfriend, and I probably was nowhere near as amusing as Poul-Henning.

Poul-Henning began by describing the original design and semantics of ioctl(2) and illustrated how, over time, it’s become more of a "kitchen sink" tool for effectively passing arbitrary structured data into the kernel. The problem with passing data in this way (as binary structs) is that one encounters a proliferation of userland programs (vidcontrol(8), camcontrol(8), atacontrol(8), and a whole host of others) whose code performs nothing more complicated than to marshal the user’s input into a struct and shove it into the kernel by means of calling ioctl(). Another downside of this is that third-party programs that want to interface with the ATA subsystem cannot call the ioctl() directly unless they are willing to risk future ABI incompatability when/if the format of the struct changes. They must resort to invoking the "foocontrol(8)" program directly.

There are a number of other downsides to ioctl(), one of which is that the return values are often far too generic (device-unspecific) to be useful to the end user. Many a sysadmin has been puzzled by invoking a “foocontrol” type utility only to have it return nothing more useful than foocontrol: Invalid argument. He also noted that sysctl(8) is also a poor tool for the same problem.

Poul-Henning argues for doing away with ioctl by simply passing text strings (name-value pairs) in and out of the kernel, and a generic userland program that would send these name-value pairs, with a scope (e.g. device class or whatnot), into the kernel. This mechanism would continue to permit structured data. However, he is not optimistic that someone will take up the flag becuase it’s not something on fire at the present time.

An audience member asked about whether the proliferation of English text in the kernel as a result of Poul-Henning’s proposal would be insensitive to internationalization. Poul-Henning responded that for all intents and purposes, whether one likes it or not, system administrators of UNIX systems have to know English, at least at this point in time, so he doesn’t see it as a huge concern. He acknowledged that this is a problem, though.