Julian C. Dunn

I‘ve been very busy at my job in system operations over at SecondMarket, trying to get our infrastructure in shape for many changes coming down the pipe. On the business side, the JOBS Act passed by Congress back in April means that the ban on general solicitation of accredited investors is being lifted, and so we expect to be able to grow our client base as a result.

More clients means more features needed to cater to them. On the technology side we have been working hard to deliver small packages of features faster, rather than in one large biweekly release: in other words, continuous delivery. I’m looking forward to the day when we can finally hand over the keys to engineering & have developers deploy whenever they want, using our Jenkins continuous integration system. Operations people have no business being a roadblock to software developers who want to get features out the door as quickly as possible. As long as the code is of high quality and doesn’t crash the servers, I’m comfortable with whatever gets deployed into production. It also means that engineers are 100% responsible for both the success and failure of their code — a simultaneous carrot & stick towards increasing quality.

The whole discussion around push-button deploys has led us purchasing an actual USB pushbutton. Made by a company called Dream Cheeky, this button — admittedly a little more flimsy than it appears in the picture — ships with only a Windows driver. Fortunately, someone has written a RubyGem and a Mac driver to interface with it. We’re taking the next logical step and making it possible to deploy with literally a button push. Continue reading…

My Christmas break project was to build a Hackintosh out of a Dell Mini 10v. The Mini 10v is a $299 NetBook that, I swear, is deliberately manufactured with on-board parts suitable for creating a Hackintosh.

There are tons of guides out there with conflicting instructions on how to create a Hackintosh on a Mini 10v. I’ll just share with you what worked for me, in a really brief way, because I know you’re busy and want to get working on your new Hackintosh! Continue reading…

This isn’t the most intellectually stimulating post, but I recently needed to build a custom serial cable to hack into an APC AP9606 Web/SNMP Management Card. (I bought one off eBay for $35.00 installed in a MasterSwitch Plus remote power strip, and the default password had been changed.

Now, I hate soldering. The last time I had to solder a serial cable was to build an RS-422 so a friend and I could get onto the console of a salvaged Challenge L, but that was a real pain. This time, I just needed a one-off to emulate an APC 940-0024 UPS communications cable.

I ended up just going to Above All Electronics and picking up a pair of RJ45-to-DB9 adapter kits — one female, and one male — and wiring the correct pins together. For the purposes of serial communications with the MasterSwitch, you really only need the following connections:

• Pin #2 Female to Pin #2 Male
• Pin #3 Female to Pin #1 Male
• Pin #5 Female to Pin #9 Male

Everything else I just wired randomly to make it neat and clean.

Then, just take a regular Cat5e network cable and connect the two adapters you built and presto, a custom serial cable!

Thanks to the folks on tek-tips.com that provided instructions for how to reset the admin password on the card.

Today I sent another SATA hard drive back to Seagate because it failed. You might recall that I have had a bad track record with SATA drives: since purchasing this PC about two years ago, I’d gone through about three Western Digital SATA drives (all replaced under warranty due to failure) until I finally got fed up about six months ago and bought a pair of Seagate Barracuda 250 GB drives. One of them failed after three months, and the other died just this past weekend. Fortunately, my PC is RAID-1 protected (and I have all the data backed up on DLT) – but seriously, why are SATA drives so prone to failure? Continue reading…

It’s that time of year again when I start to think about what computer upgrades I might want to do. I’ve had some annoying things happen with my desktop PC recently and have considered either replacing it entirely, or implementing some much-needed upgrades. Continue reading…

Highpoint has finally updated their Linux Open Source Driver for the RR174x series of SATA RAID adapters. I’m hoping that this will resolve the inability to compile the driver under newer versions of the Linux kernel, such as for the updates that have been issued for Fedora 7. However, my Linux box at home is currently turned off, since I’m on vacation. I won’t have a chance to test it until I get back, but I thought other users of this hardware might be interested!

This month in ACM Queue there’s an interesting and lengthy article entitled The Long Road to 64 Bits, which addresses why, fifteen years later after the 64-bit MIPS R4000 was announced, most systems are still not fully 64-bit clean. I use the word “clean” to mean that most systems do not run entirely in 64-bit mode; many systems are running 32-bit operating systems on 64-bit processors, or even when a 64-bit operating system is in use, running many 32-bit programs in compatibility mode.

The article is a fascinating account of how technological decisions that were made all the way back in the 1970′s, both with respect to hardware and compilers, still impose limitations today. Although many of the hardware compatibility challenges have now gone away — for example, system designers now know enough to trap address bits that they are not using for addresses rather than letting "clever" programmers get away with using them for data — the assumptions that were made by programmers back in the days of 16 and 32-bit machines with respect to the size of C data types continues to hinder the porting of programs from 32 to 64-bit. One can’t just make and hope one’s pointers all work. As Mashey puts it, some programmers got sloppy and assumed things like

sizeof(int) == sizeof(long) == sizeof(ptr) == 32

All this may sound really abstract to readers who don’t have a hardware design background (admittedly, mine is minimal, but I understand enough of the general concepts) so let’s talk about how this impacts us end-users. We run into problems like “there is no 64-bit web browser that can execute Java and Flash” because the Java and Flash plugins haven’t been ported to 64-bit clean versions. In some ways, this is an example of shocking neglect on the part of software vendors like Sun and Macromedia (pardon me, Adobe). Bug Number 4802695, entitled “Support Java Plug-in on 64-bit AMD Opteron”, has been open with Sun since January 14, 2003, and after three years there is still no resolution in sight. This should be embarrassing for Sun, which is a vendor of 64-bit Opteron and UltraSparc IIIi workstations.

Continue reading…

Spurred on by a recent thread on the TAUG mailing list, I’ve been thinking about the cost vs. benefits of Power over Ethernet (PoE). PoE is a way to use the extra 4 wires in 10/100/1000-baseT Ethernet wiring for powering devices. The obvious application is to power IP telephones. Cisco has been doing this for a while using a proprietary solution which is similar to the PoE standard, IEEE 802.3af, which defines a protocol for detecting whether a device (or PD) is capable of receiving power prior to applying it, so that non-PDs don’t get fried by sending 48VDC down the wire. The protocol also determines how much current the PD will draw, and classifies the PD into a device class. All of this is well-described by this Wikipedia article.

There are two main advantages to using PoE, so far as I can see:

1. No need to get a separate power brick for each device
2. PDs can be centrally powered from a UPS in case of mains failure

The main con is the added cost, since you now need to purchase a PoE-enabled switch (which will then draw quite a bit of power, as it needs to power the devices).

What kind of cost are we talking about here?

1. HP Procurve 2650 48-port switch (non-powered): USD $812.99
2. HP Procurve 2650-PWR 48-port switch (PoE): USD $3,312.99

This works out to about USD $50/port, which I still think is a bit high. But maybe once the cost drops to around USD $25/port, it will be worth it.

Originally I was going to mention that purchasing PoE power injectors is another solution. A power injector has two Ethernet ports and a power brick attached to it; one of the Ethernet ports is an input from the regular non-powered network, and the other is a PoE output port to your PD. But I did some digging and find that the cost of a power injector is greater than USD $50, so I suppose this would only be feasible for a small deployment.

Step back for a moment and you realize that the cost of implementing the logic for PoE is far, far less than USD $50/port. The manufacturers are clearly just capitalizing on the fact that this is “new” technology and making a huge profit on it. As such I predict that there will definitely be wiggle room on the price in the future which will make PoE deployments more cost-effective.

Looks like someone has started putting together some informal documentation for NetworkManager.

In a completely unrelated note, the upgrading of my Fedora Core 5 Thinkpad T42 to kernel 2.6.17-1.2139 has broken wireless (again). Any attempt to use NetworkManager with it causes ipw2200: Firmware error detected. Restarting. to be seen in the dmesg. However, if I run wpa_supplicant manually and then dhclient, it works.

I’m really looking forward to the day when all this is fixed, although I suspect wireless is such a bleeding edge problem space that the day won’t be coming soon.

Recently we bought a low-end IBM xSeries 306m server to handle generic IT utility tasks, such as hosting an installation of Request Tracker, Cacti and, in the near future, Nagios. The server came with a pair of 160GB SATA disks attached to a ServeRAID-8e HostRAID controller. I quickly discovered that HostRAID is an awful hack; it’s not real hardware RAID, but software-emulated RAID, utilizing the host system’s SATA controller to do the actual I/O to the disks, but with the RAID processing done in software using a proprietary driver, in my case, a driver called adpahci. In other words, it’s "A Fake RAID", which some pundits have noted collapses into the fitting acronym AFRAID.

Several admins have criticized HostRAID for a number of reasons:

• Performance is terrible because the AFRAID controller must do polled I/O (PIO) through the CPU
• The drivers are, by nature, proprietary, since the RAID logic is licensed from a third party
• Limited sophistication in array rebuilds, since the controller has a minimal BIOS and online rebuilds are not possible
• Disks in an AFRAID array are probably unusable outside of the array, given that the driver is chipset-specific

Although I don’t really care about performance for such a low-end utility box, I have been seriously bitten by the second point. We use RedHat Enterprise Linux 4 Update 3 on all production servers like my utility box. IBM only provides binary HostRAID drivers up to RHEL4 Update 2. You can allegedly rebuild the drivers using a SHIM from Adaptec, but it doesn’t work; although the SHIM package contains C drivers for all the Adaptec HostRAID controllers (aar81xx, adp94xx, adpahci, adpsata, etc.) the only binary blob you can obtain is the one for the aar81xx. Ergo, I am S.O.L. I’m stuck with a RHEL 4 Update 3 userland on a RHEL 4 Update 2 kernel.

I guess the appropriate solution if you’re going to buy this model of server (with SATA) is to ditch the on-board AFRAID and buy a ServeRAID-7t SATA controller, which has a real 80302 processor and 64MB of cache memory, or any of the other ServeRAID products which fit in the server.

On a final note, what the heck is with IBM’s insane naming schemes for all of its ServeRAID products? I can’t keep the 6i+, 7t, 7k, 7e, 8e, 8i, 6M, etc. straight — can you? Have a look at this driver matrix and your eyes will glaze over. Why don’t they name the controllers something meaningful?