CBC.ca Operations Group is hiring!

My team at the CBC is hiring for two vacancies: System Administrator and Senior System Administrator.

Our group does day-to-day server and application management of the CBC.CA infrastructure, which runs almost entirely on Linux (so RHCT-level experience is needed for the former job, and RHCE for the latter). Equivalent experience, especially in the media industry, is welcome. Above all, since we’re a communications company, excellent communication skills and a pleasant demeanor are essential.

To apply, click on the links above. The postings close on Friday, November 26.

/usr/bin/vmware-config.pl gone!

I upgraded to VMWare Workstation 6.5 recently and now /usr/bin/vmware-config.pl is gone. I only discovered this recently when I updated my kernel for a security fix, and lo and behold, the old method of making vmnet, vmmon, etc. modules for the new kernel no longer applies!

It seems like others are having the same problem and I can’t find a sensible solution other than uninstalling and reinstalling VMWare. It seems the geniuses over at the Evil Machine Corporation have decided to replace vmware-config.pl with some sort of GUI called vmware-modconfig that doesn’t seem to work right.

Why can’t people just leave working tools alone — or at least preserve the familiar API for people that don’t want to wade through 300 pages of PDFs to figure out how to fix the breakage?

Ontario Linux Fest wrap-up

I attended the second annual Ontario Linux Fest on Saturday, held at the Days Inn and Conference Centre near the airport in Toronto. These days I find that my interest in specific technologies is waning and that I’m more interested in the public and social policy angles of technology and in particular, free software. Continue reading

fedora 9 upgrade

In previous entries here I have described my unhappiness with the Highpoint series of RAID controllers. In particular I owned the 1740 4-port SATA RAID controller, but dis-satisfaction with the frequency of driver updates finally caused me to dump the 1740 for another controller. (Note that even though Fedora 9 is the current release, Highpoint has still not updated their drivers beyond Fedora 7, which is almost EOL.) Continue reading

64-bit Xen considered harmful

Recently at work, we tried to implement Xen on Intel Xeon, running a 64-bit dom0/domU. I have to say that this failed horribly, so I’m writing this post to warn others off it. My colleague Gabriel worked hard to migrate everything back to a 32-bit environment, so kudos to him.

The specific symptoms we experienced while running 64-bit Xen is that the domU’s would crash and reboot randomly under (or after) high load. One of our domU’s is a development server, which also runs a CruiseControl, a continuous integration system. This means that every minute, CruiseControl wakes up, does a cvs update to see if there are any changes, and then recompiles the project(s) if needed. Periodically we started to see error messages like

Bad pte = 32971e067, process = cvs, vm_flags = 100077, vaddr = b7f34000
[] vm_normal_page+0xb7/0xd3
[] unmap_vmas+0x3d1/0x761
[] unmap_region+0x8a/0xf0
[] do_munmap+0x148/0x19b
[] sys_munmap+0x33/0x41
[] syscall_call+0x7/0xb

After a few of these, domU would reboot. It seems like others are having the same problem on 64-bit Xen. This user was running CentOS 5.1, which is basically what we’re running (we have the real deal Red Hat Enterprise LInux 5.1).

As I said, migrating the domU back to a 32-bit dom0 seemed to fix this, so let this be a fair warning to others thinking of running a 64-bit dom0.

maddog on LTSP – and my rebuttal

One other interesting talk at Ontario Linux Fest was hearing Jon “maddog” Hall give a keynote. I remember Maddog giving a talk at Real World Linux back in 2004; in fact, I even wrote about it. Maddog’s been around the block many times, which is why I was surprised to hear him give a keynote on how the LTSP (Linux Terminal Server Project) and Linux-based thin clients are going to save the world. I’m overstating that a bit, but I feel I have to vehemently rebut. My thesis is: We’ve been on thin clients before, they were called VT100 green screens, and nobody really wants to go back – damn the peripheral factors.

Continue reading

performance tuning and optimization of high-traffic websites

A few weekends ago, I got up at the crack of dawn and headed out to the first (annual, I hope) Ontario Linux Fest. The admission price of $40 clearly signalled that this was a grassroots gathering of Linux hobbyists, but I’m sure many of those in attendance were also professional developers and/or system administrators. Although some of the talks were more show-and-tell that I would have hoped, I had to keep in mind the target audience, and I still learned a few things, particularly regarding the optimization of high traffic websites – thanks to Khalid Baheyeldin for his talk on this topic.

Continue reading

Highpoint Linux Open Source Driver for RR174x Updated

Highpoint has finally updated their Linux Open Source Driver for the RR174x series of SATA RAID adapters. I’m hoping that this will resolve the inability to compile the driver under newer versions of the Linux kernel, such as for the updates that have been issued for Fedora 7. However, my Linux box at home is currently turned off, since I’m on vacation. I won’t have a chance to test it until I get back, but I thought other users of this hardware might be interested!

installing Tomcat 5.5 on SUSE Linux Enterprise Server 9

It was busy in June and July over at $WORK, so I didn’t get a chance to write any entries here. Some of the work I’ve been doing include turning off all legacy servers (among the legacy servers are only 2 FreeBSD boxes and a handful of HP/UX dinosaurs, but the rest of the production environment is SUSE Linux Enterprise), shepherding the BlueArc storage upgrade through (a huge pallet containing disks, controllers, disk shelves, and a replacement Fibre Channel switch arrived last week), and, of course, planning our upgrade to a modern Apache/Java environment. This will consist of Apache 2.x with a Tomcat 5.5 back end — a far cry from our current Apache 1.x and Tomcat 3.x setup.

One of the major challenges is getting Tomcat 5.5 running on SLES 9 under a Java 1.5.x virtual machine. Actually, it’s not so much the “running” part — I’m sure that since it’s Java, it would just run if I did the old tar zxvf tomcat-5.5.tar.gz && make && make install dance. But we’re after sensible package management here, and that means trying to make SLES 9 behave the standard way. SLES 9 is missing a lot of the “standard” tools that folks use to manage Java apps; it has no jpackage-utils built-in, it doesn’t use the alternatives system, and it can’t talk to Yum repositories out of the box. The work instructions I developed here hack up the base OS a bit to bolt on these tools, but ultimately do the job.

The long-term solution, of course, is to move to either SLES 10 or RedHat Enterprise Linux 5. SLES 10 ships Tomcat 5.0.x out of the box (just like SLES 9) so on the surface, it doesn’t seem like much of an improvement. But they have moved to the alternatives system; jpackage-utils is bundled with the base system, and ZMD (for what it’s worth) will talk to Yum repositories. (Of course, that’s in theory: in practice, as with many Novell tools, it’s broken.) RHEL 5 seems like the obvious answer, since it ships Tomcat 5.5 right out of the box.

Anyway, that’s a bit of a digression. Here are my directions for getting Tomcat 5.5 installed and properly package-managed on SLES 9 with JPackage. Continue reading