Julian C. Dunn

We at CBC.ca have made major improvements in our web platform over the last two years. When I first returned to CBC in September 2006, we were still running Apache 1.3.29 on SuSE Linux Enterprise Server. Since then, we’ve upgraded first to Apache 2.0.59 (still on SuSE) and, with the migration to Red Hat Enterprise Linux in July of this year, to Apache 2.2.8. (You can see the evolution of our web platform over at Netcraft.)

Two days after the Canadian Federal Election, we implemented the next major upgrade of that platform and that was to convert from the prefork MPM to the worker MPM. Since we monitor the performance of all our Apache servers using Cacti, I can share some detailed information about the performance improvement that has resulted from this change. Continue reading…

I’ve long been an advocate of origin HTTP caching and acceleration for large websites, something I alluded to in the post Performance Tuning and Optimization of High-Traffic Websites, which I wrote almost eleven months ago. In the early, heady days of the World Wide Web, many vendors like CacheFlow (later BlueCoat) and Nortel made HTTP caching appliances, but there are almost no such vendors in the marketplace now. I still believe there is a sound technical reason for an origin website architecture with HTTP accelerators deployed in front of it, and I’m happy to see that one recent entrant into this space, the Varnish HTTP Accelerator, is nearing a stable 2.0 release. In this post, I’ll elaborate on why I think HTTP caching solutions went the way of the dodo, why I think they should come back, and use the feature set and stated goals of the Varnish project as evidence. Continue reading…

I returned late last week from attending Akamai Technologies’ first Global Customer Conference in Boston. Intended to bring together Akamai’s major customers, in order to share knowledge and information about current and future Akamai products, I think I derived more insight out of my conversations with other media & entertainment customers than out of the program material. I’ll explain why. Continue reading…

A while ago I wrote a post about how the Windows Media video experience is sub-optimal on non-Windows computers — in particular, Macintoshes — and why I think this will trigger a run towards Flash on-demand and eventually Flash live. Here’s a concrete example: over the last six months to a year, and perhaps longer, we’ve been dealing with a steady stream of user complaints that the nightly newscast of CBC’s The National (insert promotional tagline about “Canada’s most trusted news source, hosted by newly-announced Order of Canada member Peter Mansbridge”) is frequently “out of date”. While I haven’t totally nailed down why this might be the case, I do note that most complainants seem to be using Quicktime to play back the stream, with Flip4Mac (ugh) as the translation layer. I believe that with so many moving parts, something is inevitably going to go wrong. Continue reading…

I’m taking a long-awaited vacation next week, in part to attend my friend Kristin’s wedding down in New Jersey, but also for the Streaming Media East conference in Manhattan. My work these days requires a great deal of knowledge about video (and audio) delivery workflows for online media, and I can see many aspects of our operation ramping up in near term. Flash-based players like the Maven Networks front-end are already in use, and I can see live Flash being only six months off. It seems like Flash is suddenly on everyone’s tongue, and at least at CBC, Windows Media, while still our standard, is no longer the market darling that it once was. Continue reading…

A few years ago, when I was still in charge of the Toronto Community Co-Location Project (a project that I’m pretty sure is defunct by now), I was approached by a fellow named Da Shi, who was just starting a company called 3z Canada. He provided some competitive rates for co-location, but we ultimately sublet space from Chris Kirby. Continue reading…

VIA Rail offers WiFi service aboard its trains. I’m on my way to Montreal for a business trip, so I thought I’d try it out. My conclusion: stay away!

It seems that VIA is partnered with a company called Parsons to provide the WiFi aboard the train. Judging by the latency, I can only assume that it is a satellite link. Check out what kind of latency you get for $8.95 per 24 hours of access:

C:>tracert aphrodite.aquezada.com

Tracing route to aphrodite.aquezada.com [216.235.8.211]
over a maximum of 30 hops:

  1     1 ms     1 ms     2 ms  VIA_3454 [192.168.134.1]
  2     *        *     3071 ms  10.0.15.1
  3  4033 ms  3878 ms  3684 ms  link1.parsons.com [206.219.255.131]
  4   987 ms     *     1154 ms  66.147.156.114
  5   758 ms  1586 ms   798 ms  ge-5-0.a0.dlls.broadwing.net [216.140.4.173]
  6   899 ms  1457 ms   739 ms  216.140.4.158
  7   697 ms  1180 ms   339 ms  216.140.4.170
  8   633 ms   640 ms   859 ms  te-8-3-73.car4.Dallas1.Level3.net [4.68.63.9]
  9  2958 ms  2638 ms  1620 ms  ae-13-69.car3.Dallas1.Level3.net [4.68.19.5]
 10  1308 ms  1136 ms  1258 ms  4.68.63.162
 11  2126 ms  2132 ms  2962 ms  5.icore1.CT8-Chicago.teleglobe.net [206.82.141.2
9]
 12     *        *        *     Request timed out.
 13   944 ms  1572 ms  4343 ms  if-15-0-0-15.mcore3.TTT-Scarborough.teleglobe.ne
t [216.6.98.49]
 14   887 ms   937 ms  1901 ms  if-15-0.core1.TNK-Toronto.teleglobe.net [216.6.9
8.54]
 15  1235 ms   477 ms   320 ms  ix-1-151.core1.TNK-Toronto.teleglobe.net [216.6.
112.22]
 16  1120 ms  1936 ms  2679 ms  204.16.202.173
 17     *     3228 ms  1316 ms  216.235.0.236
 18  1519 ms  1882 ms  3716 ms  h216-235-8-211.host.egate.net [216.235.8.211]

Trace complete.

Unless you’re just doing SSH — save your money!

A few weekends ago, I got up at the crack of dawn and headed out to the first (annual, I hope) Ontario Linux Fest. The admission price of $40 clearly signalled that this was a grassroots gathering of Linux hobbyists, but I’m sure many of those in attendance were also professional developers and/or system administrators. Although some of the talks were more show-and-tell that I would have hoped, I had to keep in mind the target audience, and I still learned a few things, particularly regarding the optimization of high traffic websites – thanks to Khalid Baheyeldin for his talk on this topic.

Continue reading…

WordPress 2.2.3 was released a little while ago, and I finally thought I should say something about the upgrade process as documented. Can anyone think of a reason why I can’t just download the tarball and diff the contents against the previous version’s tarball, and then run the upgrade.php? That’s certainly what I’ve been doing so far, and have not had any problems. This way I also don’t need to “watch out” for dangling objects in my wp-content directory, since that stuff will get ignored by the patch file. I only wish the WordPress authors could issue a patch file so that I don’t need to do this myself (and it would be nice if the authors could also tar up each distribution from a directory named wordpress-x.y.z instead of just wordpress)

In other news, I decided to fly to Dallas, TX this year to attend the LISA conference, sponsored by the USENIX Association. It’s my first time going to LISA, mostly because of the expense of doing so. Fortunately, this year I have some Air Canada Aeroplan points to use, so my airfare is essentially free (except for the $131.02 in taxes that I have to pay), but registration is still costing me around $700 and the hotel will be $875. All in all, I expect to spend just short of $2,000 on the conference. Sadly, my employer doesn’t have a policy around paying for conferences. They only reimburse for training programs, and even those need to be approved via a lengthy bureaucratic process. Hopefully my manager and I will, at some point, manage to convince the HR folks that the best “training” you can give an IT person is to keep them up to date with new developments in the relevant field, rather than sending them on meaningless courses.

It was busy in June and July over at $WORK, so I didn’t get a chance to write any entries here. Some of the work I’ve been doing include turning off all legacy servers (among the legacy servers are only 2 FreeBSD boxes and a handful of HP/UX dinosaurs, but the rest of the production environment is SUSE Linux Enterprise), shepherding the BlueArc storage upgrade through (a huge pallet containing disks, controllers, disk shelves, and a replacement Fibre Channel switch arrived last week), and, of course, planning our upgrade to a modern Apache/Java environment. This will consist of Apache 2.x with a Tomcat 5.5 back end — a far cry from our current Apache 1.x and Tomcat 3.x setup.

One of the major challenges is getting Tomcat 5.5 running on SLES 9 under a Java 1.5.x virtual machine. Actually, it’s not so much the “running” part — I’m sure that since it’s Java, it would just run if I did the old tar zxvf tomcat-5.5.tar.gz && make && make install dance. But we’re after sensible package management here, and that means trying to make SLES 9 behave the standard way. SLES 9 is missing a lot of the “standard” tools that folks use to manage Java apps; it has no jpackage-utils built-in, it doesn’t use the alternatives system, and it can’t talk to Yum repositories out of the box. The work instructions I developed here hack up the base OS a bit to bolt on these tools, but ultimately do the job.

The long-term solution, of course, is to move to either SLES 10 or RedHat Enterprise Linux 5. SLES 10 ships Tomcat 5.0.x out of the box (just like SLES 9) so on the surface, it doesn’t seem like much of an improvement. But they have moved to the alternatives system; jpackage-utils is bundled with the base system, and ZMD (for what it’s worth) will talk to Yum repositories. (Of course, that’s in theory: in practice, as with many Novell tools, it’s broken.) RHEL 5 seems like the obvious answer, since it ships Tomcat 5.5 right out of the box.

Anyway, that’s a bit of a digression. Here are my directions for getting Tomcat 5.5 installed and properly package-managed on SLES 9 with JPackage. Continue reading…