It’s been a rough 48 hours for those of us at CBC.ca Operations. As you can see from the posted notice, a failure of the primary storage device hosting most of the site’s content has knocked out most of the website. Tod Maffin has posted a reasonably complete explanation on Inside The CBC, and I’m thankful for that. At the time of writing, the main volume containing the site content is still offline and fscking with no known ETA.
Originally I wasn’t going to say anything more about the outage, because Tod’s given an adequate update in his entry, and any speculation about the root cause of the outage (whether technical or managerial), and how soon it might be before we can restore service is just that — speculation. But some of the comments that have been posted on the above entry are just astounding and prompted me to write. While I am happy that many technical geeks are amongst our most enthusiastic audience members, I find the glib attitude of many of them with respect to operating the site to be very disturbing and upsetting. For example, a poster named Matt writes:
That is a terrible network design – storing all the files on a single NAS (or even on the same cluster). A site such as CBC.ca should have multi-mirrored servers at different locations each providing access if one cluster goes kaput. Youòùd think with the 150(!) union employees running cbc.ca theyòùd have one light bulb smart enough to figure this out.
It’s very easy to be an armchair sysadmin and wave a magic wand to make redundancy and extra storage devices appear out of thin air. The trouble is that many of us who work in CBC.ca Operations have been saying many of these things for years, but for a variety of reasons (one of which is money, of course) nothing has been done. Comments like this denigrate the integrity and dedication of all of the folks that work at CBC.ca and make it sound like nobody knows what they’re doing.
The other thing I don’t understand about the commenters on InsideTheCBC: If you care so little about the CBC’s viability, why would you post comments like
Did anybody ever think that maybe a supreme being simply had enough of the biased, substandard, terrorist supporting anti-USA claptrap the CBC puts out as news and wanted to give good Canadians a break? What a truly useless and substandard network the CBC is. Thank God for channel lock.
or even read a blog that discusses the CBC? This kind of I-don’t-care-but-I-really-do-care attitude genuinely puzzles me. I don’t like Fox News, but that doesn’t mean I go onto their website and forums and post about how bad a job they’re doing journalistically.
I have but one plea for everyone, both internal and external to CBC.ca: instead of pointing fingers and engaging in "what if" or "should have" discussions, let’s all work together to a) recover from this failure and restore service to Canadians, and b) build the proper infrastructure with the right redundancy, capacity, etc. to ensure something like this never happens again.