Back to Blogging Again

I was recently re-reading Steve Yegge’s You Should Write Blogs post from 2005, and started thinking about why I don’t blog anymore. I used to be a pretty prolific blogger from before the days when these things were known as “blogs”. The archives here, in fact, go back to 2003, and although there is probably some pretty cringe-worthy content that far back, I’ve never deleted a single post.

So why did I stop? It comes down to a simple reason. As I’ve moved in my career from engineering roles into leadership roles, the topics I would consider blogging about have moved from systems and software to ones of organizational dynamics, individual behavior, and management’s role in facilitating the same. To properly write about these topics would require me to write about the actions of real people with whom I work. Although I would never name them, those folks would surely see themselves in my posts, and I’m afraid of being perceived as being mean when discussing everyone’s foibles. As my friend and colleague Adam Leff said, “No matter how much we collectively say we value honest opinion and transparency, we’re all still humans with brains capable of remembering what others say of us.” This makes it very difficult to talk about challenging topics openly.

Writing is especially tough in these partisan times. Fueled by hair-trigger social media, we are all too ready to jump to conclusions about the motivations of writers and ascribe specific intent to the choice of a particular word or placement thereof. We accuse people of “sub-tweeting” when they express their observations on Twitter, or label them as “toxic” on the basis of a small number of interactions. The halcyon days of 2005 Steve Yegge, where a blogger could just publish lightly-edited thoughts without suffering severe repercussions, seem long past.

I have, however, decided to return to blogging regularly. I will sometimes be writing about difficult issues, with the objective of sharing my experiences so that we can all learn and improve together. Yet I intend to treat everyone fairly and to be kind rather than “nice”. If you see yourself in these pages please try to bear this in mind. And since I have always spent almost as much time editing my posts as writing them in the first place, you can be confident that I’ve weighed all perspectives and I’m not just shooting my mouth off after having some drinks.

With that, onward. Next up this month will be a piece I’m working on in reaction to the New York Times’s recent article on what went wrong at Etsy. I hope you’ll read it.

Building the Data General Eclipse MV/8000, or, lessons from product development in the 70’s

A couple of weeks ago I finished reading Tracy Kidder’s 1981 Pulitzer Prize-winning book, The Soul of a New Machine, in which he followed Data General (DG) engineers around for over a year while they tried to pull a miracle minicomputer, codenamed “Eagle”, out of the company. The firm’s survival was on the line. They were being slaughtered by Digital Equipment Corporation’s VAX 11/780 and they needed a winner, badly.

DG’s “new machine” ultimately became the Eclipse MV/8000, a name that, like the CDC 6600, is nearly unrecognizable to anyone today besides old computer nerds like myself. In other words, it’s damn hard to build a company that lasts. Both DG and its main rival, DEC, are gone, shoved aside by the PC revolution and the march towards inexpensive, x86-based servers running Linux. And so if you work in technology, you probably feel (or should feel) an immense sense of urgency, because there is always a surplus of discontinuous innovations just around the corner to disrupt whatever you’re working on right this instant. And those disruptors will in turn be disrupted by others, and then by yet others… Continue reading

How to be CEO: Ben Horowitz’s “The Hard Thing about Hard Things”

This is the beginning of a continuing series of reviews about books on management and leadership. Think of it as a “papers we love” but for folks who have chosen to pursue a non-technical path in their engineering careers.

At some point I figure I’d like to be CEO of my own company. So it was with great interest that I picked up Ben Horowitz’s “The Hard Thing about Hard Things”, which is partly a memoir about his time running LoudCloud and Opsware, which he eventually sold to HP in 2007 for about $1.6 billion. Along with Marc Andreessen he then went on to found the well-known Silicon Valley venture capital firm Andreessen Horowitz, which has invested in Facebook, Foursquare, GitHub, Pinterest and Twitter, among others.

From this CV it sounds like he has been wildly successful as an entrepreneur and CEO. But like all Hollywood stories (and Wikipedia articles) it glosses over the twists and turns in his career that, for instance, almost saw LoudCloud go bankrupt several times. (He also pivoted the company from being a cloud services and hosting company to a product company, nearly bankrupting it again.) Horowitz claims that many management books out there only give you advice about the happy path, but don’t teach you how to deal with adversity. That’s where the book comes in: no-nonsense advice about the many difficult situations you’ll find yourself in as a CEO. Continue reading

Continuous Deployment with a $10 USB Button

IDream Cheeky USB pushbutton picture‘ve been very busy at my job in system operations over at SecondMarket, trying to get our infrastructure in shape for many changes coming down the pipe. On the business side, the JOBS Act passed by Congress back in April means that the ban on general solicitation of accredited investors is being lifted, and so we expect to be able to grow our client base as a result.

More clients means more features needed to cater to them. On the technology side we have been working hard to deliver small packages of features faster, rather than in one large biweekly release: in other words, continuous delivery. I’m looking forward to the day when we can finally hand over the keys to engineering & have developers deploy whenever they want, using our Jenkins continuous integration system. Operations people have no business being a roadblock to software developers who want to get features out the door as quickly as possible. As long as the code is of high quality and doesn’t crash the servers, I’m comfortable with whatever gets deployed into production. It also means that engineers are 100% responsible for both the success and failure of their code — a simultaneous carrot & stick towards increasing quality.

The whole discussion around push-button deploys has led us purchasing an actual USB pushbutton. Made by a company called Dream Cheeky, this button — admittedly a little more flimsy than it appears in the picture — ships with only a Windows driver. Fortunately, someone has written a RubyGem and a Mac driver to interface with it. We’re taking the next logical step and making it possible to deploy with literally a button push. Continue reading

CBC.ca Operations Group is hiring!

My team at the CBC is hiring for two vacancies: System Administrator and Senior System Administrator.

Our group does day-to-day server and application management of the CBC.CA infrastructure, which runs almost entirely on Linux (so RHCT-level experience is needed for the former job, and RHCE for the latter). Equivalent experience, especially in the media industry, is welcome. Above all, since we’re a communications company, excellent communication skills and a pleasant demeanor are essential.

To apply, click on the links above. The postings close on Friday, November 26.

why the CBC doesn’t use “open” codecs

Every so often I hear criticism from CBC’s audience that we choose "proprietary codecs" for the distribution of our audio and video material. The arguments usually go something along the lines of:

  1. CBC is a publicly-funded organization
  2. CBC shouldn’t be beholden to proprietary technologies as it limits accessibility
  3. Therefore CBC should stream audio and video in completely open formats (e.g. Ogg Vorbis, Ogg Theora, etc.)

Nobody wants to be accused of limiting accessibility, and CBC certainly doesn’t start with a position like, "hey, we should be jerks and just lock out anyone who’s using FreeBSD/Linux/OpenSolaris/HP-UX/etc. from watching/listening to our material!" But many moving parts in the encoding and distribution ecosystem prevent us from being completely open, as I’ll explain in this article. Continue reading

it’s a strange and exciting time to be in the media

I was going to start this entry with the headline “It’s a bad time to be in the media”, but I decided to stop short of that doom-and-gloom prognostication. I won’t deny that many media organizations are suffering; some venerable institutions are closing and others are threatening to. However, I believe that those which are positioned and prepared to reinvent themselves as content factories and not as platform companies will be the winners in the long run. Doing so also involves embracing technological change and making technology a core underpinning of their workflow — something that’s going to be very difficult to digest for some. Continue reading

all I want for Christmas are some custom Apache modules

Operating an Apache httpd-based origin in conjunction with a CDN presents some interesting challenges and opportunities. For example, one can actually eliminate a lot of sophisticated cache control directives by trusting that the CDN will Do The Right Thing ™ when communicating with client browsers. Furthermore, implementation of a few judicious Apache modules and mod_expires directives can go a long way towards reducing origin bandwidth and load on the webservers.

However, dynamically-generated web pages (including those generated via SSI) can result in unnecessary cache evictions due to the inability to determine last modification time. In this article I’ll explore exactly why SSIs are so irritating from a CDN-interaction perspective and why all I want for Christmas is a CDN-aware mod_include and/or mod_expires, as per the title of this post. Continue reading

performance improvements of changing Apache MPM from prefork to worker

We at CBC.ca have made major improvements in our web platform over the last two years. When I first returned to CBC in September 2006, we were still running Apache 1.3.29 on SuSE Linux Enterprise Server. Since then, we’ve upgraded first to Apache 2.0.59 (still on SuSE) and, with the migration to Red Hat Enterprise Linux in July of this year, to Apache 2.2.8. (You can see the evolution of our web platform over at Netcraft.)

Two days after the Canadian Federal Election, we implemented the next major upgrade of that platform and that was to convert from the prefork MPM to the worker MPM. Since we monitor the performance of all our Apache servers using Cacti, I can share some detailed information about the performance improvement that has resulted from this change. Continue reading

CBC Radio Two: now in streaming MP3

Those of you who have been following the CBC Radio Two revitalization have noticed that, buried amid all the controversy about juggling classical music so that it only appears between 10 a.m. – 3 p.m., CBC launched four web-only streams on Monday: Classical, Jazz, Canadian Songwriters, and Canadian Classical. These are available twenty-four hours a day, just like any other Internet radio station. Continue reading