Following up on last year’s World IPv6 Day, the Internet Society has organized World IPv6 Launch Day for June 6th. On this day, many major ISPs and corporations will permanently launch their IPv6 presence, in recognition of the fact that the world has now exhausted the IPv4 address space and must urgently migrate to IPv6. Participating companies include Google, FaceBook, Yahoo! and CDNs like Akamai and LimeLight Networks. My question is: where on the list are the cloud infrastructure providers?Infrastructure-as-a-service (IaaS) products like Amazon’s Web Services and Rackspace’s Cloud have become extremely popular over the last few years, driven by the idea that a company, rather than purchasing its own servers and installing them in data centers, can simply rent machine time by the hour from an infrastructure vendor. The IaaS vendors, in turn, have exploited server virtualization technology like Xen and Microsoft’s Hyper-V to multiplex many virtual servers on each physical server, thereby keeping capital costs low.

The fundamental business driver is capital cost avoidance, an especially attractive characteristic for startups. New Internet companies want to get off the ground quickly and often do not have a large budget to spend up front on equipment, an enormous risk when their products are untested in the market. Now, all they need to do is raise enough cash to pay the monthly bill and they can now concentrate on making their websites a success. If they do become wildly successful and need more capacity, their operations staff can add more servers at the click of a mouse.

Unlike physical servers that a corporation might install in its own data center, each one of these cloud servers consumes a public IP address, and therein lies a huge scaling problem. Industry observers have estimated that Amazon has the capacity to run up to one million EC2 instances, which equates to approximately one million IP addresses required. (Subtract a few for Virtual Private Cloud machines, and add a few Elastic Load Balancer addresses.) The IPv4 address space was exhausted in February of last year, so it’s hard to see how Amazon or other cloud providers can sustain their torrential growth unless they already own large blocks of IP addresses.

This is what makes it so surprising that none of the IaaS vendors are at the table for World IPv6 Launch Day. One would think it would be in their best interests to quickly IPv6-enable their deployment regions, or at least create a new one that is natively IPv6, for customers who are ready and willing to deploy IPv6. After all, not every server in a company’s infrastructure needs to have a public IP address, and as long as Amazon, say, continues to provide IPv4 Elastic IPs for clients, a mixed IPv4 and IPv6 environment could be a great first step. IaaS vendors could also provide pricing incentives for clients to use IPv6-only instances, by billing them at cheaper rates in recognition of the fact that IPv4 addresses are a rare commodity.

It’s possible that an announcement could still be coming from either Amazon or Rackspace, so stay tuned.

The CBC didn’t always lack a “can-do” culture of creative excellence in technology. In fact, up until 1996, the CBC was a leader in engineering, driven internally by a group known as EHQ, or Engineering Headquarters. According to an extensive article in Broadcast Dialogue magazine [PDF], at its peak, EHQ employed 700 to 800 engineers in its Montreal and Toronto laboratories. Some notable accomplishments: building the first national satellite television distribution system, in partnership with Telesat Canada; working with Sony to develop re-usable videotape recording (VHS); moving 23 separate CBC offices & studios in Toronto into one, centralized Broadcast Centre.

That isn’t to say that EHQ was without its critics. Although EHQ pre-dated my time at the CBC, many folks I spoke with informally complained that EHQ was slow-moving and often unresponsive to business needs, designing expensive and over-engineered solutions when something cheaper might have done. One anecdote I heard involved a news director out in Western Canada who needed a new antenna on his building. Engineering came back with a schedule & cost estimate that was far beyond his budget, and they refused to budge. Infuriated, this fellow snuck up in the middle of the night with his crew and installed his own jury-rigged equipment.

Incidents like this invariably contributed to the business’s resentment of the “ivory tower engineers”. And when budget cuts were imposed on the CBC in the 1990s by both the Mulroney and Chrétien governments, the knives came out. The dismantling of EHQ must have been seen as a triumph, not only over a bureaucracy perceived as being out-of-sync with the business, but also as a victory over seemingly unnecessary technological research & development. In a world where budgets were shrinking, R&D is often seen as a frill, one that becomes a resource drain with an unpredictable return-on-investment.

The dismantling of EHQ and the evisceration of the culture of innovation came just in time for the media world to be torn asunder by the forces of the Internet. CBC had blown up the fort just in time for the barbarians to arrive at the gate. In fairness, even with EHQ, it’s highly likely that the early years of CBC online would have been equally as chaotic and riddled with the kinds of mistakes that I saw during my tenure there. After all, broadcast engineers, used to working with purpose-built, embedded systems, would have been ill-prepared to deal with the vagaries of the Internet. The early years needed to be nimble and quick, characteristics not usually associated with a slow-moving bureaucracy.

The CBC Broadcasting Centre in Toronto at night. Photo courtesy of Paul Gorbould (CC BY-NC-ND 2.0)

An engineering culture, though, could have served as the bedrock for development of long-term, mature technology solutions, once the uncertain early years passed. (There’s an analog to Google’s Site Reliability Engineering group, which takes ownership of new Google products after at least six months of stable operation.) For example, like many large media organizations with a firehose of content and an enormous back catalog, CBC has struggled mightily with one huge problem: content and digital asset management. Millions of dollars have been spent on ad-hoc, poorly-performing solutions in both digital and broadcast domains because the corporation has failed to develop a long-term strategy that it could then act on tactically. Coupled with a traditional media attitude towards the Internet — there are some senior managers in the business who still don’t believe the Internet is here to stay — and the result is a recipe for many more years of missteps.

There are a couple of key engineering takeaways, too. First, as the World Wide Web increasingly moves from being static to dynamic, the CBC will need sophisticated engineering skills more than ever. Not only will they need experts in the development of high-performance, multiplatform dynamic websites, but they’ll need both broadcast engineers with an understanding of how to integrate with the Internet, and Internet engineers who can help deploy the latest technologies behind the curtain. To build, for example, a web application at CBC today, using current-generation technologies like Ruby on Rails or MongoDB (to say nothing of building, say, a Hadoop cluster) would be an incredibly painful undertaking involving far too many stakeholders getting in the way of “just G.S.D.” Add to that a procurement process more suited to buying thousands of video cameras, with a lengthy capital process and RFPs, you get a recipe for technological stagnation in an era when, as Akamai co-founder Danny Lewin famously said, “You’re [already] behind.”

All of this may come across as pessimistic. That’s far from my intention, because actually, turning the ship around is achievable. Despite having a budget that shrinks in real terms every year (CBC’s budget remains constant and not indexed to inflation), the good news is that many of the changes are cultural. Another piece of good news: it’s really cheap to build things on the Internet. (I mean, you can store things in the cloud for ten cents a gigabyte!) The days of multi-million dollar broadcast systems as the price to entry are over. But it doesn’t mean that the same level of technological seriousness and innovation don’t need to be applied to inexpensive problems. The technology might be cheap but the business impact of doing it wrong is still as high as ever.

Tapscott is right to plant the goalposts of where the CBC should be. Getting there will entail more structural changes than he lets on.

The first underground subway line in New York City, from City Hall station in Manhattan to Bailey Ave. in the Bronx, was constructed in just four years, between 1900-1904. (By comparison, the Second Avenue subway started construction in 2007 and only the first segment, from 63rd St. to 96th St., will be completed by 2016.) Much of the tunneling work was performed by hand, in extremely dangerous conditions. Rock slides from dynamite explosions, tunnel flooding, and silicosis from inhaling rock dust were all occupational hazards. Photos from the time show men digging and blasting with no safety equipment whatsoever. Worst off were the sandhogs, men who worked at the face of the excavation. They worked two shifts of three hours each, with a two-to-three hour break in between, underground in 100 degree heat, and sometimes under atmospheric pressures of 100 psi. Moreover, racist stereotypes of the day meant that blacks were often sent to work as sandhogs; it was felt that being from Africa, they were more accustomed to the heat and humidity.

Living conditions were atrocious as well, making FoxConn’s dormitories seem tame by comparison. The New York Transit Museum‘s exhibit on subway construction stated that,

Most families shared their quarters with boarders. Conditions were wretched: tenants had to fetch water from a hydrant in the yard, outhouses reeked with filth, and animals shared cellars with people. The word ‘tenement’ became synonymous with the word ‘slum’.

In the intervening years, we in America have decided that such harsh conditions are no longer acceptable. We’ve enacted laws to ensure that such conditions will no longer be inflicted on Americans. That’s why it will take over a quarter century to build the full Second Avenue subway: because we now believe in safety, reasonable working conditions, and, of course, consultative urban planning. But back to the case of Apple products: if we, as an “developed” nation have the power to choose how our products — products that we invented — are manufactured, why would we instead export the harsh working conditions instead of that standard of life?

I think I know the answer, and it’s an uncomfortable one. We’re happy to exploit others when we don’t see them as ourselves. Whether that be Chinese migrant workers, or Lower East Side immigrants slaving away on the subways at the turn of the 20th century, it’s easier to justify turning a blind eye when you can reap the rewards without being the one getting your hands dirty.

There’s a subtle warning in here, though. America’s economic star is setting and China’s is rising. Will we still be so blasé about labor conditions when, in 25 years, we are the ones manufacturing electronic devices for middle-class and upper-middle-class Chinese consumers?

There’s obviously no need for me to explain who or what Google is, but I should mention that I was recruited for an opening in their Site Reliability Engineering (SRE) group, and what that means. SRE’s job is to act as the second stage in the evolution of new Google products. After products stabilize (at least six months running on their own), they are eligible to be adopted by SRE, who will help them grow by, for example, migrating to some of Google’s in-house infrastructure tools like BigTable or MapReduce. This is particularly important for large web applications that come from companies Google acquire; SRE’s job is to help shepherd these products into the “Google Way”. SRE also has an operational role, keeping the biggest Google products like AppEngine or the core search product running smoothly. As such I was told that SRE is the biggest engineering organization within Google.

As I discovered along the way, Google is a company with little organizational hierarchy. Obviously, many engineers like this: it means a minimum of management “overhead” getting in the way of doing great work. There’s also not a huge variety of roles in the (technical) organization. If you’re not an engineer, you don’t really have a place. On the flip side, the system makes engineers wear lots of hats: product manager, system administrator, project manager and QA tester in addition to being a software developer. This works well if the hiring process is rigorous and continues to bring in people who can, again, do things the “Google Way” and adapt to life in an organization where engineers self-organize into autonomous groups to just G.S.D.

All this leads me to try and explain what a TPM is supposed to do. I presume that the danger in having groups of high-functioning engineers determining their own work is that it can lead to a disconnect with the business goals, product features, and timelines. My best attempt at defining the role would be “cat herder”: using both good and bad cop techniques to push very individualistic engineers into shipping product, having the technical wherewithal to speak to engineers on their level, and even plunging in with both hands when necessary.

Eventually it dawned on me that the lack of structure & defined responsibilities at Google do not make it a good fit for me. For a program manager job, I was interviewed over the course of a full day on large-scale system design (requiring mathematics and a whiteboard!), project management, programming, UNIX systems administration, networking, and even more project management. These weren’t just cursory interviews, but ones in which I had to demonstrate a fair amount of expertise.

Google is a very unique company. They’ve managed to build an engineering-centric shop with many in-house solutions to solve problems for their scale; not many companies either have that scale, or the resources (financial or personnel) to pull that off. And they have a rigorous hiring program that seems to have worked for them, even as they’ve grown to 30K+ employees. However, I think they’ll eventually have a tough time finding generalists with the requisite amount of knowledge to do it all, and well. (One Googler told me that they’ve pretty much either interviewed or hired everyone in the Bay Area that they want to, implying that they’re having trouble growing their team.) Eventually, I believe, even Google will have to recognize that not everyone can do everything well, and they’ll have to specialize. Maybe at that point, I’ll be willing to consider them again.

The public (and This American Life listeners) will now be left with an hour-long, self-flagellating examination of how authenticity and credibility in journalism has yet again been ruined. Mike Daisey will be branded as just another John D’Agata, someone who invents facts for no better reason than that it makes for a better story. (I wonder what kind of service he’ll get now when he calls AppleCare.) And, as iPad 3 sales exceed sales records, Apple’s share price will hit $1,000 in the next six months and be a better capitalized company than the lowest 50 countries on the Human Development Index combined, while everyone forgets about the actual issue that was being addressed here: appalling labour practices in overseas factories making electronic widgets for the developed world.

Shame on you, Mike Daisey. Not being a “journalist” is no reason to lie. And now you have done more damage with a bunch of fabrications than if you had just stayed home and not “reported” this story.

Keep in mind that I’m strictly a subway user, so I haven’t had occasion to test any of the bus-related apps.

Top App: ExitStrategy NYC

The main purpose of ExitStrategy is to tell you what subway carriage you should board so that when you disembark, the exit you want is right in front of you. It lets you use the time spent waiting for a train to position yourself optimally on the platform. For instance, if you’re taking the downtown F and getting off at my stop (Carroll Street), you want to get on the last carriage, because the President Street exit is at the back of the train.

ExitStrategy is extremely well-designed. Just click on any station on the map, choose a direction (uptown or downtown) and it’ll show you a pictogram of the train at the platform in question, with all the exits labelled. The headlights on the train are a nice touch, showing you which direction the train is moving.

Pros: In addition to its basic functionality, the app also comes with copies of the high-resolution neighborhood maps that you find posted in a station. The bus maps are a nice touch, too.

Cons: The maps can be slightly buggy when zooming in or out. Sometimes the app won’t redraw the maps to the right zoom level, leading to fuzziness, which is particularly annoying with the neighborhood maps. Also, the overall map used is from June 2010, so it doesn’t reflect system improvements. For example, Lawrence St. station on the R line now connects directly to Jay St./Metrotech on the A/C/F, and as of Fall 2011, the B only makes express stops in Brooklyn on its way to Brighton Beach.

NextStop

NextStop shows a countdown clock for any given station telling you how soon before the next train(s) is/are scheduled to arrive. Obviously, it’s just an estimate based on the average inter-arrival rate of trains, but it should give you a rough idea, especially late at night when trains are infrequent, whether you should boot it to the station or stay at the bar for another fifteen minutes (to give a real-life example).

Pros: The app includes a location detector, so if you’re in an unfamiliar area and you want to know what stations are nearby & when the next trains are scheduled to arrive, you can ask the app to just “Use My Current Location”. The app also includes an up-to-date copy of the MTA’s map from October 2011 thereby incorporating the changes to R and B/Q service I alluded to above.

Cons: The “Alerts” and “Advisories” functions seem to be broken. Also, you can save Favorites (line, station and direction tuples) but you can’t edit their parameters nor the name; you can only delete them and start over. Finally, there’s no coordination with planned work, so the app might tell you that a train is about to arrive at a station at which there’s no current service.

CityTransit

CityTransit is mainly a subway map with a few extra frills on the side, like a “Lines” feature where you can see a list of all the stops on a particular line and whether the stops are serviced all the time. It also has a service advisories tab, where it queries the MTA’s website for current service advisories affecting any line you choose.

Pros: Map zooming is smooth, unlike ExitStrategy. Also, the main user interface is well-designed, with the four main features as buttons along the bottom: Maps, Lines, Locate, and Service.

Cons: The application hasn’t been updated since October 2010, which means that it uses the outdated June 2010 map. This also affects the “Lines” feature. The service advisories feature could also use some improvement, partly because the MTA’s Service Status feed is a godawful mess of unnecessarily-escaped HTML character entities and therefore could use some cleaning up by app developers. Also, the app costs $2.99, which I don’t think is good value for an app that has these issues.

HopStop

HopStop is actually an all-purpose route planner; you can use it to plan your trip by walking, taxi or bicycle as well. It does a reasonably-good job at planning subway routes, taking into account any construction (not sure how they do this as I can’t see any API on the MTA’s developer website that provides this info). The route planner walks you through your trip step-by-step and shows walking maps to and from your starting and ending stations respectively.

Pros: Good route planner, perhaps the only one out there that’s usable on-the-run.

Cons: The user interface for inputting start and end is quite clunky, with text fields for the addresses plus two dropdowns each for the “Area”. Also, the text fields don’t use the iPhone’s autocorrection feature, so you must enter the address precisely — quite a challenge if you’re walking down the street and in a hurry. Finally, the walking maps are really ugly; they look like they’re generated from MapQuest.

MetroCalc

MetroCalc is a nifty app that’ll let you determine how much money to add to your MetroCard in order to make the final balance divisible by $2.25 (or really, whatever fare you decide, e.g.  Express Bus = $5.50). The app’s needed because the MTA gives you $0.70 bonus for every $10 you add to your MetroCard, so to avoid odd balances, you need something that’ll solve the equation money_to_add = (2.25 * rides – initial_balance) / 1.07 such that rides is an integer. (I imagine the author just uses a lookup table.)

Pros: Does what it’s supposed to, and it’s free. Enter the amount of money you have left on your MetroCard and the app tells you what to add to it.

Cons: Mismatch between the app’s user interface and what the MetroCard vending machine shows as the “final” balance. This isn’t really the app’s fault, because the final balance shown on the MetroCard machine doesn’t include the bonus, whereas the app does. (See the illustrations below.)

Screenshot from MetroCalc showing the possible sums you can add to an $8.05 MetroCard to get an even number of fares.

MetroCard vending machine with a pending transaction to add $11.40. The total doesn't match MetroCalc because it doesn't include the bonus.

I’ll be at the NY Tech Meetup rally in front of Senators Schumer and Gillibrand’s offices at 12:30 p.m. If you’re in New York and concerned about the future of the Internet, I’d encourage you to join me there.

Jarvis never really defines publicness, and maybe that’s because the book addresses two distinct forms of publicness; personal publicness, and institutional publicness. These don’t really have that much to do with one another, other than the fact that platforms for personal publicness, like Facebook or Twitter, can also be used by institutions to be public. I’m all for institutional publicness, especially disclosure of as much government data  as possible. Any tools that cause governments to divulge materials, whether willingly or unwillingly, are valuable to society.

But the underlying principle driving institutional disclosure — the betterment of society — is not the same principle driving personal disclosure. Individuals are motivated to disclose their own information primarily because it satisfies a base human need for attention. Twitter, Facebook, Tumblr, and other social media services are biased towards self-promotion and ego-stroking. Underlying all the lofty talk about “connecting people” and “creating communities” is the fact that most people’s tweets or Facebook updates are simply subtle or overt statements about how cool they are. That’s not to say that users of these services can’t overcome that inherent bias and have meaningful conversations. But I think it happens a lot more rarely than social media advocates would have you think.

That’s what ultimately disappointed me about Jarvis’s book. As someone who already has had a substantial online presence (I’ve had a website since 1995), I’ve clearly bought into the notion that a public presence is helpful. People can find me easily, and my ideas have a platform that I control. But I find the type of public life advocated by Jarvis — to build one’s personal brand for the primary purpose of marketing oneself for professional and personal gain — treads too far on the side of raw narcissism. He holds Seth Godin up as an example of someone who has a strong brand and therefore someone to be emulated. Godin gives Jarvis unsolicited advice on how the book itself can be leveraged for professional gain:

Godin is to blame for my writing books. He sat me down one day and said I was a fool if I didn’t write one — and I would further be a fool if I thought the book was the goal. No, he said, the book would buiuld my public reputation, which would lead to other business.

The notion that the book is not primarily a conduit for ideas but simply a means to an end, personal gain, is what’s awful. The end state would be a world of pure branding and personal marketing. This is the kind of publicness that I can neither agree with nor wish to emulate.

Sadly, I don’t see the tide turning anytime soon. As much as I may disagree with it, Jarvis’s description of the new publicness is reality. To the detriment of real conversations about real ideas, those who spend the majority of their time actually working are overshadowed by those who spend 10% of the time working and 90% of the time talking about their work. We have met the enemy and he is us.

But what struck me after hearing Sean had nothing to do with Chef. Instead, I came to the conclusion that pure systems administration is eventually going to die out as a profession. The developer is now king (or queen), and that’s not a bad thing.

Let’s step back for a minute and talk about CM tools in general. Traditional CM tools — to the extent that they existed before cfengine et. al. – know nothing about the underlying semantics of what you ask them to do. At CBC, we had a set of elaborate shell and Perl scripts that were written in-house, collectively known as ASC, Application Server Control, to do so-called configuration management of the origin infrastructure. ASC’s sole job was to revision control configurations, perform deploy and rollback operations, and perhaps do some auditing. But it was prescriptive, not descriptive. Most of the time I spent monkeying with ASC was debugging how it was doing things.

Enter Chef (or Puppet, LCFG, cfengine, BCFG2; pick your poison). These are all configuration management tools that allow you to describe your infrastructure  in a fourth-generation language (4GL) way. You describe the features that certain hosts should have, and the tools, using canned recipes, makes it happen. (“Make me a MySQL server,” for instance.) Another advantage of these tools is that they (can) keep track of the state of your infrastructure, and you can query that database to make decisions about new deployments. “How many MySQL servers do I have?” for example. Or even “Which node is the MySQL master?” and then kicking off another job on a new MySQL slave to automatically start replicating from the right server.

Had it not been for the development of IaaS — infrastructure as a service — everything that I’ve told you would not be particularly noteworthy. But IaaS, or “cloud computing”, now allows anyone to provision new (virtual) servers inexpensively. No more waiting around for the system administrator to order a couple servers from Dell, wait a few weeks for them to arrive, rack them up, configure them, etc. Developers, armed with a tool like Chef and its huge cookbook of canned recipes for making many standard infrastructure components, can fire up everything they need to support their application themselves. Therein lies the demise of system administration as a standalone profession and the rise of “devops”.

I admit that when I first heard the concept of “devops”, I snickered. “Give developers the keys to the infrastructure and they’ll surely break it beyond repair and expect the sysadmins to fix it,” I thought. But it’s finally dawned on me that “devops” isn’t just some buzzword concept that someone has thought up to make sysadmins’ lives hell. It’s the natural evolution of both professions. By bringing development and system administration closer together, it does two things. First, it makes developers operationally accountable for their code, because they are the ones that get paged in the middle of the night, not some “operations team” upon whom they can offload that responsibility. And secondly, it makes those on the systems side of the house better at their jobs, because they can use newly-acquired programming skills to manage infrastructure resources in a more natural way.

So will IaaS and sophisticated configuration management tools kill the system administrator? I believe so — but that’s not a bad thing. System administrators have got to stop thinking of servers/disk/memory/whatever as “their resources” that “they manage”. Cloud computing has shown us that all of that stuff is just a service, dedicated to nothing more than serving up an application, which is what really matters. If sysadmins want to remain relevant, they’ll get on board and start learning a bit more about programming.

Today, Brisbane wrote a blog post entitled Should The Times Be a Truth Vigilante? An unfortunate headline, to be sure, if you’re talking about the New York Times. Unfortunately, many readers and media commentators failed to read beyond the headline and understand what Brisbane is really asking, which is not whether reporters should be dogged in their pursuit of the truth. Of course they should. What he’s asking is whether a hard news reporter should be calling out potentially untruthful statements made by sources, or whether the reporter’s job is to simply quote the source accurately.

It’s easy for everyone to pile on Brisbane. After all, if a reporter is certain that a source is lying, and has the evidence to back it up, then it’s incumbent on them to say that, right? But rarely is reality so neat and tidy. This isn’t “Law and Order”, after all. The example Brisbane uses is whether Clarence Thomas was to be believed when he stated that he “misunderstood” a financial disclosure form by failing to report his wife’s earnings from the Heritage Foundation. This is not something that can be proved truthful or not, unless one were able to get inside Thomas’s head.

Brisbane, in response to a question from Jim Romenesko, says that “in this case a lot of people responded to a question I was not asking.” I completely agree. So let me offer my opinion on the question that Brisbane was asking. I think reporters should avoid inserting their own judgments as to the truth of a source’s statement, unless they are absolutely certain one way or another. While it may be unbelievable to some readers that Clarence Thomas, a sitting Supreme Court judge, could misunderstand a financial disclosure form, he deserves the benefit of the doubt. It should be up to the reader, not the reporter, to decide whether Thomas is either lying or an idiot.

The public already likes to criticize reporters for a perceived lack of impartiality. Having them add opinions about their sources’ credibility to stories would only create even more noise, especially on such hot-button topics as the Israeli-Palestinian conflict. The resultant letters that would invariably land on Brisbane’s desk would make today’s pile-on seem like a vacation day.