Last night, at a meeting of NYLUG, the New York City Linux Users’ Group, I watched Sean O’Meara whip through a presentation about Chef, the system configuration management (CM) tool. I was impressed. The last time(s) I tried to play with automation tools like cfengine and Puppet I got very frustrated at their complexity. The folks at Opscode have definitely succeeded at bringing simplicity (as much as can be had) to the CM space.
But what struck me after hearing Sean had nothing to do with Chef. Instead, I came to the conclusion that pure systems administration is eventually going to die out as a profession. The developer is now king (or queen), and that’s not a bad thing.
Let’s step back for a minute and talk about CM tools in general. Traditional CM tools — to the extent that they existed before cfengine et. al. — know nothing about the underlying semantics of what you ask them to do. At CBC, we had a set of elaborate shell and Perl scripts that were written in-house, collectively known as ASC, Application Server Control, to do so-called configuration management of the origin infrastructure. ASC’s sole job was to revision control configurations, perform deploy and rollback operations, and perhaps do some auditing. But it was prescriptive, not descriptive. Most of the time I spent monkeying with ASC was debugging how it was doing things.
Enter Chef (or Puppet, LCFG, cfengine, BCFG2; pick your poison). These are all configuration management tools that allow you to describe your infrastructure in a fourth-generation language (4GL) way. You describe the features that certain hosts should have, and the tools, using canned recipes, makes it happen. (“Make me a MySQL server,” for instance.) Another advantage of these tools is that they (can) keep track of the state of your infrastructure, and you can query that database to make decisions about new deployments. “How many MySQL servers do I have?” for example. Or even “Which node is the MySQL master?” and then kicking off another job on a new MySQL slave to automatically start replicating from the right server.
Had it not been for the development of IaaS — infrastructure as a service — everything that I’ve told you would not be particularly noteworthy. But IaaS, or “cloud computing”, now allows anyone to provision new (virtual) servers inexpensively. No more waiting around for the system administrator to order a couple servers from Dell, wait a few weeks for them to arrive, rack them up, configure them, etc. Developers, armed with a tool like Chef and its huge cookbook of canned recipes for making many standard infrastructure components, can fire up everything they need to support their application themselves. Therein lies the demise of system administration as a standalone profession and the rise of “devops”.
I admit that when I first heard the concept of “devops”, I snickered. “Give developers the keys to the infrastructure and they’ll surely break it beyond repair and expect the sysadmins to fix it,” I thought. But it’s finally dawned on me that “devops” isn’t just some buzzword concept that someone has thought up to make sysadmins’ lives hell. It’s the natural evolution of both professions. By bringing development and system administration closer together, it does two things. First, it makes developers operationally accountable for their code, because they are the ones that get paged in the middle of the night, not some “operations team” upon whom they can offload that responsibility. And secondly, it makes those on the systems side of the house better at their jobs, because they can use newly-acquired programming skills to manage infrastructure resources in a more natural way.
So will IaaS and sophisticated configuration management tools kill the system administrator? I believe so — but that’s not a bad thing. System administrators have got to stop thinking of servers/disk/memory/whatever as “their resources” that “they manage”. Cloud computing has shown us that all of that stuff is just a service, dedicated to nothing more than serving up an application, which is what really matters. If sysadmins want to remain relevant, they’ll get on board and start learning a bit more about programming.
I think predicting the death of traditional systems administration is premature – devops is great, and Chef et al are essential pieces of software if you manage any kind of modern application deployment – but who’s managing the physical hardware at your IaaS provider?
Who’s responsible for the company’s internal network, and making sure the VPN works, and that the office internet connection works?
When you reach a certain size that paying the IaaS premium no longer makes sense, who’s going to manage your colocated hardware? Or, indeed, who’s going to decide when it makes sense to make that change?
The role of a sysadmin is definitely changing, but that doesn’t mean that the traditional role has gone away – tools like Chef are force multipliers to the traditional sysadmin, making it possible for one person to manage more systems than before, but they’re not a replacement for the traditional sysadmin.
John, note that in big companies the role of managing hardware is already completely abstracted away from the sysadmins. At my large internet company if you have a hardware-related issue you file a ticket and wait for Siteops to deal with it.
I think we are already well on the path towards the ‘death of the sysadmin’. Maybe we’ll still call ourselves sysadmins but clearly our role will be very different.
Great post, Julian!
As a sys admin who’s been working in the trenches for about 15 years I can definitely relate to the shift in mindset (as articulated by Julian). I find that as budgets continue to get squeezed and OPEX continues to be a “dirty” word for IT management, the traditional role of the sysadmin has to change to that of systems facilitator (IaaS as a case in point). In my shop we design platforms that we can then use to rapidly deploy virtual systems for our developers, etc (mix of x86 gear, RISC gear) and despite our headcount not being augmented in face of a 4x growth in infrastructure over a 4 year period, we continue to innovate and adapt to newer ways of skinning the proverbial cat.
So we see ourselves as developers of systems and try and write software to help make the process of developing systems a easier, more seamless experience.
Puppet,cfengine and chef are good tools to manage servers, but I agree with Jonh that this predict is premature. These tools do not support all the OS: (AIX support is not so well), they can’t manage some resources. so unless you are just using Linux , still need to do regular sysadmin work.
Great post. Great comments.
I agree with John. Somebody’s still got to be the cloud wrangler on the infrastructure side – but there will be less of us. I think the squeeze you’re going to see can be perceived along the lines of the ITIL service lifecycle. As an AIX consultant I’ve already seen this as I was spending more of my time on Design-Transition and leaving Operations to… people who charged a lot less per hour, F/T, employed by the client, with maybe an operator certification.
The market is going to squeeze the profession, but the systems babysitters are going to be the ones finding it difficult to transition.
That said, in the ERP and Financial unix worlds things are not going to change as fast. Most AIX operators and admins I know would be terrified to use a tool like Chef to mess with the ODM for numerous support-agreement-related and voodoo-related reasons.
One thing has become abundantly clear from what I’ve seen and that is that developers have zero, naught, not a shred of an idea of compliance, ITIL, best practise or anything that a modern systems administrator maintains. It’s all well and good to say duh, I’m a devop and I can install a user via a command. But it’s entirely another to program business rules, governance, security and all the things that big organisations have to cater to, into an automation tool and even harder still to get a developer to care about implementing any of that.
Puppet, in my mind, is a complete step in the wrong directions. But I guess that makes perfect sense when you look at the way IT has progressed over the past 40 years. Instead of anything becoming easier, faster, better, code got slopier, faster to market, less perfomant and more marketable. That is what puppet is about. Marketing. Changing the landscape in order to shift funds into the pockets of Puppeteers. It’s a crap model, for a crap time in computing. Within 20 years the IT crowd will wake up and return to do one task and do it well, to optimisation, performance and fault free operation. oo will be gone, in it’s current form at least, and all this nonesensical IT for the drone will go back to using intelligence to advance for the greater good. Something puppet simply does not do.
I can’t disagree more with all the things that you said here. It’s a developer-centric world, because developers make features and products that people want to buy, and traditional sysadmins that simply “operate systems” do not. And I also disagree with your broad, sweeping generalizations that basically amount to “developers are idiots, only sysadmins are the rockstars”. If you set up the incentives right for developers, they *will* worry about all the things you mentioned.
But somehow I feel like anything I say isn’t going to particularly be convincing here.
Sysadmins are far from being dead or replaced by “Devops” folks. Pointing and clicking a few times to instantly spin up a new server or database for the Devops guys might be a quick way to get a server online for them to push code. Building and deploying servers are the least taxing task a sysadmin has to deal with. Patch management, security scanning and risk mitigation, disaster recovery plans and testing them. Proper documentation. Firmware updates, hardware failures, network devices. A sysadmin does many things, many behind the scenes to keep things working. Someone’s gotta run all those blade chassis, SAN array’s, networking firewalls, routers, load balancers ect….you think a guy who writes apps all day has the time or will to attend to the mundane care and feeding of a large I.T. system? You think they’ll know what to do when part of that converged cloud platform has a hardware failure and alarms are going off all over the place? I don’t think so. Sysadmins will always exist. We just may all end up in large datacenters supporting clouds.