System administration is a tough job these days. The toolchain changes frequently, the basic technology is moving fast, even in the age of Linux 3.x series. RedHat does a great job by introducing new features into their Linux distributions such as SELinux, upstart – systemd just to name two of the changes. The IT systems seem to grow more than the common techniques of system administration are able to deal with. In the old days few shell scripts and ssh were able to cope with the requirements, today tools like Fabric, func, puppet or chef seem to change the art of operations radically.
This is a start of a small series of blog articles about managing infrastructure in 2012. I’ll adapt some of the development methods and explain how we can use them to build more reliable systems with better teams.
Sidenote
By the way – is that really true or is it just something which is being hyped by blogs, the devops community and the agile movement of lean. Seriously, the first release of cfengine happened in 1993. I used cfengine in my first project in 1999, this was a long time before puppet has been released. So, let’s be serious – the tools were there already, however, noone used it in the way we are thinking of infrastructure today.
What has changed?
So what has changed recently in the sysadmin world? Maybe it’s the way the business world pretends to work these days. Scalability was always an issue, however, with public APIs, realtime trading systems and the hype of cloud based systems, scalability became more and more important. We can’t just afford to lack performance on our websites, be offline or fail to answer requests.
Does your team scale?
Due to vastly improved requirements to the infrastructure the tools became sharper and the sysadmins became smarter. Now they are being called devops and have to deal with everything. Everything is about scaling things but does your team scale, too?
From my experience I can tell that this is an serious issue. Your team won’t be able to scale with the requirements, the tools and the change of methods unless everyone is higly motivated and open for changes. Usually it’s just not the case and things are likely to move slower if you try to catch up.
Don’t call for heroes!
There are several different methods to deal with the situation. The far worst thing is the call for heroes. Those guys are smart, will code everything in puppet in a few minutes, install new servers in seconds and will take over. While this is a solution for a short time, you have to make sure to spread their knowledge and their philosophy to build systems. I was working for a company and became one of their heroes. I’ve built several systems and saved the company a bunch of trouble. However, noone was able to understand the systems. When I quit the company I had to hand over the systems and it was an interesting experience. Explaining what was clear to me and what was new to them, took some time. Now I know how hard it actually is to write documentation as a developer about your code.
Pair your admins and team up
Basically, we have to accept our new tools such as RHEL Satellite (Spacewalk), puppet and clouds/virtualization. We are still very valuable, there is no reason to think why sysadmins are no longer needed. We are needed – however, we need to adapt some parts of our thinking.
Infrastructure work is no longer mucking with bare metal servers, it’s about to manage an infrastructure in all its glory. Virtual servers, cloud instances, bare metal servers, virtual switches and real firewalls. It’s all ours!
Let’s take the challenge!