The Sysadmin Is Not Dead Just Bad Ones Are Being Recogonized

This is a blog post in reply to a recent post discussion on LinkedIn entitled “The Sysadmin Is Dead” while I don’t entirely disagree with all of the content in the post there seems to be these system engineers who simply suck and apparently want to configure every config file by hand and do manual installations of every single server they have to support. I’m not sure who trained these people but those people sucked too. I guess I was lucky because when I was mentored I was taught “A Good System Admin Is A Lazy System Admin” and that was over 10 years ago.

I remember when I was 21 years old I was a systems administrator at CSG Systems, Inc and I was mentored by a great team of individuals. They took me under their wing and helped me build a strong foundation as a system admin. My first task when I started was to learn how to jumpstart (yes this was in the Solaris days) servers and how we needed to make sure every single system was built the same way. Then I was tasked with taking over the backup infrastructure. I can remember running out of tapes and having to manually expire them and one of my mentors reminded me about automating everything, soon I had a suite of scripts that could predict months ahead of time when we would run out of tapes so I could bug my manager that we needed to put in an order. I remember how the application and database teams would bug me every day about the success or failure of their systems backups, so being the “Lazy” system administrator that I was I scripted up a solution that emailed them a simple report every day about failed and successful backups. Then came the problem of sending tapes off-site we had a NOC of junior personnel who completed basic tasks for us such as sending tapes off-site, I had to go to the datacenter with them and figure out what tapes where ready for off-site shipment this usually took a long time (this was before netbackup had the bolt on automation suite, I can’t even remember the name of it because I had already written all the functionality and more on my own) so I scripted up a solution where the NOC received a daily email with tape numbers and expiration dates then they would go to the tape library and open the tape caddy and all the tapes which needed to be sent off-site where waiting for them to pull out.

Next came all that manual work involved with adding the right packages and manual completing all of the security requirements set by the security manager. What a pain in the butt. Well guess what back then there was no Chef or Puppet so I leveraged what was available, I took an opensource application called Titan (hosted by fish.com which is now a pet store) and added a ton of scripts which did everything from add users, enable NIS, installed the monitoring software, etc… All you had to do was specify the “Type” of system you wanted webserver, database server, dmz proxy server, and all the heavy lifting was completed by the script. Not only did Titan configure the system and lock down permissions to be compliant with our security officer but it also ran in cron every day and emailed me if a system was “out of compliance”.

Soon I was off into the consulting world and I remember contracting out to a company where I assisted them in building out their Linux infrastructure. I remember working with one of their Active Directory administrators (apparently in the Windows world people are tasked with one part of the system to manage) anyway I remember he asked me why I was able to bill out $120/hr for my consulting services and I simply explained that I was capable of doing more then adding users and groups to Active Directory not only was I capable of building out an Active Directory infrastructure but I also designed backup solutions, self-healing monitoring solutions, automated system deployments and configurations, I knew VMware inside and out and was always leveraging new technology to make my job easier, I explained I’m the quintessential “Lazy” systems engineer. I rather go to the movies in the middle of the day and have my servers fix them selves. For instance if the JVM crashed for the 4th time that day the monitoring system will recover the service and email development information about the crash.  Now don’t get me wrong I’m all for the “DevOps Movement” but I’m not convinced its a movement yet its just that people are starting to define what a good systems engineer is vs what a dumb ass is.

I can’t tell you the number of people I have interviewed with either me as a candidate for their company or them as a candidate for mine where they told me that they like to drop the DVD into each new server and manually decide how that system should be built.  When people tell me this and I am the interviewee I immediately tell the person I am not a fit for their company because I couldn’t imagine a world where I actually have to burn a DVD with the latest RHEL or CentOS version on it and complete the installation by hand.

While I want to be clear that I am 100% into and for DevOps there is nothing that burns my hide more then people who walk around telling me that this concept of infrastructure as code and the devops movement is some new idea and how its the “future” of the system administrator. I laugh at them knowing their ideas come from some blog written by John Allspaw and they actually have such limited experience that they are finally seeing the light. Don’t get me wrong I love using Puppet and Fabric and a wide range of other tools which now help us systems administrators have a consistent framework by which to leverage our infrastructure automation but lets be real here, if only now your thinking that system automation and infrastructure as code is a new concept well welcome to the world of the “Good” systems engineer you now can officially stop being a dumb ass!