However, I would like to use SSL for the http transport and I found some problems with this. The web server srv.pkgbox.de is using a certificate which has been signed by CACERT. After I imported the CACERT root certificates into my web browser everything was fine and the small lock symbol was closed. Easy. However using hg clone to get a copy of my new repository threw the following error:

$ hg clone https://jbond@srv.pkgbox.de/hg/world-domination-project
abort: error: _ssl.c:503: error:14090086:
SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Huh. There was a problem with the certificates. The solution is quite simple. You have get the root certificates from CACERT and store them in a single file in the PEM format. CACERT provides two certificates, a root certificate and a class 3 certificate, we have to get both of them. So, let’s do some work in the terminal window and set up everything for certificates.

$ mkdir ~/.certs
$ curl -s -k http://www.cacert.org/certs/root.crt \
   http://www.cacert.org/certs/class3.crt > ~/.certs/ca-certificates.crt

Ok, now we have the root certificates from CACERT and concatenated them in a single file inside the subdirectory ~/.certs/. Let’s use the openssl tools and check if our certificate chain is valid.

$ openssl s_client \
   -CAfile ~/.certs/ca-certificates.crt -connect srv.pkgbox.de:443 < /dev/null | \
   grep Verify
   [...]
   DONE
   Verify return code: 0 (ok)

Cool – openssl verifies the certificate and everything is working (for openssl). Now we only need to tell Mercurial to use the file with the certificates in (~/.certs/ca-certificates.crt). Mercurial uses a configuration file named hgrc. This file can either be system wide in /etc/mercurial/hgrc or can be used on a user basis in ~/.hgrc – or inside any .hg directory of your project. Let’s use the user configuration file as I want to set it up for all my repositories.

$ cat <<_EOF_ > ~/.hgrc
[web]
cacerts = ~/.certs/ca-certificates.crt
_EOF_

Voila, let’s check if everything is working for us and try to clone our sources again.

$ hg clone https://jbond@srv.pkgbox.de/hg/world-domination-project
http authorization required
realm: Mercurial Repositories
user: jbond
password:
destination directory: world-domination-project
requesting all changes
[...]
updating to branch default
16 files updated, 0 files merged, 0 files removed, 0 files unresolved

Hej, it’s working. Time to get some more work done.

The Perl packages really need some love there – some are horribly outdated and broken. There seem to be Perl packages in pkgsrc which are older than the versions which are shipped with the Perl core distribution. It’s alot of work.

Here what I will do in the next days – let’s see how far I can get:

• update the packages (at least the most)
• clean up the licenses
• check the dependencies inside the Makefile.PL (script)

snowflake:p5-Archive-Tar rhaen$ cvs ci ../../doc/CHANGES-2012 
cvs commit: Up-to-date check failed for `../../doc/CHANGES-2012'
cvs [commit aborted]: correct above errors first!

Can i haz  working version control system?

./configure && make && make install

approach. Just don’t do it – it’s nasty. Build a package with your favorite package manager and deploy it. As I am working with RHEL based systems, the package format of my choice is RPM and I love it. As I am in the same situation such as you – I decided to build a public repository.

Voila – another repository to add – check out my small project page about this. For the unpatient people here is the quick way:

yum install http://rpm.pkgbox.org/pkgbox-repo-6.noarch.rpm

That should do the trick. Even for i386 hosts – usually I am building for x86_64. However, I am using mock to build the packages, but thats a different story.

Enjoy the RPMs (and leave some feedback)

btw, this is just my personal repo – don’t expect too much – it’s the stuff I need for my daily business such as:

• The wonderful Perl MongoDB driver by Kristina Chodorow
• The skyrocketing collectd statistics daemon by Florian octo Forster
• The next generation Perl webframework Mojolicious by Sebastian Riedel

This is a start of a small series of blog articles about managing infrastructure in 2012. I’ll adapt some of the development methods and explain how we can use them to build more reliable systems with better teams.

Sidenote

By the way – is that really true or is it just something which is being hyped by blogs, the devops community and the agile movement of lean. Seriously, the first release of cfengine happened in 1993. I used cfengine in my first project in 1999, this was a long time before puppet has been released. So, let’s be serious – the tools were there already, however, noone used it in the way we are thinking of infrastructure today.

What has changed?

So what has changed recently in the sysadmin world? Maybe it’s the way the business world pretends to work these days. Scalability was always an issue, however, with public APIs, realtime trading systems and the hype of cloud based systems, scalability became more and more important. We can’t just afford to lack performance on our websites, be offline or fail to answer requests.

Does your team scale?

Due to vastly improved requirements to the infrastructure the tools became sharper and the sysadmins became smarter. Now they are being called devops and have to deal with everything. Everything is about scaling things but does your team scale, too?

From my experience I can tell that this is an serious issue. Your team won’t be able to scale with the requirements, the tools and the change of methods unless everyone is higly motivated and open for changes. Usually it’s just not the case and things are likely to move slower if you try to catch up.

Don’t call for heroes!

There are several different methods to deal with the situation. The far worst thing is the call for heroes. Those guys are smart, will code everything in puppet in a few minutes, install new servers in seconds and will take over. While this is a solution for a short time, you have to make sure to spread their knowledge and their philosophy to build systems. I was working for a company and became one of their heroes. I’ve built several systems and saved the company a bunch of trouble. However, noone was able to understand the systems. When I quit the company I had to hand over the systems and it was an interesting experience. Explaining what was clear to me and what was new to them, took some time. Now I know how hard it actually is to write documentation as a developer about your code.

Pair your admins and team up

Basically, we have to accept our new tools such as RHEL Satellite (Spacewalk), puppet and clouds/virtualization. We are still very valuable, there is no reason to think why sysadmins are no longer needed. We are needed – however, we need to adapt some parts of our thinking.

Infrastructure work is no longer mucking with bare metal servers, it’s about to manage an infrastructure in all its glory. Virtual servers, cloud instances, bare metal servers, virtual switches and real firewalls. It’s all ours!

Let’s take the challenge!

Kanban
Author: David J. Anderson
Recommended: yes
Amazon:

• €39,50 (Kanban Book)
• €14,44 (Kanban Kindle Edition)

This book is a goldmine of information about all questions of Kanban. David explorer Kanban for software development. He is the guru without any question. I got his book (signed by him) in one of his courses. This book is just outstanding. A review will follow at later time.

Personal Kanban – Mapping world | Navigating life
Author: Jim Benson, Tonianne DeMaria Barry
Recommended: yes
Amazon:

• €19,00 (Personal Kanban Book)
• €7,21 (Personal Kanban Kindle Edition)

Getting started with Kanban
Author: Paul Klipp
Recommended: HIGHLY!
Amazon:

• €1,03 (Getting Started with Kanban Kindle Edition)

That’s an incredible short introduction into the Kanban methodology. Ok, the book features a website which provides a Kanban system as SAAS. However, everything is well explained independently. If you need to get grip on Kanban fast, this is your book to read!
Please note – there is also a free PDF version. The Kindle version works perfect and it’s cheap – no reason to work with the PDF. Please look at the PDF version here: Getting started with Kanban.

In the old administration days one sysadmin was responsible for the installation of a server and the configuration system. Right now one sysadmin uses a central management server and fires up several virtual boxes or cloud instances with a few commands. But what about the monitoring? Is the monitoring included with the setup of your cloud instances? Lucky you. Usually those kind of tasks are only for brave sysadmins – and they are getting fewer every day.

Graph drawn by rrd2graph from collectd data

I gave a talk on Tuesday about a more flexible toolchain for monitoring than the common used nagios setup. I had some good experiences with this toolchain at the company I am working for and it seems to scale reasonably well.

collectd is small daemon which collects the data on every instance/server and stores the information in rrd databases using librrd or via rrdcached. The perfomance data is being pushed to central servers, too. I like to call them collectors. They gather all the data from the different servers/instances. The advantage is clear – nagios can get all the values from those databases from every server at one place. This is not only an advantage for the firewall, it also saves the nagios server alot of work. The graphs are being drawn using rrd2graph (see screenshot) which is part of the n2rrd project.

The best thing about using collectd is it’s way of configuration. I suggest to setup three different configuration levels. A base configuration which is exactly the same on every host. It just has the most basic configuration in. A second level called environment configuration deals with networking stuff like who is the collector server in a certain zone. The third layer is called service configuration. This layer deals with services running on the server such as mysql, httpd, nginx, etc.

If you are using a configuration management tool such as puppet or RedHat Satellite it’s an easy task to build classes or system service groups which care for the configuration layers. Use RPMs during the kickstart process to install the collectd binaries and provide them with a sane first /etc/collectd.conf.

collectd provides a nice plugin structure with about 90 pre-made plugins and it’s easy to extend by using it’s Perl, Java, Python apis.

Oh, did I mention? This works for Windows based servers, too! Please check out the slides for more information. The first link is a mindmap in xmind format with a small management summary. If you are interested in the slides of the presentation, please use the second link which is in PDF format.

Monitoring 2.0 – Presentation Summary (xmind – Mindmap)

Presentation about monitoring Unix environments with modern tools (pdf)

That’s a difficult question. Having interviews, talks or even exams might not give the right answer but you will get a certain feeling about someone and his skills. However, even the old geeks tend to be silent when Gabor Szabo (blog) is asking stuff about regular expressions in his talks on conferences. That’s interesting. People who are able to do the most complicated stuff with regex are quiet when they are asked simple regex. Is that the kind of people we are referring to as DevOps? Probably not.

So how do we find those multitalented people who are using Perl for sysadmin stuff, understand the complexity of J2EE and its architectures and are communicative to their colleagues – even when they can’t provide the answer at this time (referring to the regex problem). Actually, I don’t know. During interviews I have used the following simple questions to get a feeling about the Perl knowledge of the candidate:

1. What’s the correct way to test if two scalars are the same – string comparison?
2. What does the keyword “say” do?
3. What is the meaning of the code fragement $c = $a // $b ?
4. What is the purpose of the command line tool prove?
5. What is the taint mode of Perl?

Those are just 5 simple questions. Let me explain why I like to ask them and why I think they can give you more insights into the job candidate.

Question 1 – Tries to sort out if the person is actually using Perl. The most common answers for this are: “==”, “=” or “$string1=~ /$string2/”. Once again, Gabor has some answers for this in a short YouTube video – Comparing scalars. If the candidate is very nervous and fails instantly, be nice and start a dialogue. Discuss the different options – if he has no clue at all, he will fail anyway in this question.

Question 2 – Asking about “modern” Perl knowlege. Most of the Google code snippets don’t show results about the keyword say. However, if the person is using a more current version or if he read the history he knows about the keyword.

Question 3 – Tricky one and might even be complicated for professionals. Code constructs like this one are common:


if ( ! defined $a ) {
$b = $a;
}
else {
$b = "Hello world";
}


Haven’t seen constructs like this – check your legacy Perl code. There are nice examples in the perlop documentation it. This is something special, however, there is a reason why this operator exists – if you want someone working for you who knows how to use the tools – check if he really knows them.

Question 4 – Huh, don’t know the answer? That doesn’t matter – really. A lot of people are using Test::More for a long time and know about TAP. If the person knows TAP and Test::More already, it’s ok that they don’t have the meaning of prove handy. If they don’t know anything about TAP or Test::More, you will have another hint of their level of Perl knowledge. Do you expect a DevOp person to be the glue between development and sysadmins without knowledge of testing stuff?

Question 5 – wow, that’s legacy stuff. Really, I mean it – what was the last time that you’ve use the taint mode for old CGI stuff? But is it such a bad idea to use the taint mode? Actually, the taint mode is something unpopular today. I wrote several scripts some decades ago which used the taint mode. I will fail horribly when I am trying to use the taint mode today – however, understanding about the history stuff is never a bad idea.

Those are just some questions which might help you to check the knowledge of Perl of a candidate. You’ll understand their ways of thinking – their way to solve problems. If you can use the interviews to ask them face to face – that’s awesome and might give you an even deeper insight. If you are asking these questions on the phone – be nice! They are not as easy as you might think now when you are relaxed and not in a situation of job application. Help them on their way – test their knowledge, look at their problem solving skills and motivate them to pass the tests.

When you want to attract DevOps – try to be as nice and communicative as you expect them to be!

Here are some simple facts about github usage which might be helpful:

Fact I: Use github right

github is not just a tool or a place to put a clone of your git repo on. If you decided to put things on github, use github as a platform. Be polite, answer to issues and decide about the incoming pull requests. Some projects use github as a place for backing up their repositories. “A copy of the original repo is on github” – is a common term. Forget about it. If someone forks and contributes to your project make sure to answer appropriately and fast. Those people already forked the repo, downloaded everything, looked into the sources and put some effort into it. They don’t deserve a break about 1-2 weeks for contributing. Firing a patch to a mailing list is just easy. Working out a pull request is much more complicated. You probably have to rebase your stuff on top of the repo, merge stuff..you’ve figured already.

If you don’t encourage contributions – don’t use github.

Fact II: Use github for communication

Denying a pull request isn’t bad – it’s just code. Be nice and explain why you don’t accept the pull request. I’ve sent several pull requests to @kraih for Mojolicious. He rejected most of them, however, I always got a nice reply. Sometimes he even took my commit and changed it a little bit. I was visible as the author and he took the rule as committer. git is something social politcal statement alike, too and github visualizes this nicely.

Use github to encourage contributions!

Fact III: Wiki your community

The wiki on github is very basic. However, it provides some useful features which are helpful for the Perl people. You can choose the format in which the pages of the wiki are being created in. Using the pod format allows people to clone your wiki and read it in offline mode using perldoc. This is an awesome and in my opinion underrated feature. The only problem with the wiki on github is that you have to enable write access to the project repo to allow users to push content. I’ll hope they’ll fix it one day.

The wiki is usually driven by the community. The headline person of the project can contribute and give insights into different code parts and can share his vision. The community will care for examples, discussion and keep the wiki up to date. Therefore the wiki needs to be writable for everyone. I am serious, open the wiki for everyone! All the content is stored in git anyway so you can easily revert things if they got broken.

Be spirited, share your vision and be open to your community!

Let’s assume we already have a project named “myproject“. As I am a Perl programmer we will use Perl sources and a perlish layout of the project. However, this should work for every project! As you see in this screenshot I’ve some source layout (given by common Perl nature) and a “.git” directory. I’ve worked with the sources and ran git init and added all the files to the repo.

project toplevel folder with .git directory

Therefore we have an existing repo which is fine for local work, however, sharing via HTTP requires some steps:

• Clone the existing repo to a bare repository
• Enable the post-update hook
• Setup the apache to share the repo

Let’s start with cloning the repository to a bare repo. A bare repo is just a git repository with a different layout. You won’t see any active checkouts aka sources, you’ll only find the structure of the “.git” repository in it. As we used the directory “/Library/WebServer/Documents/repo” as the base directory for my last article we’ll continue to work at this directory base. We will clone our small project “myproject” to “/Library/WebServer/Documents/repo/myproject.git”. This allows to follow the later steps in the setup article.

We start inside the directory of our project, so the “.git” repo is on the top level.

$ sudo git clone --bare . \
   /Library/WebServer/Documents/repo/myproject.git
$ sudo chown -R _www:_www \
   /Library/WebServer/Documents/repo/myproject.git
$ cd /Library/WebServer/Documents/repo/myproject.git
$ sudo -u _www mv hooks/post-update.sample \
   hooks/post-update
$ sudo -u _www hooks/post-update

From this point you can follow the old article about “Set up your Apache server for sharing”

Enjoy your new git experience!

$ morbo [options if needed] <scriptname>

Everytime you change the script, morbo will detect this and will reload the application. Best part of it, it will detect syntax errors and warn you about them. This is very helpful for me as it separates a few steps. First you develop the Mojolicious::Lite application, then you’ll save it, morbo does the syntax check for you and you see if things are broken from the code side, next you’ll test the web application with a browser and check if everything is working there. To run unit tests all the time is self explanatory.

So yes, I am a fan of it. Oh, there is something even nicer about it. morbo doesn’t only check the script for changes, it subscribes automatically to the lib and to the templates directory, too. If they don’t exist, morbo will just do the right thing. So once you’ve a basic Mojolicious::Lite application and decide to inflate it from one file to a directory structure your setup doesn’t need to be changed. That’s cool. Speaking from the sysadmin side of life: It’s a horror to track changes in your deployment configuration. If you are part of the development team you might find ways to deal with it. If you are a sysadmin which only cares for the platform you are doomed.

Here is the simple way I like to work with morbo in my MacOS X environment. I am doing my development stuff in a subfolder of my home directory. I’ve put the folder as a subfolder of my Library folder so I won’t see it all the time. Nothing is more annoying as millions of folders around! Please note: The Library folder is not visible by default, you might want to create the structure in your home directory, if you are uncomfortable with the Unix shell.

You can unhide the Library folder in your home directory by running

$ chflags nohidden ~/Library/

in the Terminal.app.

I’ve used perlbrew to install Perl version 5.14.1 in my home directory. This way I can install modules and play with them without messing around with the MacOS X system stuff. As I’ve said before – I like to keep things separated – call it sysadmin paranoia. Let’s create the basic structure.

$ mkdir -p ~/Library/Mojolicious/{external,external/repo,log}
$ mkdir -p ~/Library/Mojolicious/{static,templates}
$ cd Library/Mojolicious
$ mojo generate lite_app default.pl

Voila, we now have a basic directory structure and a standard Mojolicious::Lite application named default.pl. Let’s see what else do we have.

external
directory for external resources, I clone mojo from github into this directory. I use it as reference as well as for installations of the latest code of Mojolicious.
external/repo
directory for my personal git repositories which are Mojolicious related
log
log directory for Mojolicious (default setting)

static
directory for static resources such as images, css, js files
templates
template directory for Mojolicious applications (default setting)

Now that we have a basic structure and a basic setup we should look on the automatic startup of the morbo server and our just created default.pl Mojolicious::Lite application.

MacOS X uses a special daemon for launching services which is called launchd. The daemon is controlled by a command which is named launchctl. launchctl will be used to load a xml style description of the service into the daemon and start it. You can specify custom services and you’t have to be the user root to work with the daemon, that’s nice. So, let’s look into the xml service file for our morbo service.

launchd property file for morbo - the Moljolicous web server (see gist)

That’s an image, click to enlarge it. I’ve used it for better visibility, you can find the full source file in the following gist.

Here is a quick review of the important parts:

<key>Label</key>
<string>us.mojolicio.default</string>

Every application/service needs to have it’s own label. The label is used to identify the service at a later point. Use a dsl syntax to keep things sorted. My own applications will be named de.pkgbox.mojolicious.<name>.

<key>ProgramArguments</key>
<array>
    <string>morbo</string>
    <string>–listen</string>
    <string>http://*:8080</string>
    <string>/Users/rhaen/Library/Mojolicious/default.pl</string>
</array>

This is a little bit tricky. I don’t need to specify the full path to morbo as launchd will use my user environment to run the ProgrammArguments. However, I have to specify the full path to my Mojolicious::Lite application named default.pl. I am using some arguments to morbo to keep it more usuable for me but let’s work down the way. Usually you have to use a Program key and specify ProgramArguments as an array for more arguments. If you discard the Program key it will use the first element of the ProgramArguments as Program key. Simple.
I like to have morbo running on port 8080. Maybe during my development phase I want to run another Mojo application on the default port 3000. That’s just a good way to make sure that normal development doesn’t interfere with your basic environment.

Ok, here is the next part of the setup:

<key>RunAtLoad</key>
<true/>
<key>WorkingDirectory</key>
<string>/Users/rhaen/Library/Mojolicious</string>
<key>StandardErrorPath</key>
<string>/Users/rhaen/Library/Mojolicious/log/output.log</string>
<key>StandardOutPath</key>
<string>/Users/rhaen/Library/Mojolicious/log/output.log</string>
</dict>

RunAtLoad means that we want to start the morbo server right from the start. We don’t want to run it on demand neither we just want to register it, we want it running! The WorkingDirectory is an important setting. launchd will chdir to this directory before running the Program and the ProgramArguments. That’s needed to get the correct environment for Mojolicious for the log, static and templates directory. Last but not the least we use a StandardErrorsPath/StandardOutPath for some basic log files of the launchd daemon. Does something fail? Was there an error? Did morbo crash? We’ll find the information in this file!
Please note: There is an optional setting named KeepAlive which will try to restart the morbo server once it fails. This can be set, however, I would like to know when morbo crashes and why it crashes. That’s why I am not using this setting.

That was alot of stuff! Store all the information above in a file named “us.mojolicio.default.plist” and save it in the external directory of our basic structure. I like to keep the main copy there – just for separation, you know sysadmins paranoia.

In order to activate everything we need to run a few commands. First we need to place a symlink inside the ~/Library/LaunchAgent folder. After that we need to register aka load the file and we are done.

$ mkdir -p ~/Library/LaunchAgents
$ cd ~/Library/LaunchAgents
$ ln -s ~/Library/Mojolicious/external/us.mojolicio.default.plist .
$ launchctl load us.mojolicio.default.plist

Why do we use a symlink? Why don’t we just place the file in the correct place? The answer is simple. We kept everything Mojolicious related together. We can run git init inside the Mojolicious folder and have everything version controlled. One place – one repo, sysadmin paranoia…you know.
By loading the configuration file we informed launchd about a new service and launchd already took over. The new service has been started and the morbo server is serving default.pl at port 8080. Why don’t you fire up a web browser to check it?

Uninstalling everything

Got lost? Want to get rid of everything and start from scratch? That’s simple. First we need to unload the morbo service from launchd. Next we’ll remove the symlink and delete the Mojolicious folder. Voila – finished. Dear developers, that’s sysadmin style. Clean up after your play – dear sysadmins, start coding!

$ cd ~/Library/LaunchAgents/
$ launchctl unload us.mojolicio.default.plist
$ rm -rf ~/Library/LaunchAgents/us.mojolicio.default.plist
$ rm -rf ~/Library/Mojolicious

Make sure to leave some feedback!