pkgbox

Feed Rss

This is more a rant….or probably the answer why sysadmins should not work on Win32.

[user@***:~]$ mojo generate lite_app rest_auth.pl
      1 [main] perl (2260) C:\dev\cygwin\bin\perl.exe: *** fatal error - cygheap base mismatch detected - 0x612768B0/0xDC68B0.
This problem is probably due to using incompatible versions of the cygwin DLL.
Search for cygwin1.dll using the Windows Start->Find/Search facility
and delete all but the most recent version.  The most recent version *should*
reside in x:\cygwin\bin, where 'x' is the drive on which you have
installed the cygwin distribution.  Rebooting is also suggested if you
are unable to find another cygwin DLL.
Segmentation fault
[user@***:~]$
[user@***:~]$ mojo generate lite_app rest_auth.pl
  [exist] /home/user
  [write] /home/user/rest_auth.pl
  [chmod] rest_auth.pl 744
[user@***:~]$

 

Oh yeah. If you expect to establish a devops culture in your company, give them the tools for it. Oh, and care for their mindset.

I really love your stuff, your design, your hotdogs and your stores (without mobile network). Who remembers the good times when the sysadmin life starts – probably in a startup company? No money, just a small apartment without any stuff inside – but a warm place to rest after the fights with servers, OS and Linux kernel updates. Sometimes we sat round the table when the younger colleagues told us that they move to a new apartment or that they plan to drive to IKEA at weekend and could do with some help.

That time was awesome. Everyone wanted to help and got paid in hotdogs. Eat as many as you like, that’s fine – they are 1.50€ only. Usually you are sick after the first one and so it’s cheap. Maybe because of those memories people won’t forget you. I even know sysadmins who started to name server hostnames after your product names. Like: Kvartal for a managment server, Billy for the file server (or even Samla) – everyone is able to speak IKEA.

 

 

However, after you started to give your products such weird names I doubt that someone will chose your product names for hostnames again. I don’t want to have a server called: Äppelkaka. Sorry, no way! Sounds like: there is more than one way to do it…

 

Please name your products with more care in the future.

 

Regards

Rhaen (Uli)

I run a small web server for my personal things like the rpm packages, some Perl stuff, this blog and some other things. Among these things are the source code repositories such as the git repos and the Mercurial repositories. I had to switch from git to Mercurial as my new employer uses Mercurial and I need some training. Setting up a source code repository which is shared using the web server and the provided hgwebdir.cgi script is easy. I won’t explain all the details in this post, there are several good tutorials out there.

However, I would like to use SSL for the http transport and I found some problems with this. The web server srv.pkgbox.de is using a certificate which has been signed by CACERT. After I imported the CACERT root certificates into my web browser everything was fine and the small lock symbol was closed. Easy. However using hg clone to get a copy of my new repository threw the following error:

$ hg clone https://jbond@srv.pkgbox.de/hg/world-domination-project
abort: error: _ssl.c:503: error:14090086:
SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Huh. There was a problem with the certificates. The solution is quite simple. You have get the root certificates from CACERT and store them in a single file in the PEM format. CACERT provides two certificates, a root certificate and a class 3 certificate, we have to get both of them. So, let’s do some work in the terminal window and set up everything for certificates.

$ mkdir ~/.certs
$ curl -s -k http://www.cacert.org/certs/root.crt \
   http://www.cacert.org/certs/class3.crt > ~/.certs/ca-certificates.crt

Ok, now we have the root certificates from CACERT and concatenated them in a single file inside the subdirectory ~/.certs/. Let’s use the openssl tools and check if our certificate chain is valid.

 

$ openssl s_client \
   -CAfile ~/.certs/ca-certificates.crt -connect srv.pkgbox.de:443 < /dev/null | \
   grep Verify
   [...]
   DONE
   Verify return code: 0 (ok)

Cool – openssl verifies the certificate and everything is working (for openssl). Now we only need to tell Mercurial to use the file with the certificates in (~/.certs/ca-certificates.crt). Mercurial uses a configuration file named hgrc. This file can either be system wide in /etc/mercurial/hgrc or can be used on a user basis in ~/.hgrc – or inside any .hg directory of your project. Let’s use the user configuration file as I want to set it up for all my repositories.

$ cat <<_EOF_ > ~/.hgrc
[web]
cacerts = ~/.certs/ca-certificates.crt
_EOF_

Voila, let’s check if everything is working for us and try to clone our sources again.

$ hg clone https://jbond@srv.pkgbox.de/hg/world-domination-project
http authorization required
realm: Mercurial Repositories
user: jbond
password:
destination directory: world-domination-project
requesting all changes
[...]
updating to branch default
16 files updated, 0 files merged, 0 files removed, 0 files unresolved

Hej, it’s working. Time to get some more work done.

Updating p5-* packages in pkgsrc

01.27.2012, Comments Off, NetBSD, Perl, Unix, by .

I am updating Perl packages in pkgsrc right now. pkgsrc is the package management system from the NetBSD project and it is great. It works on different platforms and different operating systems such as Linux, Solaris and even on FreeBSD.

The Perl packages really need some love there – some are horribly outdated and broken. There seem to be Perl packages in pkgsrc which are older than the versions which are shipped with the Perl core distribution. It’s alot of work.

Here what I will do in the next days – let’s see how far I can get:

  • update the packages (at least the most)
  • clean up the licenses
  • check the dependencies inside the Makefile.PL (script)
As I’ve mentioned on a talk – it’s not only fun to maintain Perl packages for a package distribution. But it seems to be fair – compared to someone who follows the KDE packages :)
Ah folks – here is something which I don’t like:
snowflake:p5-Archive-Tar rhaen$ cvs ci ../../doc/CHANGES-2012 
cvs commit: Up-to-date check failed for `../../doc/CHANGES-2012'
cvs [commit aborted]: correct above errors first!

Can i haz  working version control system?

Everyone needs software which is not in the repositories. That happens from time to time and usually you end up building the sources by hand. Especially when it comes to Perl modules. However, if you want to install the stuff on more than one server you won’t take the usual:

./configure && make && make install

approach. Just don’t do it – it’s nasty. Build a package with your favorite package manager and deploy it. As I am working with RHEL based systems, the package format of my choice is RPM and I love it. As I am in the same situation such as you – I decided to build a public repository.

Voila – another repository to add – check out my small project page about this. For the unpatient people here is the quick way:

yum install http://rpm.pkgbox.org/pkgbox-repo-6.noarch.rpm

That should do the trick. Even for i386 hosts – usually I am building for x86_64. However, I am using mock to build the packages, but thats a different story.

 

Enjoy the RPMs (and leave some feedback)

 

btw, this is just my personal repo – don’t expect too much – it’s the stuff I need for my daily business such as:

System administration is a tough job these days. The toolchain changes frequently, the basic technology is moving fast, even in the age of Linux 3.x series. RedHat does a great job by introducing new features into their Linux distributions such as SELinux, upstart – systemd just to name two of the changes. The IT systems seem to grow more than the common techniques of system administration are able to deal with. In the old days few shell scripts and ssh were able to cope with the requirements, today tools like Fabric, func, puppet or chef seem to change the art of operations radically.

This is a start of a small series of blog articles about managing infrastructure in 2012. I’ll adapt some of the development methods and explain how we can use them to build more reliable systems with better teams.

 

Sidenote

By the way – is that really true or is it just something which is being hyped by blogs, the devops community and the agile movement of lean. Seriously, the first release of cfengine happened in 1993. I used cfengine in my first project in 1999, this was a long time before puppet has been released. So, let’s be serious – the tools were there already, however, noone used it in the way we are thinking of infrastructure today.

 

What has changed?

So what has changed recently in the sysadmin world? Maybe it’s the way the business world pretends to work these days. Scalability was always an issue, however, with public APIs, realtime trading systems and the hype of cloud based systems, scalability became more and more important. We can’t just afford to lack performance on our websites, be offline or fail to answer requests.

 

Does your team scale?

Due to vastly improved requirements to the infrastructure the tools became sharper and the sysadmins became smarter. Now they are being called devops and have to deal with everything. Everything is about scaling things but does your team scale, too?

From my experience I can tell that this is an serious issue. Your team won’t be able to scale with the requirements, the tools and the change of methods unless everyone is higly motivated and open for changes. Usually it’s just not the case and things are likely to move slower if you try to catch up.

 

Don’t call for heroes!

There are several different methods to deal with the situation. The far worst thing is the call for heroes. Those guys are smart, will code everything in puppet in a few minutes, install new servers in seconds and will take over. While this is a solution for a short time, you have to make sure to spread their knowledge and their philosophy to build systems. I was working for a company and became one of their heroes. I’ve built several systems and saved the company a bunch of trouble. However, noone was able to understand the systems. When I quit the company I had to hand over the systems and it was an interesting experience. Explaining what was clear to me and what was new to them, took some time. Now I know how hard it actually is to write documentation as a developer about your code.

 

Pair your admins and team up

Basically, we have to accept our new tools such as RHEL Satellite (Spacewalk), puppet and clouds/virtualization. We are still very valuable, there is no reason to think why sysadmins are no longer needed. We are needed – however, we need to adapt some parts of our thinking.

Infrastructure work is no longer mucking with bare metal servers, it’s about to manage an infrastructure in all its glory. Virtual servers, cloud instances, bare metal servers, virtual switches and real firewalls. It’s all ours!

 

Let’s take the challenge!

Kanban, Devops and agility

01.04.2012, Comments Off, Devops, Kanban, by .

I read alot of different books about Kanban during the last days. Christmas holidays are awesome. I have found several books quite useful – here is a minimalistic presentation of them. I will/might review them later in a more detailed way.

Kanban
Author: David J. Anderson
Recommended: yes
Amazon:

This book is a goldmine of information about all questions of Kanban. David explorer Kanban for software development. He is the guru without any question. I got his book (signed by him) in one of his courses. This book is just outstanding. A review will follow at later time.

 

Personal Kanban – Mapping world | Navigating life
Author: Jim Benson, Tonianne DeMaria Barry
Recommended: yes
Amazon:

This book is about Kanban as a personal management system. It offers some insights how to use a personal management system based on Kanban. It shows some tipps and tricks, however, it’s not a fully appliable system. You need to tailor your own system and you’ll be guided. This book is more about the why and not that much about the doing.

Getting started with Kanban
Author: Paul Klipp
Recommended: HIGHLY!
Amazon:

That’s an incredible short introduction into the Kanban methodology. Ok, the book features a website which provides a Kanban system as SAAS. However, everything is well explained independently. If you need to get grip on Kanban fast, this is your book to read!
Please note – there is also a free PDF version. The Kindle version works perfect and it’s cheap – no reason to work with the PDF. Please look at the PDF version here: Getting started with Kanban.

Designing monitoring systems can be thrillseeking. I have seen all kinds of homegrown monitoring systems built on top of nagios or mrtg. Usually all those solutions lack one thing: decent configuration possibilities.

In the old administration days one sysadmin was responsible for the installation of a server and the configuration system. Right now one sysadmin uses a central management server and fires up several virtual boxes or cloud instances with a few commands. But what about the monitoring? Is the monitoring included with the setup of your cloud instances? Lucky you. Usually those kind of tasks are only for brave sysadmins – and they are getting fewer every day.

Graph drawn by rrd2graph from collectd data

Graph drawn by rrd2graph from collectd data

I gave a talk on Tuesday about a more flexible toolchain for monitoring than the common used nagios setup. I had some good experiences with this toolchain at the company I am working for and it seems to scale reasonably well.

collectd is small daemon which collects the data on every instance/server and stores the information in rrd databases using librrd or via rrdcached. The perfomance data is being pushed to central servers, too. I like to call them collectors. They gather all the data from the different servers/instances. The advantage is clear – nagios can get all the values from those databases from every server at one place. This is not only an advantage for the firewall, it also saves the nagios server alot of work. The graphs are being drawn using rrd2graph (see screenshot) which is part of the n2rrd project.

The best thing about using collectd is it’s way of configuration. I suggest to setup three different configuration levels. A base configuration which is exactly the same on every host. It just has the most basic configuration in. A second level called environment configuration deals with networking stuff like who is the collector server in a certain zone. The third layer is called service configuration. This layer deals with services running on the server such as mysql, httpd, nginx, etc.

If you are using a configuration management tool such as puppet or RedHat Satellite it’s an easy task to build classes or system service groups which care for the configuration layers. Use RPMs during the kickstart process to install the collectd binaries and provide them with a sane first /etc/collectd.conf.

collectd provides a nice plugin structure with about 90 pre-made plugins and it’s easy to extend by using it’s Perl, Java, Python apis.

 

Oh, did I mention? This works for Windows based servers, too! Please check out the slides for more information. The first link is a mindmap in xmind format with a small management summary. If you are interested in the slides of the presentation, please use the second link which is in PDF format.

 

Monitoring 2.0 – Presentation Summary (xmind – Mindmap)

Presentation about monitoring Unix environments with modern tools (pdf)

Teaching new colleagues the wonders of Perl can be a tough thing because it’s not clear where to start. Do I have to start from scratch with just the basic things or does someone who had Perl mentioned in his CV qualifies as a Perl person already?

That’s a difficult question. Having interviews, talks or even exams might not give the right answer but you will get a certain feeling about someone and his skills. However, even the old geeks tend to be silent when Gabor Szabo (blog) is asking stuff about regular expressions in his talks on conferences. That’s interesting. People who are able to do the most complicated stuff with regex are quiet when they are asked simple regex. Is that the kind of people we are referring to as DevOps? Probably not.

So how do we find those multitalented people who are using Perl for sysadmin stuff, understand the complexity of J2EE and its architectures and are communicative to their colleagues – even when they can’t provide the answer at this time (referring to the regex problem). Actually, I don’t know. During interviews I have used the following simple questions to get a feeling about the Perl knowledge of the candidate:

  1. What’s the correct way to test if two scalars are the same – string comparison?
  2. What does the keyword “say” do?
  3. What is the meaning of the code fragement $c = $a // $b ?
  4. What is the purpose of the command line tool prove?
  5. What is the taint mode of Perl?

Those are just 5 simple questions. Let me explain why I like to ask them and why I think they can give you more insights into the job candidate.

Question 1 – Tries to sort out if the person is actually using Perl. The most common answers for this are: “==”, “=” or “$string1=~ /$string2/”. Once again, Gabor has some answers for this in a short YouTube video – Comparing scalars. If the candidate is very nervous and fails instantly, be nice and start a dialogue. Discuss the different options – if he has no clue at all, he will fail anyway in this question.

Question 2 – Asking about “modern” Perl knowlege. Most of the Google code snippets don’t show results about the keyword say. However, if the person is using a more current version or if he read the history he knows about the keyword.

Question 3 – Tricky one and might even be complicated for professionals. Code constructs like this one are common:


if ( ! defined $a ) {
$b = $a;
}
else {
$b = "Hello world";
}

Haven’t seen constructs like this – check your legacy Perl code. There are nice examples in the perlop documentation it. This is something special, however, there is a reason why this operator exists – if you want someone working for you who knows how to use the tools – check if he really knows them.

Question 4 – Huh, don’t know the answer? That doesn’t matter – really. A lot of people are using Test::More for a long time and know about TAP. If the person knows TAP and Test::More already, it’s ok that they don’t have the meaning of prove handy. If they don’t know anything about TAP or Test::More, you will have another hint of their level of Perl knowledge. Do you expect a DevOp person to be the glue between development and sysadmins without knowledge of testing stuff?

Question 5 – wow, that’s legacy stuff. Really, I mean it – what was the last time that you’ve use the taint mode for old CGI stuff? But is it such a bad idea to use the taint mode? Actually, the taint mode is something unpopular today. I wrote several scripts some decades ago which used the taint mode. I will fail horribly when I am trying to use the taint mode today – however, understanding about the history stuff is never a bad idea.

 

Those are just some questions which might help you to check the knowledge of Perl of a candidate. You’ll understand their ways of thinking – their way to solve problems. If you can use the interviews to ask them face to face – that’s awesome and might give you an even deeper insight. If you are asking these questions on the phone – be nice! They are not as easy as you might think now when you are relaxed and not in a situation of job application. Help them on their way – test their knowledge, look at their problem solving skills and motivate them to pass the tests.

When you want to attract DevOps – try to be as nice and communicative as you expect them to be!

I like to look at things that work well and try to figure why they work and what are the success factors.This is more fun at projects which are successful than looking at projects which failed. Usually the spirit of successful projects is more straight forward and doesn’t sound like an excuse. “Blah, blah, blah messed up the code repository and we are just in the phase of picking up everything.” That doesn’t sound like a great project to me and already got to the point of finger pointing. In a great project which is successful and well accepted someone might pick up the broken things and fix them. Oh, wait, even there might be something like a vision which is helpful for people to follow.

Here are some simple facts about github usage which might be helpful:

 

Fact I: Use github right

github is not just a tool or a place to put a clone of your git repo on. If you decided to put things on github, use github as a platform. Be polite, answer to issues and decide about the incoming pull requests. Some projects use github as a place for backing up their repositories. “A copy of the original repo is on github” – is a common term. Forget about it. If someone forks and contributes to your project make sure to answer appropriately and fast. Those people already forked the repo, downloaded everything, looked into the sources and put some effort into it. They don’t deserve a break about 1-2 weeks for contributing. Firing a patch to a mailing list is just easy. Working out a pull request is much more complicated. You probably have to rebase your stuff on top of the repo, merge stuff..you’ve figured already.

If you don’t encourage contributions – don’t use github.

 

Fact II: Use github for communication

Denying a pull request isn’t bad – it’s just code. Be nice and explain why you don’t accept the pull request. I’ve sent several pull requests to @kraih for Mojolicious. He rejected most of them, however, I always got a nice reply. Sometimes he even took my commit and changed it a little bit. I was visible as the author and he took the rule as committer. git is something social politcal statement alike, too and github visualizes this nicely.

Use github to encourage contributions!

 

Fact III: Wiki your community

The wiki on github is very basic. However, it provides some useful features which are helpful for the Perl people. You can choose the format in which the pages of the wiki are being created in. Using the pod format allows people to clone your wiki and read it in offline mode using perldoc. This is an awesome and in my opinion underrated feature. The only problem with the wiki on github is that you have to enable write access to the project repo to allow users to push content. I’ll hope they’ll fix it one day.

The wiki is usually driven by the community. The headline person of the project can contribute and give insights into different code parts and can share his vision. The community will care for examples, discussion and keep the wiki up to date. Therefore the wiki needs to be writable for everyone. I am serious, open the wiki for everyone! All the content is stored in git anyway so you can easily revert things if they got broken.

Be spirited, share your vision and be open to your community!