pkgbox - Page 2

Publishing your existing git repo (Xcode)

09.05.2011, 1 Comment, Devops, MacOS X, Unix, by rhaen.

I’ve received alot of feedback about the setup of a git server on MacOS X. Some people had some problems with Xcode and existing git repositories and sharing them. My article about setting up a git repository was starting with an empty repository, here is the description on how to start with existing git repositories. I won’t explain everything from scratch – it’s just something like an addon to the article.
Please make sure to follow all the steps in the tutorial, except using a new repo, I will show you how to prepare an existing repo for sharing.

Let’s assume we already have a project named “myproject“. As I am a Perl programmer we will use Perl sources and a perlish layout of the project. However, this should work for every project! As you see in this screenshot I’ve some source layout (given by common Perl nature) and a “.git” directory. I’ve worked with the sources and ran git init and added all the files to the repo.

project toplevel folder with .git directory

Therefore we have an existing repo which is fine for local work, however, sharing via HTTP requires some steps:

Clone the existing repo to a bare repository
Enable the post-update hook
Setup the apache to share the repo

Let’s start with cloning the repository to a bare repo. A bare repo is just a git repository with a different layout. You won’t see any active checkouts aka sources, you’ll only find the structure of the “.git” repository in it. As we used the directory “/Library/WebServer/Documents/repo” as the base directory for my last article we’ll continue to work at this directory base. We will clone our small project “myproject” to “/Library/WebServer/Documents/repo/myproject.git”. This allows to follow the later steps in the setup article.

We start inside the directory of our project, so the “.git” repo is on the top level.

$ sudo git clone --bare . \
   /Library/WebServer/Documents/repo/myproject.git
$ sudo chown -R _www:_www \
   /Library/WebServer/Documents/repo/myproject.git
$ cd /Library/WebServer/Documents/repo/myproject.git
$ sudo -u _www mv hooks/post-update.sample \
   hooks/post-update
$ sudo -u _www hooks/post-update

From this point you can follow the old article about “Set up your Apache server for sharing”

Enjoy your new git experience!

Morbo as a service on MacOS X

09.03.2011, Comments Off, Ironman, MacOS X, Mojolicious, Perl, Unix, by rhaen.

I am a developer and a sysadmin – this qualifies me as a lazy person. I don’t want to do things more often than needed. If I can automate my sysadmin part of my life I am happy to do so. One thing about developing stuff with Mojolicious is to start the development server by hand. Sebastian (sri) did a fantastic job in writing morbo, the development server for Mojolicious. morbo is a very capable HTTP 1.1 compliant web server and it’s part of the Mojolicious distribution. The generic syntax to use the daemon with a Mojolicious::Lite application is:

$ morbo [options if needed] <scriptname>

Everytime you change the script, morbo will detect this and will reload the application. Best part of it, it will detect syntax errors and warn you about them. This is very helpful for me as it separates a few steps. First you develop the Mojolicious::Lite application, then you’ll save it, morbo does the syntax check for you and you see if things are broken from the code side, next you’ll test the web application with a browser and check if everything is working there. To run unit tests all the time is self explanatory.

So yes, I am a fan of it. Oh, there is something even nicer about it. morbo doesn’t only check the script for changes, it subscribes automatically to the lib and to the templates directory, too. If they don’t exist, morbo will just do the right thing. So once you’ve a basic Mojolicious::Lite application and decide to inflate it from one file to a directory structure your setup doesn’t need to be changed. That’s cool. Speaking from the sysadmin side of life: It’s a horror to track changes in your deployment configuration. If you are part of the development team you might find ways to deal with it. If you are a sysadmin which only cares for the platform you are doomed.

Here is the simple way I like to work with morbo in my MacOS X environment. I am doing my development stuff in a subfolder of my home directory. I’ve put the folder as a subfolder of my Library folder so I won’t see it all the time. Nothing is more annoying as millions of folders around! Please note: The Library folder is not visible by default, you might want to create the structure in your home directory, if you are uncomfortable with the Unix shell.

You can unhide the Library folder in your home directory by running

$ chflags nohidden ~/Library/

in the Terminal.app.

I’ve used perlbrew to install Perl version 5.14.1 in my home directory. This way I can install modules and play with them without messing around with the MacOS X system stuff. As I’ve said before – I like to keep things separated – call it sysadmin paranoia. Let’s create the basic structure.

$ mkdir -p ~/Library/Mojolicious/{external,external/repo,log}
$ mkdir -p ~/Library/Mojolicious/{static,templates}
$ cd Library/Mojolicious
$ mojo generate lite_app default.pl

Voila, we now have a basic directory structure and a standard Mojolicious::Lite application named default.pl. Let’s see what else do we have.

external
directory for external resources, I clone mojo from github into this directory. I use it as reference as well as for installations of the latest code of Mojolicious.
external/repo
directory for my personal git repositories which are Mojolicious related
log
log directory for Mojolicious (default setting)

static
directory for static resources such as images, css, js files
templates
template directory for Mojolicious applications (default setting)

Now that we have a basic structure and a basic setup we should look on the automatic startup of the morbo server and our just created default.pl Mojolicious::Lite application.

MacOS X uses a special daemon for launching services which is called launchd. The daemon is controlled by a command which is named launchctl. launchctl will be used to load a xml style description of the service into the daemon and start it. You can specify custom services and you’t have to be the user root to work with the daemon, that’s nice. So, let’s look into the xml service file for our morbo service.

launchd property file for morbo - the Moljolicous web server (see gist)

That’s an image, click to enlarge it. I’ve used it for better visibility, you can find the full source file in the following gist.

Here is a quick review of the important parts:

<key>Label</key>
<string>us.mojolicio.default</string>

Every application/service needs to have it’s own label. The label is used to identify the service at a later point. Use a dsl syntax to keep things sorted. My own applications will be named de.pkgbox.mojolicious.<name>.

<key>ProgramArguments</key>
<array>
   <string>morbo</string>
   <string>–listen</string>
   <string>http://*:8080</string>
   <string>/Users/rhaen/Library/Mojolicious/default.pl</string>
</array>

This is a little bit tricky. I don’t need to specify the full path to morbo as launchd will use my user environment to run the ProgrammArguments. However, I have to specify the full path to my Mojolicious::Lite application named default.pl. I am using some arguments to morbo to keep it more usuable for me but let’s work down the way. Usually you have to use a Program key and specify ProgramArguments as an array for more arguments. If you discard the Program key it will use the first element of the ProgramArguments as Program key. Simple.
I like to have morbo running on port 8080. Maybe during my development phase I want to run another Mojo application on the default port 3000. That’s just a good way to make sure that normal development doesn’t interfere with your basic environment.

Ok, here is the next part of the setup:

<key>RunAtLoad</key>
<true/>
<key>WorkingDirectory</key>
<string>/Users/rhaen/Library/Mojolicious</string>
<key>StandardErrorPath</key>
<string>/Users/rhaen/Library/Mojolicious/log/output.log</string>
<key>StandardOutPath</key>
<string>/Users/rhaen/Library/Mojolicious/log/output.log</string>
</dict>

RunAtLoad means that we want to start the morbo server right from the start. We don’t want to run it on demand neither we just want to register it, we want it running! The WorkingDirectory is an important setting. launchd will chdir to this directory before running the Program and the ProgramArguments. That’s needed to get the correct environment for Mojolicious for the log, static and templates directory. Last but not the least we use a StandardErrorsPath/StandardOutPath for some basic log files of the launchd daemon. Does something fail? Was there an error? Did morbo crash? We’ll find the information in this file!
Please note: There is an optional setting named KeepAlive which will try to restart the morbo server once it fails. This can be set, however, I would like to know when morbo crashes and why it crashes. That’s why I am not using this setting.

That was alot of stuff! Store all the information above in a file named “us.mojolicio.default.plist” and save it in the external directory of our basic structure. I like to keep the main copy there – just for separation, you know sysadmins paranoia.

In order to activate everything we need to run a few commands. First we need to place a symlink inside the ~/Library/LaunchAgent folder. After that we need to register aka load the file and we are done.

$ mkdir -p ~/Library/LaunchAgents
$ cd ~/Library/LaunchAgents
$ ln -s ~/Library/Mojolicious/external/us.mojolicio.default.plist .
$ launchctl load us.mojolicio.default.plist

Why do we use a symlink? Why don’t we just place the file in the correct place? The answer is simple. We kept everything Mojolicious related together. We can run git init inside the Mojolicious folder and have everything version controlled. One place – one repo, sysadmin paranoia…you know.
By loading the configuration file we informed launchd about a new service and launchd already took over. The new service has been started and the morbo server is serving default.pl at port 8080. Why don’t you fire up a web browser to check it?

Uninstalling everything

Got lost? Want to get rid of everything and start from scratch? That’s simple. First we need to unload the morbo service from launchd. Next we’ll remove the symlink and delete the Mojolicious folder. Voila – finished. Dear developers, that’s sysadmin style. Clean up after your play – dear sysadmins, start coding!

$ cd ~/Library/LaunchAgents/
$ launchctl unload us.mojolicio.default.plist
$ rm -rf ~/Library/LaunchAgents/us.mojolicio.default.plist
$ rm -rf ~/Library/Mojolicious

Make sure to leave some feedback!

Test::Mojo – Intro into testing with CSS3 selectors

09.01.2011, Comments Off, Ironman, Mojolicious, Perl, Win32, by rhaen.

Testing with Mojo is great fun. The Mojolicious framework provides an easy way to test your web applications without any setup of web servers, proxy servers etc. Just use the module Test::Mojo which is part of the Mojolicious framework and Mojo will do the right thing for you.

There is a nice tutorial how to test things for Mojolicious and Mojolicious::Lite applications in the wiki on github (Testing Mojolicious applications). This short tutorial explains most of the testing stuff in an easy way with good examples.

More fun with Mojolicious and testing:
Here is another trick to test stuff with Test::Mojo. You can use CSS3 selectors inside your tests to check values.

Consider the following simple html webpage:

<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.01//EN”>
<html>
<head>
    <title>Hello</title>
</head>
<body>
    <ul>
      <li>
        <a href=”Goodbye.html”>Goodbye page</a>
      </li>
      <li>
        <a href=”Hello.html” target=”_blank”>Hello page</a>
      </li>
    </ul>
</body>
</html>

What’s the correct way to test the web page? The answer is simple: Just use Test::Mojo for it and use CSS3 selectors to get the stuff your need.

Let’s fire up a text editor and get started. First we need our basic Perl header and a few modules. I’ve set up the variable $url for easier access to the web page. Please note: we are testing against a web server here. We use the Mojo layers to access a web server somewhere on the net (in this case – the localhost). This is a great way to build up a testing suite for your web page which is already in production. It also provides a way to check your existing web page for content elements – like is the right date displayed? It doesn’t matter what your site is running on, using Mojo for testing is just fine.

Back to our code. Here is what we’ve got already.

#!/usr/bin/env perl
 
use strict;
use warnings;
use Test::More tests => 6;
use Test::Mojo;
 
my $url = 'http://localhost/~rhaen/index.html';
my $t = Test::Mojo->new;

Next thing will be to check for the correct title. Is that the right web page we are checking? Is the title element existent and does it provide the correct title? We can use Test::Mojo to request the web page and use Mojo::DOM to extract the title.

$t->get_ok($url)
    ->text_is('html head title'
        => 'Hello', 'correct title');

The next part will use CSS selectors, too. We want to check the text for the “Hello.html” page. We use a CSS selector to select all links with the attribute “target”. Please note, the “Goodbye.html” link doesn’t have the target attribute so we won’t get it in our selection.

$t->get_ok($url)
    ->text_is('a[target]'
        => 'Hello page', 'correct text for a tag');

The last part of this short Test::Mojo with CSS selectors introduction will be how to test the value of an attribute. The answer is quite simple but it requires some thinking behind the concept. Test::Mojo doesn’t provide a method to test for attribute values. A possible way uses a different concept. We will use a CSS3 selector and select the correct value with it. Then we’ll check if the element we’ve just selected exists. It’s that’s simple. And here goes the code:

$t->get_ok($url)
    ->element_exists('a[href=Hello.html]', 'Yep, attribute value ok');

So we’ve just checked if there is an anchor element which points to “Hello.html”. It does, so the test succeeds.

This was just a short introduction into Test::Mojo with CSS selectors. Make sure to checkout the documentation for more examples.

You can find the full code example on github in the following gist. Make sure to change the $url variable to a webserver with the index.html (also found in the gist).

$self->hello_world();

08.27.2011, Comments Off, Ironman, MacOS X, Mojolicious, Perl, Shell, Unix, Win32, by rhaen.

I’ve moved in here tonight – I am your new neighbour.

This blog travelled through the highlands of the internet and is now located here. I just bought this new home so there won’t be any changes in the next time. Please update your booksmarks to the new site, I’ve redirected the most useful stuff so you shouldn’t encounter too many 404′s.

For my personal weblog about motorcycling please follow this link: einspuriges.de

Due to the change I’ve lost all the comments :( That’s not nice, however, I’ll count on you.

MacOS X and Cocoa/Carbon environment

08.26.2011, 5 Comments, Ironman, MacOS X, Perl, by rhaen.

When I use perlbrew or local::lib for my custom libraries I struggle with the MacOS X user interface. It just doesn’t recognize changes to my .bashrc/.profile file. So when I want to run a custom PERL5LIB path – it works fine on the shell (aka Terminal.app) but not in my beloved TextMate.

All MacOS X applications (Carbon/Cocoa) are sharing the same way of setup. When a user logs into MacOS X, a special file is being read. The file is located inside a subdirectory “~/.MacOSX” and is named “environment.plist”.

The file is in plist format, the easiest way to create such an environment is by using the terminal with the following commands:

$ mkdir -p ~/.MacOSX
$ touch ~/.MacOSX/environment.plist
$ open ~/.MacOSX/environment.plist

The wonderful PropertyList editor opens and you can enter the desired values in a key value fashion. After you entered the values, save them and close the editor. You have to log out and back in to activate the changes. Now your MacOS X environment contains the new settings.

environment.plist for Carbon/Cocoa environment on MacOS X

Setting up a git server on MacOS X Lion

08.26.2011, 22 Comments, MacOS X, Shell, Unix, by rhaen.

Please note: The blog moved to this new location – you’ve been redirected. I’ve lost the comments in the migration proccess :( – your feedback is appreciated, I’ll still include your comments and hints if the process fails for you!

Time to rewrite one of the most requested articles in this blog. Getting up a git server on MacOS X Lion real fast. One thing which is new and which is totally awesome: git is part of the Xcode 4.0 toolchain – so there is no reason to install any toolchain like MacPorts or homebrew to get up a git server. Just install/upgrade your developer tools to Xcode 4.0 and git will be available on your machine! Please note – I’ll update this article from time to time, see the change log below the article to reflect latest changes.

Let’s look into the different steps which are needed:

General information
Install/Update everything
Set up a bare git repository for sharing
Set up your Apache server for sharing

General information

Everything which is about a path to a command, a command or a reference is written in italic letters such as: which, ls. Commands which have to been entered inside the Terminal.app will be written in a code style face. Notice the $ in front of the command. This sign will be an indicator what I typed and where the output starts. Sometimes the command line tends to get longer. In Unix it’s totally fine to break these lines using the \ sign. If are doing the steps manually, just leave the \ sign and put everything in ONE line.

Feedback by some users:
There are some differences between MacOS X Lion client and MacOS X Lion server. The main difference between those two version is the document root of the the apache web server. On MacOS X client the default path to the document root is: /Library/WebServer/Documents/.

On MacOS X server the document root is located at: /Library/Server/Web/Data/Sites/Default.

I will describe everything for the client version. If you are running the server version, make sure to use the correct directory for your version of MacOS X Lion. So just exchange the path from client version to server version and you should be fine. I’ll provide detailed information for a more general setup at a later time.

Install/Update everything

This is the easy part – everything has already been installed if you updated to Xcode 4.0 The Apache web server is part of the MacOS X OS anyway. If you want to check that you have a working git, just open the Terminal.app and run the which command. This will print the the path to the git binary. If this step doesn’t print you anything you failed the update of the Xcode toolchain. Usually the output looks like this:

$ which git
/usr/bin/git

Time to move on with the configuration.

Set up a bare git repository for sharing

Please note: if you are already working with an existing repo you want to share – have a look at this article (Publishing your existing git repo).

In order to share a repository with the apache web server we need to setup a bare repository. Let’s dive into a few parts of the development processes. Usually you start coding and after a short time you realize, that you should store things in version control – or you start with version control from scratch. It doesn’t matter which is the right way – we just want to get started quickly.

Let’s change to the terminal app and create a bare repo!

Change the working path to the web server document root and create an empty directory named myproject.git inside it. Change into the directory, run git init –bare to set up a bare repo. We’ll have to use the sudo command as we are not allowed to write inside the web server document root by default. It’s only a sequence of short steps.

Here are the steps inside the Terminal.app window:

$ cd /Library/WebServer/Documents/
 
$ sudo mkdir -p repo/myproject.git
Password:

* Please note – it’s your user password which is requested! *

$ cd repo/myproject.git/
$ sudo git init --bare
Initialized empty Git repository in
/Library/WebServer/Documents/repo/myproject.git/

Voila, our first repo has been setup! However, it’s not very useful for now. The web server is still turned off, the only way to access it is by using a direct path on the filesystem.

We now prepare the git repo to be shared by the web server and – to be updated by the web server. We’ll start with the commands inside the git repo and work our way to the web server configuration. When we want the web server to behave to commits and clones correctly we need to enable the post-update hook. This hook runs a certain git command every time someone updates or pushes code to the repo. After that we’ll allow the web server to read and write the repo files. Right now, the web server can only read them.

The Terminal.app it still open and we are still located inside the directory of our git repo – good. The first command shows the working path – it should be our project.

$ pwd
/Library/WebServer/Documents/repo/myproject.git
$ sudo mv hooks/post-update.sample \
hooks/post-update
$ sudo chown -R _www:_www \
/Library/WebServer/Documents/repo/myproject.git
$ sudo -u _www hooks/post-update

The last step was running the enabled post-update hook. This hook sets some references about revisions in our (still) empty repository. We now have a fully enabled git repository which is ready to be served by a web server. Yay! Let’s now move on to the steps for configuring the web server.

Set up your Apache server for sharing

The scenario for our repo is quite simple. We want to setup two different groups for our repository. One group is able to read the repository – aka to clone it. The second group is able to write to our repository, hence to commit changes back to it.

So – what’s our job now: Create a password file for the web server, create a group file for the web server, configure the web server.

The password file contains user and passwords and will be created using the htpasswd command. The syntax is simple. We need to supply an additional command flag for the first user in order to create the file first.

$ sudo htpasswd -c \
/etc/apache2/other/htpasswd-git rhaen
Password:
New password:
Re-type new password:
Adding password for user rhaen

It looks a bit confusing. The first password prompt was for the sudo command, the second and third were for the htpasswd command. If you go straight to the tutorial it might not be necessary to enter the sudo password as it is being cached for a while.

Now we created a htpasswd-git file with one user – named rhaen in it. Just add isolde as second user.

$ sudo htpasswd \
/etc/apache2/other/htpasswd-git isolde
New password:
Re-type new password:
Adding password for user isolde

The group file is a plain text file which contains groups and the users in. I like to use the editor vi, however, just use any text editor you like. Make sure to save the file as plain text. The contents of the text file will look like this:

myproject-reader: rhaen isolde
myproject-writer: rhaen

I removed the “.git” suffix from the project for better readability. We refer to the group name inside the configuration of the web server. Also note that the user isolde is only in the group of readers. Save this file to your desktop with the name htgroup-git and move it to the web server config directory.

$ sudo mv ~/Desktop/htgroup-git \
/etc/apache2/other/

Now to the final step: the configuration of the web server. Use your favorite text editor and copy/paste the following configuration snippet listed below, download it and open it in an editor. Here is just an example image what it looks like.

A screenshot of the configuration file gitrepo.conf

The image shows the configuration at it’s glance. The download link is here. Save this file as gitrepo.conf on your desktop. Before we enable this file, let’s have a closer look in it. The first directory part disallows all kinds of access to the directory named repo and below. That’s the reason why our git repo will be protected. It also configures the apache web server which password and which group file is the right one.

The second directory is the configuration about our project. It separates the user in reader and writer and requires the certain group for it.

As we are using DAV for the git push we need to enable a lock directory. OS X has a default one which is configured but it doesn’t exist. Let’s create one – we have already enabled it in our configuration file on the first line.

$ sudo mkdir -p /var/www/DavLock
$ sudo chown _www:_www /var/www/DavLock

Let’s move this file to the correct place and restart the web server.

$ sudo mv ~/Desktop/gitrepo.conf \
/etc/apache2/other/

There you go!

Feedback from a user:
Something which is very important for the correct function of the web server is to enable “web sharing” in your system preferences. Go “System Preferences” – Sharing – and click the checkbox web sharing. After that run the following command:

$ sudo apachectl restart

Now it’s time to test everything! Open your web browser and try to access

http://localhost/repo

You should get an error. Fine! (This results from the deny all rule). Change to the terminal and try to clone the repo:

$ git clone http://rhaen@localhost/repo/myproject.git
Cloning into myproject...
Password:
warning: You appear to have cloned an empty repository.

YES! You know have a fully working git repo with clone and even push support ready! Add some code into it and push the changes! Enjoy!

Troubleshooting

If you see the following error:
 
$ git clone http://<username>@localhost/repo/myproject.git
fatal: could not create work tree dir 'myproject'.: Permission denied

You try to clone into a directory which is not writable for you. Please try to clone on your MacOS X desktop – this should always work. Change to the desktop:

$ cd ~/Desktop

and clone again.

This is just a tutorial to get started and it won’t solve everything. Give me feedback what worked for you – I’ll monitor my mails carefully and try to help you. Thanks for all the feedback so far! A lot of feedback has been provided so far, thanks to everyone who contributed so far. Here is a small change log what happened:

ChangeLog:

2011-07-29 updated article – included information about the web-sharing control panel
2011-08-07 updated, added MacOS X Lion server information
2011-08-11 updated, added DAV lock directive (feedback from user)
2011-08-27 updated, moved blog to a new location, included feedback about cloning errors
2011-09-04 updated, added anchors, changed post-update

Embedding files in Mojolicious::Lite apps

09.24.2010, Comments Off, Mojolicious, Perl, by rhaen.

Mojolicious::Lite comes in very handy when just a simple script is needed. You can still use the full power of Mojolicious but everything is packed in one file which is awesome. That’s a very elegant way to ship your application to the customer. But what about inline graphics, is that possible? There had been a discussion about this topic on the Mojolicious mailinglist a while ago. In the meantime this feature was implemented and here is a small example on how embedding images inside a Mojolicious::Lite application works.

I decided to have a small favicon shipped with one of my scripts. I found a great website named favicon.cc which has tons of favicons available. Most of the favicons are under Creative Commons license which makes them easy to use. I chose a small cloud as Mojolicious logo is small cloud. Voila – a nice one was found easily. Here it is, a cloud made by Christian Paul under CC license.

If you scroll down the page you’ll find a textbox with the base64 code in which looks like this:

<link href=”data:image/x-icon;base64,iV…

I copied everything starting behind the comma until the two equal signs (including them). Please note the (base64) after the filename. This will make sure that the data will be decoded properly by the renderer. Here is another secret: If you decide to inflate your application the base64 encoded files will be inflated too!

Here is the sample code for an application which is capable to serve a request to /favicon.ico in Mojolicious::Lite.

#!/usr/bin/env perl
 
use Mojolicious::Lite;
 
app->start;
__DATA__
 
@@ favicon.ico (base64)
iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABm
JLR0QAAAAAAAD5Q7t/AAAB5ElEQVQ4y+2SPWtTcRTGf/ctNzE1
JEQDNaHSF99tcZAEpBRXXRSlk+DYyVk/Q4cuxTXubt2UDhHFwV
Cp0CzWNLkELJGSkGjSNMm9938cYkJK9Bv4wIHnOZw3OA/8hzYu
srmi5J0WkZDJnZkgxVqPastjZTbMwnmbRscndsZg6cpFbWJANl
eURhdW00laJy6b2yUep5Mko0HeFo747DRJxWy+N3osz03x7N61
Qe/7L2VZ3yrIWnZXaq2eDNH31CleqXVERKTT6chadlfyX6sCYH
6qHHNrPsHzBwmUUnieB4Ch63ieP+KpmI1SCjMQBGDHaQKgV1se
yeggqZTC931838d13VN8qA1N2Hxyna6YZHNF0Vdmw2xul6i3+4
gISql/hud5+L6PbdusppPknRZmPGwSCZlEQhaGJqMrlFITL9N1
fcR/1H6SigYwj/uKoKVjGRqgoZsBoD8xwDAMdF3Hsixe5w/5dt
jkdio4eOP6VkFKdZdfXZ+XT5eITwVGZ7uui2VZiGb8WQIbbw6Y
iwiPli9rIx/s7VfkQ7kDwN2bF8gXj8hcSnAjFQHg1bsDnHoPgL
OW8OLhojbhxL39ipxIgB2nycdym5lzIQCa7T7zcYvF6YG+n1mY
dOI4hiYZR+bq9F9rfwNS4g/5j28nMwAAAABJRU5ErkJggg==

Web development can be fun again!

Changes to this article:
Please note: There had been some changes to this blog entry recently. Setting a static route is not required for Mojolicious::Lite, it’s clever enough to serve the right thing outside the DATA section. If you inflate this application the files from data section which are base64 encoded will be created inside the public directory of the application (as they are static files).

Folding Perl POD section with TextMate

09.09.2010, Comments Off, MacOS X, Perl, by rhaen.

Right now I am working on some POD documentation for a smart Perl project (more to come later). I stumbled across this project and decided to give it a chance. As I a newbie to this project and to this Perl module I am not able to understand everything. As the documentation is still a work in progress it’s even harder to follow the concepts and the ideas. However, this is the best situation to contribute to a project. There isn’t such a thing as an over documented project.
I am using MacOS X with TextMate to do the editing. One thing which bugs me with TextMate is the bad style which is used to highlight POD documentation. I want different colors for the =head sections and a different color maybe even a different font setting for the POD text. I was searching on google and found a blog post on Ingo Lantschers blog about POD folding.

I looked into it and decided to change his suggestion a little bit. Open the Edit Languages inside the Bundle Editor, scroll down to Perl – select the “L” Perl. Search the line which mentions the foldingStartMarker and foldingStopMarker and replace them with the following lines.
foldingStartMarker = ‘(/\*|(\{|\[|$)\s*$|=pod|=head|<<\s+"\s*__EOF__")';
foldingStopMarker = '(\*/|^\s*(\}|\]|$|=cut|\s*__EOF__))’;

(Please note – one option like foldingStartMarker is always one line until the ‘;’. So it’s two lines to change.)
Please note: The fold starts when TextMate founds the tags =pod or any of the =head sections. You must end the section with =cut in order to show TextMate the end of the documentation block. For here and now documents Ingo introduced a special marker called END_HERE. I decided to change this into __EOF__ as this is my style of writing here and now documents.

Thanks for the prework Ingo!

References:

Ingo Lantschers blog post about folding

Running iostat with timestamps

09.29.2009, Comments Off, Shell, Unix, by rhaen.

Have you ever encountered the problem how to track the performance of your system? What about the memory? What about the network? What about my disk activity? There are several tools which collect data, maybe you already heard of mrtg. For a quick stress test just to collect some data or if you are at a customers site and are not allowed to install tools, just make it simple and use the standard Unix tools. NetBSD has all the tools needed such as vmstat, iostat(8), systat (nifty interface), nfsstat….

Plotting iostat data with gnuplot

Just use them to collect data and plot them using gnuplot. You are able to get all the data you want and produce some nifty graphs out of it without the usage of Perl, etc. Unfortunatelly all these tools don’t have timestamps. However, we need those timestamps for our graphing things. awk(1) to the rescue!

# iostat(8) -x 5 | awk '/wd0/ {print strftime("%H:%M:%S"),$0}'
15:44:20 wd0 5.17  0  0.00 0.00   17.62   0 0.00     0.00
15:44:25 wd0 0.00  0  0.00 0.00    0.00   0 0.00     0.00
15:44:30 wd0 0.00  0  0.00 0.00    0.00   0 0.00     0.00

This will run the command iostat(8) every 5 seconds and shows the extended statistics. However, we just want to see the wd0 device, so we use awk(1) to grab the line with it from the output and put a timestamp in front of it. Note: Collecting data with iostat(8)every 5 seconds might be way too much. A period of writing data usually lasts longs than 5 seconds of a server life. In my experience about 30 second is just fine. Pipe the output into the tee(1) command – it looks geeky and you’ll save it on your disk, too. If you want to process the data with gnuplot make sure to remove the _FIRST_ line of the output. This line is an average for the system since it’s uptime and we might ruin our data collection with it as we just want to see the test data. If you have problems to capture the output, just run it inside a screen and use the command CTRL-A H to capture the output to the screenlog.0. I like this way as my shell might get disconnected, however the data is still captured.
After we collected the data over a certain amount of time, we are able to produce graphs out of it. Your customer will follow your argumentation more likely if they can see data visualized. Excel is, of course, a good choice – however you can use gnuplot to plot some fancy graphs out of it. When I was a consultant for IT things I used to carry a small USB stick with me. I stored gnuplot for Win32 on it and the basic script for gathering data. I ran the command iostat(8) on my Soekris for a couple of seconds to draw a graph out of it. It’s nothing skyrocketing – just an example how things work together. Here is the gnuplot code I needed to plot the data from the iostat(8)command. It’s the cvs update running on the Soekris (on a very slow cf card).

gnuplot <<_EOF_
set terminal png
set out "iostat-rs_ws.png"
set title "iostat during cvs up -dP"
set xdata time
set timefmt "%H:%M:%S"
set xrange ["19:28:33":"19:50:08"]
set xlabel "Time"
set ylabel "Operations per second"
set format x "%H:%M"
plot "iostat.dat" using 1:4 title "r/s" with lines, \
"iostat.dat" using 1:8 title "w/s" with lines
_EOF_

Simple, eh? The file iostat.dat is the file which contains all the data from the iostat(8)command. The syntax using 1:4 is the way to say gnuplot what columns to get the data from. The graph shows the phase of cvs when the checks for updates are passed and the update begins. Funny, never saw it that way. Ok, anyway – now it’s your turn. You have everything you need to draw your own stuff. Let’s plot some fancy graphics. You can use gnuplot to visualize your sar output as well!

References/Notes This works for all kind of Unixes. If your awk complains about the syntax you should switch to gawk. I’ve tested it on NetBSD and it works just fine.

How do I calculate yesterdays date for shell scripts?

09.20.2008, 1 Comment, Shell, Unix, by rhaen.

Well, you don’t. I encountered this problem several times in my career as Unix admin and usually the answer is: you don’t. Just use a different timezone and you are set. Let me explain the problem. Usually you want a cronjob to do work like grepping through logfiles with the date of the day before. This ends up in a horrible mix of expr and awks to find out the day yesterday. An even worse thing is to use perl to get the date the far worst thing is to install GNU date for this. Just look at the following example:

$ echo $(date)
Sun Sep 21 12:03:20 CEST 2008
$ echo $(TZ=CEST23CEST date)
Sat Sep 20 12:06:50 CEST 2008

Explanation: We use the environment variable TZ (timezone) to set a timezone which is 23 hours before our current timezone. As we don’t use EXPORT to set the timezone the environment is changed just for the only command. This is an easy way to get 23 hours back, plenty of time for your cronjob needs. The same method works on every timezone, of course.

References