Image Ulrich Habel
About me/Impressum

NetBSD

search

Google


Search WWW
Search rhaen daily

archives

categories

feeds

11/01/2008

The road of instability

It's annoying. This small Soekris device running NetBSD 4 is unstable as hell, however I found the reason for it. The flash card with the pkgsrc stuff on it has some failures from time to time. Think I need to replace it soon. Sorry for the outages, they should be fixed soon.

written by: Ulrich Habel (rhaen)

[/General] [permanent link]

10/19/2008

Time for Rockbox and Perl graphing

Wheeew, I was on holiday and thanks to FLOSS Weekly I decided to install Rockbox on my iPod right before I left. It's quite amazing what these guys reverse engineered and how well the support for the Apple iPod is. I own a 30gb iPod video device - nothing really fancy, however I wanted to have a look on it. No clue what I am talking about? Rockbox is an alternative firmware for your MP3 player. You'll just update your iPod to Rockbox and have a whole bunch of new features available. The most impressive part is the "I-need-no-iTunes" part. Rockbox will find all the songs on your iPod and builds a tag database out of it. Actually you can use rsync to sync your music on the iPod. I really like it.
There are also alot of small applications bundled with Rockbox. One of them is battery_bench. It's writes the current state of your battery frequently to a textfile on the iPod disk. Do you want to know how long the battery of your iPod lasts? Here is a way to draw a graph from the collected data. It's quite simple, charge the iPod until it's fully charged, play an album in an endless loop and run the battery_bench tool. After the iPod turned off due to power loss, get the datafile from the iPod and draw fancy graphs out of it.

I really had some nice feedback about the awk gnuplot thing and I wanted to introduce you to another gem in our pkgsrc collection. The gem is called p5-GDGraph. It's a great Perl module to draw graphics out of data, so let's look into the example of the iPod.

The collected data is stored in a file called battery_bench.txt and looks like this:

[...]
Battery type: 400 mAh      Buffer Entries: 1000
  Time:,  Seconds:,  Level:,  Time Left:,  Voltage[mV]:, C:, S:, U:
  02:18:34,  08314,     100%,     11:25,         4223,     -,  -,  -
  02:19:34,  08374,     100%,     11:25,         4223,     -,  -,  -
  02:20:34,  08434,     100%,     11:25,         4206,     -,  -,  -
[...]
This structure can be easily caught by Perl. I will read the file line by line, and store the data inside an array. As I wanted to draw a graph out of the time for the x-value and the Voltage for the y-axis, I will put those values into two seperated arrays. After I got all the data from the file, I will build an array of arrays out of the data, this will be passed to p5-GDGraph to draw a nice graph out of it (just like the one in the article).
1.) Get the data out of the file
my $battery_bench = "battery_bench.txt";
my @plot_data;
my @x_data;
my @y_data;

# Pulling the data out of the data file
open DATA, $battery_bench
  or die "Can't open file: $battery_bench\n";
while (<DATA>) {
        # we don't want the explanation, just the data
    if (m/^[0-9]{2}/) {
        my $dataline = $_;
        $dataline =~ s/\s//g;
        my @dataparts = split( /,/, $dataline );
        push( @x_data, $dataparts[0] );
        push( @y_data, $dataparts[4] );
    }
}
close DATA;
After we read the whole file we'll have all the data inside the two arrays, @x_data and @y_data. You already guessed it, it's the data for the X and for the Y values. The Perl module however expects the data inside an array of arrays. The construction of it is quite easy and will be shown in the next section.
2.) Pass the data to the graphing module GD::Graph::lines
# Setting up the graph
my $graph = GD::Graph::lines->new( 600, 300 );
@plot_data = ( [@x_data], [@y_data] );
$graph->set(
    x_label           => 'Time played',
    y_label           => 'Voltage (mV)',
    title             => 'Voltage graph of my iPod',
    transparent       => 0,
    x_labels_vertical => 1,
    x_label_skip      => 25,
) or die $graph->error;
my $gd = $graph->plot( \@plot_data ) or die $graph->error;
Voila, you just plotted the first graph out of it. You can use different output formats to get the actual graph as an image out of the Perl code.
# Write the graph to a file
open( IMG, '>battery_bench.png' ) or die $!;
binmode IMG;
print IMG $gd->png;
close IMG;
It's a nice way to draw some quick graphs with Perl without messing around with RRD databases. Of course this module is also an excellent way to draw graphs out of iostat, sar and other tools. If you want to draw multiple lines inside a single graph just add another array to the @plot_data. The maintainer of the module inside the pkgsrc packages collection keeps this module always updated. He's already done a great job about it, thanks for keeping it uptodate.

You can find the full example with code and graphs inside the tar.gz file inside the references. References:

written by: Ulrich Habel (rhaen)

[/pkgsrc] [permanent link]

09/30/2008

Running iostat with timestamps

Have you ever encountered the problem how to track the performance of your system? What about the memory? What about the network? What about my disk activity? There are several tools which collect data, maybe you already heard of mrtg. For a quick stress test just to collect some data or if you are at a customers site and are not allowed to install tools, just make it simple and use the standard Unix tools. NetBSD has all the tools needed such as vmstat, iostat(8), systat (nifty interface), nfsstat....
Just use them to collect data and plot them using gnuplot. You are able to get all the data you want and produce some nifty graphs out of it without the usage of Perl, etc. Unfortunatelly all these tools don't have timestamps. However, we need those timestamps for our graphing things. awk(1) to the rescue!

# iostat(8) -x 5 | awk '/wd0/ {print strftime("%H:%M:%S"),$0}'
15:44:20 wd0 5.17  0  0.00 0.00   17.62   0 0.00     0.00
15:44:25 wd0 0.00  0  0.00 0.00    0.00   0 0.00     0.00
15:44:30 wd0 0.00  0  0.00 0.00    0.00   0 0.00     0.00
This will run the command iostat(8) every 5 seconds and shows the extended statistics. However, we just want to see the wd0 device, so we use awk(1) to grab the line with it from the output and put a timestamp in front of it. Note: Collecting data with iostat(8) every 5 seconds might be way too much. A period of writing data usually lasts longs than 5 seconds of a server life. In my experience about 30 second is just fine. Pipe the output into the tee(1) command - it looks geeky and you'll save it on your disk, too. If you want to process the data with gnuplot make sure to remove the _FIRST_ line of the output. This line is an average for the system since it's uptime and we might ruin our data collection with it as we just want to see the test data. If you have problems to capture the output, just run it inside a screen and use the command CTRL-A H to capture the output to the screenlog.0. I like this way as my shell might get disconnected, however the data is still captured.
After we collected the data over a certain amount of time, we are able to produce graphs out of it. Your customer will follow your argumentation more likely if they can see data visualized. Excel is, of course, a good choice - however you can use gnuplot to plot some fancy graphs out of it. When I was a consultant for IT things I used to carry a small USB stick with me. I stored gnuplot for Win32 on it and the basic script for gathering data. I ran the command iostat(8) on my Soekris for a couple of seconds to draw a graph out of it. It's nothing skyrocketing - just an example how things work together. Here is the gnuplot code I needed to plot the data from the iostat(8) command. It's the cvs update running on the Soekris (on a very slow cf card).
gnuplot <<_EOF_
set terminal png
set out "iostat-rs_ws.png"
set title "iostat during cvs up -dP"
set xdata time
set timefmt "%H:%M:%S"
set xrange ["19:28:33":"19:50:08"]
set xlabel "Time"
set ylabel "Operations per second"
set format x "%H:%M"
plot "iostat.dat" using 1:4 title "r/s" with lines, \
"iostat.dat" using 1:8 title "w/s" with lines
_EOF_
Simple, eh? The file iostat.dat is the file which contains all the data from the iostat(8) command. The syntax using 1:4 is the way to say gnuplot what columns to get the data from. The graph shows the phase of cvs when the checks for updates are passed and the update begins. Funny, never saw it that way. Ok, anyway - now it's your turn. You have everything you need to draw your own stuff. Let's plot some fancy graphics. You can use gnuplot to visualize your sar output as well!

References/Notes This works for all kind of Unixes. If your awk complains about the syntax you should switch to gawk. I've tested it on NetBSD and it works just fine.

Update:

I noticed that my manpage plugin for Blosxom just substituted every iostat occurence with manpage link. This is fixed now. Sorry for the noise.

written by: Ulrich Habel (rhaen)

[/NetBSD] [permanent link]

Severe hard disk crash

The topic is sad and so is the news. The harddrive of my laptop failed badly, it sounds like a childrens toy right now, the BIOS still recognises the drive, however accessing the drive is impossible. It's also a sad news as this harddrive isn't manufactured anymore and a replacement drive is far away. I wrote an article about several CF card to IDE adapters as a solution for this problem and right now I am on my way to setup my small NetBSD on a fresh formatted hard drive (read: 8GB CompactFlash card). The cf card is of the 133x speed class and is somehwat usable after some tweaks to NetBSD. There are certain problems, however, this is part of this blog - we'll sort them out over time and I'll write about it.
This is also the reason why p2c doesn't contain pkgsrc-wip in the next days. I just needed the brand new CompactFlash card from my Soekris - it'll be back shortly.

written by: Ulrich Habel (rhaen)

[/NetBSD] [permanent link]

09/21/2008

How do I calculate yesterdays date for shell scripts?

Well, you don't. I encountered this problem several times in my career as Unix admin and usually the answer is: you don't. Just use a different timezone and you are set. Let me explain the problem. Usually you want a cronjob to do work like grepping through logfiles with the date of the day before. This ends up in a horrible mix of expr and awks to find out the day yesterday. An even worse thing is to use perl to get the date the far worst thing is to install GNU date for this. Just look at the following example:

rhaen@wiesel.pkgbox.org:rhaen $ echo $(date)
Sun Sep 21 12:03:20 CEST 2008
rhaen@wiesel.pkgbox.org:rhaen $ echo $(TZ=CEST23CEST date)  
Sat Sep 20 12:06:50 CEST 2008
Explanation: We use the environment variable TZ (timezone) to set a timezone which is 23 hours before our current timezone. As we don't use EXPORT to set the timezone the environment is changed just for the only command. This is an easy way to get 23 hours back, plenty of time for your cronjob needs. The same method works on every timezone, of course.

References

written by: Ulrich Habel (rhaen)

[/NetBSD] [permanent link]