The road of instability
It's annoying. This small Soekris device running NetBSD 4 is unstable as hell, however I found the reason for it. The flash card with the pkgsrc stuff on it has some failures from time to time. Think I need to replace it soon. Sorry for the outages, they should be fixed soon.
written by: Ulrich Habel (rhaen)
10/19/2008Time for Rockbox and Perl graphing
Wheeew, I was on holiday and thanks to FLOSS Weekly I decided to install
Rockbox on my iPod right before I left. It's quite amazing what these
guys reverse engineered and how well the support for the Apple iPod is.
I own a 30gb iPod video device - nothing really fancy, however I wanted
to have a look on it. No clue what I am talking about? Rockbox is an
alternative firmware for your MP3 player. You'll just update your iPod to
Rockbox and have a whole bunch of new features available. The most
impressive part is the "I-need-no-iTunes" part. Rockbox will find all
the songs on your iPod and builds a tag database out of it. Actually you
can use rsync to sync your music on the iPod. I really like it.
There are also alot of small applications bundled with Rockbox. One of
them is battery_bench. It's writes the current state of your battery
frequently to a textfile on the iPod disk. Do you want to know how long
the battery of your iPod lasts? Here is a way to draw a graph from the
collected data. It's quite simple, charge the iPod until it's fully
charged, play an album in an endless loop and run the battery_bench
tool. After the iPod turned off due to power loss, get the datafile from
the iPod and draw fancy graphs out of it.
I really had some nice feedback about the awk gnuplot thing and I wanted
to introduce you to another gem in our pkgsrc collection. The gem is
called p5-GDGraph. It's a great Perl module to draw graphics out of
data, so let's look into the example of the iPod.
The collected data is stored in a file called battery_bench.txt and looks
like this:
[...] Battery type: 400 mAh Buffer Entries: 1000 Time:, Seconds:, Level:, Time Left:, Voltage[mV]:, C:, S:, U: 02:18:34, 08314, 100%, 11:25, 4223, -, -, - 02:19:34, 08374, 100%, 11:25, 4223, -, -, - 02:20:34, 08434, 100%, 11:25, 4206, -, -, - [...]This structure can be easily caught by Perl. I will read the file line by line, and store the data inside an array. As I wanted to draw a graph out of the time for the x-value and the Voltage for the y-axis, I will put those values into two seperated arrays. After I got all the data from the file, I will build an array of arrays out of the data, this will be passed to p5-GDGraph to draw a nice graph out of it (just like the one in the article).
1.) Get the data out of the file
my $battery_bench = "battery_bench.txt";
my @plot_data;
my @x_data;
my @y_data;
# Pulling the data out of the data file
open DATA, $battery_bench
or die "Can't open file: $battery_bench\n";
while (<DATA>) {
# we don't want the explanation, just the data
if (m/^[0-9]{2}/) {
my $dataline = $_;
$dataline =~ s/\s//g;
my @dataparts = split( /,/, $dataline );
push( @x_data, $dataparts[0] );
push( @y_data, $dataparts[4] );
}
}
close DATA;
After we read the whole file we'll have all the data inside the two
arrays, @x_data and @y_data. You already guessed it, it's the data for the
X and for the Y values. The Perl module however expects the data inside
an array of arrays. The construction of it is quite easy and will be
shown in the next section. 2.) Pass the data to the graphing module GD::Graph::lines
# Setting up the graph
my $graph = GD::Graph::lines->new( 600, 300 );
@plot_data = ( [@x_data], [@y_data] );
$graph->set(
x_label => 'Time played',
y_label => 'Voltage (mV)',
title => 'Voltage graph of my iPod',
transparent => 0,
x_labels_vertical => 1,
x_label_skip => 25,
) or die $graph->error;
my $gd = $graph->plot( \@plot_data ) or die $graph->error;
Voila, you just plotted the first graph out of it. You can use different
output formats to get the actual graph as an image out of the Perl code.
# Write the graph to a file open( IMG, '>battery_bench.png' ) or die $!; binmode IMG; print IMG $gd->png; close IMG;It's a nice way to draw some quick graphs with Perl without messing around with RRD databases. Of course this module is also an excellent way to draw graphs out of iostat, sar and other tools. If you want to draw multiple lines inside a single graph just add another array to the @plot_data. The maintainer of the module inside the pkgsrc packages collection keeps this module always updated. He's already done a great job about it, thanks for keeping it uptodate.
You can find the full example with code and graphs inside the tar.gz file inside the references. References:
- iostat - gnuplot, drawing graphs
- Rockbox - Open Source Jukebox firmware
- Rockbox battery_bench
- CPAN GD::Graph
- pkgsrc.se p5-GDGraph
- FLOSS Weekly - Podcast with Leo Laporte and Randal Schwartz
- battery bench - Perl script and example data
written by: Ulrich Habel (rhaen)
09/30/2008Running iostat with timestamps
Have you ever encountered the problem how to track
the performance of your system? What about the memory? What about the
network? What about my disk activity? There are several tools which
collect data, maybe you already heard of mrtg. For a quick stress test
just to collect some data or if you are at a customers site and are not
allowed to install tools, just make it simple and use the standard Unix
tools. NetBSD has all the tools needed such as vmstat, iostat(8), systat
(nifty interface), nfsstat....
Just use them to collect data and plot them using gnuplot. You are able
to get all the data you want and produce some nifty graphs out of it
without the usage of Perl, etc. Unfortunatelly all these tools don't
have timestamps. However, we need those timestamps for our graphing
things. awk(1) to the rescue!
# iostat(8) -x 5 | awk '/wd0/ {print strftime("%H:%M:%S"),$0}' 15:44:20 wd0 5.17 0 0.00 0.00 17.62 0 0.00 0.00 15:44:25 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00 15:44:30 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00This will run the command iostat(8) every 5 seconds and shows the extended statistics. However, we just want to see the wd0 device, so we use awk(1) to grab the line with it from the output and put a timestamp in front of it. Note: Collecting data with iostat(8) every 5 seconds might be way too much. A period of writing data usually lasts longs than 5 seconds of a server life. In my experience about 30 second is just fine. Pipe the output into the tee(1) command - it looks geeky and you'll save it on your disk, too. If you want to process the data with gnuplot make sure to remove the _FIRST_ line of the output. This line is an average for the system since it's uptime and we might ruin our data collection with it as we just want to see the test data. If you have problems to capture the output, just run it inside a screen and use the command CTRL-A H to capture the output to the screenlog.0. I like this way as my shell might get disconnected, however the data is still captured.
After we collected the data over a certain amount of time, we are able to produce graphs out of it. Your customer will follow your argumentation more likely if they can see data visualized. Excel is, of course, a good choice - however you can use gnuplot to plot some fancy graphs out of it. When I was a consultant for IT things I used to carry a small USB stick with me. I stored gnuplot for Win32 on it and the basic script for gathering data. I ran the command iostat(8) on my Soekris for a couple of seconds to draw a graph out of it. It's nothing skyrocketing - just an example how things work together. Here is the gnuplot code I needed to plot the data from the iostat(8) command. It's the cvs update running on the Soekris (on a very slow cf card).
gnuplot <<_EOF_ set terminal png set out "iostat-rs_ws.png" set title "iostat during cvs up -dP" set xdata time set timefmt "%H:%M:%S" set xrange ["19:28:33":"19:50:08"] set xlabel "Time" set ylabel "Operations per second" set format x "%H:%M" plot "iostat.dat" using 1:4 title "r/s" with lines, \ "iostat.dat" using 1:8 title "w/s" with lines _EOF_Simple, eh? The file iostat.dat is the file which contains all the data from the iostat(8) command. The syntax using 1:4 is the way to say gnuplot what columns to get the data from. The graph shows the phase of cvs when the checks for updates are passed and the update begins. Funny, never saw it that way. Ok, anyway - now it's your turn. You have everything you need to draw your own stuff. Let's plot some fancy graphics. You can use gnuplot to visualize your sar output as well!
References/Notes This works for all kind of Unixes. If your awk complains about the syntax you should switch to gawk. I've tested it on NetBSD and it works just fine.
Update:
I noticed that my manpage plugin for Blosxom just substituted every iostat occurence with manpage link. This is fixed now. Sorry for the noise.written by: Ulrich Habel (rhaen)
Severe hard disk crash
The topic is sad and so is the news. The harddrive of my laptop failed badly,
it sounds like a childrens toy right now, the BIOS still recognises the
drive, however accessing the drive is impossible. It's also a sad news as this
harddrive isn't manufactured anymore and a replacement drive is far away. I
wrote an article about several CF card to IDE adapters as a solution for this
problem and right now I am on my way to setup my small NetBSD on a fresh
formatted hard drive (read: 8GB CompactFlash card). The cf card is of the 133x
speed class and is somehwat usable after some tweaks to NetBSD. There are
certain problems, however, this is part of this blog - we'll sort them out
over time and I'll write about it.
This is also the reason why p2c doesn't contain pkgsrc-wip in the next days.
I just needed the brand new CompactFlash card from my Soekris - it'll be
back shortly.
written by: Ulrich Habel (rhaen)
09/21/2008How do I calculate yesterdays date for shell scripts?
Well, you don't. I encountered this problem several times in my career as Unix admin and usually the answer is: you don't. Just use a different timezone and you are set. Let me explain the problem. Usually you want a cronjob to do work like grepping through logfiles with the date of the day before. This ends up in a horrible mix of expr and awks to find out the day yesterday. An even worse thing is to use perl to get the date the far worst thing is to install GNU date for this. Just look at the following example:
rhaen@wiesel.pkgbox.org:rhaen $ echo $(date) Sun Sep 21 12:03:20 CEST 2008 rhaen@wiesel.pkgbox.org:rhaen $ echo $(TZ=CEST23CEST date) Sat Sep 20 12:06:50 CEST 2008Explanation: We use the environment variable TZ (timezone) to set a timezone which is 23 hours before our current timezone. As we don't use EXPORT to set the timezone the environment is changed just for the only command. This is an easy way to get 23 hours back, plenty of time for your cronjob needs. The same method works on every timezone, of course.
References
written by: Ulrich Habel (rhaen)