Running iostat with timestamps
Have you ever encountered the problem how to track
the performance of your system? What about the memory? What about the
network? What about my disk activity? There are several tools which
collect data, maybe you already heard of mrtg. For a quick stress test
just to collect some data or if you are at a customers site and are not
allowed to install tools, just make it simple and use the standard Unix
tools. NetBSD has all the tools needed such as vmstat, iostat(8), systat
(nifty interface), nfsstat....
Just use them to collect data and plot them using gnuplot. You are able
to get all the data you want and produce some nifty graphs out of it
without the usage of Perl, etc. Unfortunatelly all these tools don't
have timestamps. However, we need those timestamps for our graphing
things. awk(1) to the rescue!
# iostat(8) -x 5 | awk '/wd0/ {print strftime("%H:%M:%S"),$0}' 15:44:20 wd0 5.17 0 0.00 0.00 17.62 0 0.00 0.00 15:44:25 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00 15:44:30 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00This will run the command iostat(8) every 5 seconds and shows the extended statistics. However, we just want to see the wd0 device, so we use awk(1) to grab the line with it from the output and put a timestamp in front of it. Note: Collecting data with iostat(8) every 5 seconds might be way too much. A period of writing data usually lasts longs than 5 seconds of a server life. In my experience about 30 second is just fine. Pipe the output into the tee(1) command - it looks geeky and you'll save it on your disk, too. If you want to process the data with gnuplot make sure to remove the _FIRST_ line of the output. This line is an average for the system since it's uptime and we might ruin our data collection with it as we just want to see the test data. If you have problems to capture the output, just run it inside a screen and use the command CTRL-A H to capture the output to the screenlog.0. I like this way as my shell might get disconnected, however the data is still captured.
After we collected the data over a certain amount of time, we are able to produce graphs out of it. Your customer will follow your argumentation more likely if they can see data visualized. Excel is, of course, a good choice - however you can use gnuplot to plot some fancy graphs out of it. When I was a consultant for IT things I used to carry a small USB stick with me. I stored gnuplot for Win32 on it and the basic script for gathering data. I ran the command iostat(8) on my Soekris for a couple of seconds to draw a graph out of it. It's nothing skyrocketing - just an example how things work together. Here is the gnuplot code I needed to plot the data from the iostat(8) command. It's the cvs update running on the Soekris (on a very slow cf card).
gnuplot <<_EOF_ set terminal png set out "iostat-rs_ws.png" set title "iostat during cvs up -dP" set xdata time set timefmt "%H:%M:%S" set xrange ["19:28:33":"19:50:08"] set xlabel "Time" set ylabel "Operations per second" set format x "%H:%M" plot "iostat.dat" using 1:4 title "r/s" with lines, \ "iostat.dat" using 1:8 title "w/s" with lines _EOF_Simple, eh? The file iostat.dat is the file which contains all the data from the iostat(8) command. The syntax using 1:4 is the way to say gnuplot what columns to get the data from. The graph shows the phase of cvs when the checks for updates are passed and the update begins. Funny, never saw it that way. Ok, anyway - now it's your turn. You have everything you need to draw your own stuff. Let's plot some fancy graphics. You can use gnuplot to visualize your sar output as well!
References/Notes This works for all kind of Unixes. If your awk complains about the syntax you should switch to gawk. I've tested it on NetBSD and it works just fine.
Update:
I noticed that my manpage plugin for Blosxom just substituted every iostat occurence with manpage link. This is fixed now. Sorry for the noise.written by: Ulrich Habel (rhaen)
Severe hard disk crash
The topic is sad and so is the news. The harddrive of my laptop failed badly,
it sounds like a childrens toy right now, the BIOS still recognises the
drive, however accessing the drive is impossible. It's also a sad news as this
harddrive isn't manufactured anymore and a replacement drive is far away. I
wrote an article about several CF card to IDE adapters as a solution for this
problem and right now I am on my way to setup my small NetBSD on a fresh
formatted hard drive (read: 8GB CompactFlash card). The cf card is of the 133x
speed class and is somehwat usable after some tweaks to NetBSD. There are
certain problems, however, this is part of this blog - we'll sort them out
over time and I'll write about it.
This is also the reason why p2c doesn't contain pkgsrc-wip in the next days.
I just needed the brand new CompactFlash card from my Soekris - it'll be
back shortly.
written by: Ulrich Habel (rhaen)
09/21/2008How do I calculate yesterdays date for shell scripts?
Well, you don't. I encountered this problem several times in my career as Unix admin and usually the answer is: you don't. Just use a different timezone and you are set. Let me explain the problem. Usually you want a cronjob to do work like grepping through logfiles with the date of the day before. This ends up in a horrible mix of expr and awks to find out the day yesterday. An even worse thing is to use perl to get the date the far worst thing is to install GNU date for this. Just look at the following example:
rhaen@wiesel.pkgbox.org:rhaen $ echo $(date) Sun Sep 21 12:03:20 CEST 2008 rhaen@wiesel.pkgbox.org:rhaen $ echo $(TZ=CEST23CEST date) Sat Sep 20 12:06:50 CEST 2008Explanation: We use the environment variable TZ (timezone) to set a timezone which is 23 hours before our current timezone. As we don't use EXPORT to set the timezone the environment is changed just for the only command. This is an easy way to get 23 hours back, plenty of time for your cronjob needs. The same method works on every timezone, of course.
References
written by: Ulrich Habel (rhaen)
09/08/2008Restore of my Soekris 4801
It's not easy to write new and exciting things for a blog when all the
readers are NetBSD insiders, however, here might be something that you
don't know. Do you remember when I wrote about the backup of my Soekris
4801 box with dump(8) over ssh(1)? I used a remote pipe and dumped right
on stdout. The stream was piped into cat(1) and was written to a disk on
a remote computer. Hubert F. sent me a message on IRC and asked about
the restore - just to keep it handy at one place. Today I had some
sparetime and decided to play around with qemu in order to test the
restore. I was quite amazing about the results (ok, it just worked).
I booted the qemu with a spare 1GB disk space (same size as my Soekris
CF card) and the NetBSD installation ISO image file. Booting from the
image was quite standard, however, a lot of the tools for the restore
are lacking. No ssh, no rsh - you name it. I remembered that the NetBSD
ftp client implementation is able to fetch file by http. I looked inside
the ftp manpage and found a possibility to use a pipe for stdout with
ftp. I set up a small webserver on my laptop and provided the dump file
in one of it's directories. I was able to recover the qemu image by
restore over ftp. That's cool, eh?
# newfs /dev/wd0a [...] # mount /dev/wd0a /mnt # ftp -o '| (cd /mnt; restore xf -)' http://10.0.2.2/mybackupfile.dmpAs you could see inside the screenshot I am ready to restore the small Soekris device. It's not a problem to use a user account with password, of course. I really like this way to recover a crashed system. It's really simple - NetBSD alike. Please note that this kind of restore just recovers your filesystem. In order to get a booting device you have to installboot(8) the bootblocks.
# installboot -m i386 -oconsole=com0,ioaddr=0x3f8,speed=9600 \
/dev/wd0d /usr/mdec/bootxx_ffsv1
What's left? We need a small bootimage for netbooting a Soekris. I am
working on it.
References:
written by: Ulrich Habel (rhaen)
08/31/2008The backup of my Soekris 4801
I've read an article about Unix system administration recently. One of
the golden rules, if I remember correctly it was the first one, is
backup your systems regulary. Backup - stop reading this article, think
of a backup. Well, I read the article till the end and looked at my
laptop - no backup. I looked at my Soekris - no backup, I looked at my
NetBSD toaster - no backup. Huh, time to do something like a backup. I
decided to look at all the fancy tools for this task like: bacula,
arkaya and several others. A posting from Mathias Scheler on -current
users brought the idea: why not use dump(8) for this purpose?
#!/bin/ksh # # Full Dump of Soekris TIMESTAMP=$(date "+%Y.%m.%d-%H%M") DMPFILE=/server/$TIMESTAMP-Soekris_Dump.dmp dump -0uan -f - / | ssh rhaen@wiesel "dd of=$DMPFILE"Voila - a level 0 dump using ssh to a different server. A restore test showed that this concept is working well. I'll setup a cron for this and dump on an external USB device. Did I mention I love the Soekris USB port of the 4801?
References:
written by: Ulrich Habel (rhaen)
