Running iostat with timestamps
Have you ever encountered the problem how to track
the performance of your system? What about the memory? What about the
network? What about my disk activity? There are several tools which
collect data, maybe you already heard of mrtg. For a quick stress test
just to collect some data or if you are at a customers site and are not
allowed to install tools, just make it simple and use the standard Unix
tools. NetBSD has all the tools needed such as vmstat, iostat(8), systat
(nifty interface), nfsstat....
Just use them to collect data and plot them using gnuplot. You are able
to get all the data you want and produce some nifty graphs out of it
without the usage of Perl, etc. Unfortunatelly all these tools don't
have timestamps. However, we need those timestamps for our graphing
things. awk(1) to the rescue!
# iostat(8) -x 5 | awk '/wd0/ {print strftime("%H:%M:%S"),$0}' 15:44:20 wd0 5.17 0 0.00 0.00 17.62 0 0.00 0.00 15:44:25 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00 15:44:30 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00This will run the command iostat(8) every 5 seconds and shows the extended statistics. However, we just want to see the wd0 device, so we use awk(1) to grab the line with it from the output and put a timestamp in front of it. Note: Collecting data with iostat(8) every 5 seconds might be way too much. A period of writing data usually lasts longs than 5 seconds of a server life. In my experience about 30 second is just fine. Pipe the output into the tee(1) command - it looks geeky and you'll save it on your disk, too. If you want to process the data with gnuplot make sure to remove the _FIRST_ line of the output. This line is an average for the system since it's uptime and we might ruin our data collection with it as we just want to see the test data. If you have problems to capture the output, just run it inside a screen and use the command CTRL-A H to capture the output to the screenlog.0. I like this way as my shell might get disconnected, however the data is still captured.
After we collected the data over a certain amount of time, we are able to produce graphs out of it. Your customer will follow your argumentation more likely if they can see data visualized. Excel is, of course, a good choice - however you can use gnuplot to plot some fancy graphs out of it. When I was a consultant for IT things I used to carry a small USB stick with me. I stored gnuplot for Win32 on it and the basic script for gathering data. I ran the command iostat(8) on my Soekris for a couple of seconds to draw a graph out of it. It's nothing skyrocketing - just an example how things work together. Here is the gnuplot code I needed to plot the data from the iostat(8) command. It's the cvs update running on the Soekris (on a very slow cf card).
gnuplot <<_EOF_ set terminal png set out "iostat-rs_ws.png" set title "iostat during cvs up -dP" set xdata time set timefmt "%H:%M:%S" set xrange ["19:28:33":"19:50:08"] set xlabel "Time" set ylabel "Operations per second" set format x "%H:%M" plot "iostat.dat" using 1:4 title "r/s" with lines, \ "iostat.dat" using 1:8 title "w/s" with lines _EOF_Simple, eh? The file iostat.dat is the file which contains all the data from the iostat(8) command. The syntax using 1:4 is the way to say gnuplot what columns to get the data from. The graph shows the phase of cvs when the checks for updates are passed and the update begins. Funny, never saw it that way. Ok, anyway - now it's your turn. You have everything you need to draw your own stuff. Let's plot some fancy graphics. You can use gnuplot to visualize your sar output as well!
References/Notes This works for all kind of Unixes. If your awk complains about the syntax you should switch to gawk. I've tested it on NetBSD and it works just fine.
Update:
I noticed that my manpage plugin for Blosxom just substituted every iostat occurence with manpage link. This is fixed now. Sorry for the noise.Severe hard disk crash
The topic is sad and so is the news. The harddrive of my laptop failed badly,
it sounds like a childrens toy right now, the BIOS still recognises the
drive, however accessing the drive is impossible. It's also a sad news as this
harddrive isn't manufactured anymore and a replacement drive is far away. I
wrote an article about several CF card to IDE adapters as a solution for this
problem and right now I am on my way to setup my small NetBSD on a fresh
formatted hard drive (read: 8GB CompactFlash card). The cf card is of the 133x
speed class and is somehwat usable after some tweaks to NetBSD. There are
certain problems, however, this is part of this blog - we'll sort them out
over time and I'll write about it.
This is also the reason why p2c doesn't contain pkgsrc-wip in the next days.
I just needed the brand new CompactFlash card from my Soekris - it'll be
back shortly.
