Running iostat with timestamps
Have you ever encountered the problem how to track
the performance of your system? What about the memory? What about the
network? What about my disk activity? There are several tools which
collect data, maybe you already heard of mrtg. For a quick stress test
just to collect some data or if you are at a customers site and are not
allowed to install tools, just make it simple and use the standard Unix
tools. NetBSD has all the tools needed such as vmstat, iostat(8), systat
(nifty interface), nfsstat....
Just use them to collect data and plot them using gnuplot. You are able
to get all the data you want and produce some nifty graphs out of it
without the usage of Perl, etc. Unfortunatelly all these tools don't
have timestamps. However, we need those timestamps for our graphing
things. awk(1) to the rescue!
# iostat(8) -x 5 | awk '/wd0/ {print strftime("%H:%M:%S"),$0}' 15:44:20 wd0 5.17 0 0.00 0.00 17.62 0 0.00 0.00 15:44:25 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00 15:44:30 wd0 0.00 0 0.00 0.00 0.00 0 0.00 0.00This will run the command iostat(8) every 5 seconds and shows the extended statistics. However, we just want to see the wd0 device, so we use awk(1) to grab the line with it from the output and put a timestamp in front of it. Note: Collecting data with iostat(8) every 5 seconds might be way too much. A period of writing data usually lasts longs than 5 seconds of a server life. In my experience about 30 second is just fine. Pipe the output into the tee(1) command - it looks geeky and you'll save it on your disk, too. If you want to process the data with gnuplot make sure to remove the _FIRST_ line of the output. This line is an average for the system since it's uptime and we might ruin our data collection with it as we just want to see the test data. If you have problems to capture the output, just run it inside a screen and use the command CTRL-A H to capture the output to the screenlog.0. I like this way as my shell might get disconnected, however the data is still captured.
After we collected the data over a certain amount of time, we are able to produce graphs out of it. Your customer will follow your argumentation more likely if they can see data visualized. Excel is, of course, a good choice - however you can use gnuplot to plot some fancy graphs out of it. When I was a consultant for IT things I used to carry a small USB stick with me. I stored gnuplot for Win32 on it and the basic script for gathering data. I ran the command iostat(8) on my Soekris for a couple of seconds to draw a graph out of it. It's nothing skyrocketing - just an example how things work together. Here is the gnuplot code I needed to plot the data from the iostat(8) command. It's the cvs update running on the Soekris (on a very slow cf card).
gnuplot <<_EOF_ set terminal png set out "iostat-rs_ws.png" set title "iostat during cvs up -dP" set xdata time set timefmt "%H:%M:%S" set xrange ["19:28:33":"19:50:08"] set xlabel "Time" set ylabel "Operations per second" set format x "%H:%M" plot "iostat.dat" using 1:4 title "r/s" with lines, \ "iostat.dat" using 1:8 title "w/s" with lines _EOF_Simple, eh? The file iostat.dat is the file which contains all the data from the iostat(8) command. The syntax using 1:4 is the way to say gnuplot what columns to get the data from. The graph shows the phase of cvs when the checks for updates are passed and the update begins. Funny, never saw it that way. Ok, anyway - now it's your turn. You have everything you need to draw your own stuff. Let's plot some fancy graphics. You can use gnuplot to visualize your sar output as well!
References/Notes This works for all kind of Unixes. If your awk complains about the syntax you should switch to gawk. I've tested it on NetBSD and it works just fine.
Update:
I noticed that my manpage plugin for Blosxom just substituted every iostat occurence with manpage link. This is fixed now. Sorry for the noise.written by: Ulrich Habel (rhaen)
Severe hard disk crash
The topic is sad and so is the news. The harddrive of my laptop failed badly,
it sounds like a childrens toy right now, the BIOS still recognises the
drive, however accessing the drive is impossible. It's also a sad news as this
harddrive isn't manufactured anymore and a replacement drive is far away. I
wrote an article about several CF card to IDE adapters as a solution for this
problem and right now I am on my way to setup my small NetBSD on a fresh
formatted hard drive (read: 8GB CompactFlash card). The cf card is of the 133x
speed class and is somehwat usable after some tweaks to NetBSD. There are
certain problems, however, this is part of this blog - we'll sort them out
over time and I'll write about it.
This is also the reason why p2c doesn't contain pkgsrc-wip in the next days.
I just needed the brand new CompactFlash card from my Soekris - it'll be
back shortly.
written by: Ulrich Habel (rhaen)
09/21/2008How do I calculate yesterdays date for shell scripts?
Well, you don't. I encountered this problem several times in my career as Unix admin and usually the answer is: you don't. Just use a different timezone and you are set. Let me explain the problem. Usually you want a cronjob to do work like grepping through logfiles with the date of the day before. This ends up in a horrible mix of expr and awks to find out the day yesterday. An even worse thing is to use perl to get the date the far worst thing is to install GNU date for this. Just look at the following example:
rhaen@wiesel.pkgbox.org:rhaen $ echo $(date) Sun Sep 21 12:03:20 CEST 2008 rhaen@wiesel.pkgbox.org:rhaen $ echo $(TZ=CEST23CEST date) Sat Sep 20 12:06:50 CEST 2008Explanation: We use the environment variable TZ (timezone) to set a timezone which is 23 hours before our current timezone. As we don't use EXPORT to set the timezone the environment is changed just for the only command. This is an easy way to get 23 hours back, plenty of time for your cronjob needs. The same method works on every timezone, of course.
References
written by: Ulrich Habel (rhaen)
09/13/2008multimedia/ffmpeg-devel - Converting to iPod video
Do you know the ffmpeg-package? It's very useful to convert nearly any kind of video to an iPod compatible video format. Just convert your favorite video from the googletech channel on YouTube to an iPod format and take it with you. That's quite simple. Just take a video you'd like to convert and use the following commandline to convert it:
ffmpeg -i foo.avi -f mov -b 1800 -vcodec mpeg4 -qmin 3 \
-qmax 5 -s 320x180 -acodec libfaac -ab 128 foo.ipod.mov
Please note - this example is for a 16:9 aspect ratio, change the
-s parameter to 320x240 for 4:3 aspect radio.Update:
I was asked if there is a way to detect the resolution of a video file. There certainly is and it's just easy. As we are using ffmpeg to detect the solution we could use it to identify the resolution for us.
ffmpeg -i sample.flv 2>&1 | awk '/Stream/ && /Video/ {gsub(/,/," ");print $6}'
If you want to check the results mplayer is also a handy tool.
mplayer -identify sample.flv -ao null -vo null -frames 0 2>/dev/null | \ grep -E -e '^ID_VIDEO_(WIDTH|HEIGHT)'btw, don't blame me if you Linux box bails out - this is NetBSD land
written by: Ulrich Habel (rhaen)
pbulk - The diff(erence)
Joerg Sonnenberger is the creator of pbulk - a cool tool for running software bulk builds on pkgsrc. I really like it's way to build things, it creates reproducable build results, generates nice reports, I like it. However, a few problems with pbulk, at least for me. I don't like the way the config file pbulk looks like. I usually enjoy preconfigured files with comments. I made a config file which suits my needs perfectly and I am using it on NetBSD 4, NetBSD -current on i386 and for the builds on Sun Solaris 9/sparc.
The patch for it was denied some time ago as it was to restrictive for NetBSD. I worked on it, however I never submitted a new version for it. Anyway, here it is, the diff to the regular pbulk.conf diff. Just apply it, give me feedback about it - enjoy it. For me it's an easy way to set up things, so I put it on the webpage.
cd /usr/pkgsrc ftp -o - -V http://pkgbox.org/Sources/pbulk.conf.diff | patch -p0Please note: I will try to keep this file updated, but maybe there will be a few days until this file is fixed after pbulk changed a few things. It's fine to use this diff for NetBSD and non NetBSD-platforms.
References:
Update
There was an error in the command line a- was missing. Sorry about it
written by: Ulrich Habel (rhaen)
09/08/2008Restore of my Soekris 4801
It's not easy to write new and exciting things for a blog when all the
readers are NetBSD insiders, however, here might be something that you
don't know. Do you remember when I wrote about the backup of my Soekris
4801 box with dump(8) over ssh(1)? I used a remote pipe and dumped right
on stdout. The stream was piped into cat(1) and was written to a disk on
a remote computer. Hubert F. sent me a message on IRC and asked about
the restore - just to keep it handy at one place. Today I had some
sparetime and decided to play around with qemu in order to test the
restore. I was quite amazing about the results (ok, it just worked).
I booted the qemu with a spare 1GB disk space (same size as my Soekris
CF card) and the NetBSD installation ISO image file. Booting from the
image was quite standard, however, a lot of the tools for the restore
are lacking. No ssh, no rsh - you name it. I remembered that the NetBSD
ftp client implementation is able to fetch file by http. I looked inside
the ftp manpage and found a possibility to use a pipe for stdout with
ftp. I set up a small webserver on my laptop and provided the dump file
in one of it's directories. I was able to recover the qemu image by
restore over ftp. That's cool, eh?
# newfs /dev/wd0a [...] # mount /dev/wd0a /mnt # ftp -o '| (cd /mnt; restore xf -)' http://10.0.2.2/mybackupfile.dmpAs you could see inside the screenshot I am ready to restore the small Soekris device. It's not a problem to use a user account with password, of course. I really like this way to recover a crashed system. It's really simple - NetBSD alike. Please note that this kind of restore just recovers your filesystem. In order to get a booting device you have to installboot(8) the bootblocks.
# installboot -m i386 -oconsole=com0,ioaddr=0x3f8,speed=9600 \
/dev/wd0d /usr/mdec/bootxx_ffsv1
What's left? We need a small bootimage for netbooting a Soekris. I am
working on it.
References:
written by: Ulrich Habel (rhaen)
09/04/2008Ready for tomboy (0.11.3)!
Hooray! Do you know tomboy? Well,
ok - tomboy is a small desktop application for the Gnome desktop. It's
just a small sticky notes application. Tomtom is written in Mono and
requires tons of dependencies. But why should someone with a computer
should small sticky notes? You've already guessed it - for writing down
small things that you won't forget. I just read a book about Getting
things done by David Allen. He says everything what's in your mind is an
open loop. Your brain keeps you reminding frequently about these items,
you should write them down. As I am a person who writes alot of stickies
I decided to look for an application which I could use for such purpose.
Tomboy has some really great features for such an idea. It allows you to
link your stickies together. So you can have your project stickies and
write down several task. If you click on the task a new sticky shows you
more about this task. That's great! Of course, there is HTML export,
Evolution support, plugin support, plus plus plus - the feature list is
amazing.
I decided to give tomboy a shot and couldn't find it inside
pkgsrc, not even in wip. So I made my own pkgbox pkg. pkgbox pkgs are my
own personal packages which I make to keep things together. pkgsrc is a
great framework and building a package is usually quite easy. I won't
import it into pkgsrc, neither to wip - take this package as
it-works-for-me package and feel free to use it. Send me some feedback
if it's working for you and if you think that I should import it
somewhere. I am not a mono guy, I don't even understand the problems :)
so, feel free to send me your coments about this.
The homepage
for tomboy is located at Gnome - use this link
Download tomboy 0.11.3 pkgsrc pkg
You'll get a tar.gz archive. Just untar it inside your pkgsrc directory.
It will create a structure like pkgbox/tomboy in it (just like wip).
rhaen@arkanum.pkgbox.org:rhaen $ cd /usr/pkgsrc rhaen@arkanum.pkgbox.org:pkgsrc $ tar xzf ~/tomboy-pkg.tar.gz rhaen@arkanum.pkgbox.org:pkgsrc $ cd pkgbox/tomboy rhaen@arkanum.pkgbox.org:pkgsrc $ make package
written by: Ulrich Habel (rhaen)
