After seeing a nice traffic graph on a friend's site, I decided to install mrtg. Being a perfectionist with slow boxes, the defaults (ie. use mainly snmp) didn't make me happy.
I ended up writing about 10 small perlies that either query /proc or sift through logfiles. It worked, but not very nicely: lotsa perl processes starting every 5 minutes and quite some code duplication. Rethink. Improve.
So here's the new, combined, all-in-one solution: a single script that does the data gathering all in one go. It's called bigstat.
bigstat reads a couple of simple things from /proc (memory usage, interface traffic counters, cpu stats and load average), but all that is hardly new and not worth talking about. It also gets at the firewall packet drop counters and plots df -k vs. df -i.
What is notable IMHO is that bigstat also deals with sources that are decidedly not nice enough to provide convenient counters or gauges: my apache access log for example, the mail logs and my inn logs (the "common" way of getting apache info for example involves parsing the html output of mod_status...bleh, for mail people have parsed mailstats output and for inn there's nothing ready-made except for CNFS storage.)
My approach there is to use some builtin logtail functionality together with persistent counter and offset files: go over the unread/new parts of the log, look for some useful pattern and increase the counters. Then save the last counter states in the persistent counter file and also remember where in the file we were (plus inode). This guarantees counter continuity regardless of log rotation. bigstat then writes all the fourliner data mrtg expects into separate files, and mrtg just reads them via a 2-liner shell wrapper around cat. This works great and is very efficient.
If you need some inspiration to set up something similar: help yourself to my examples here. The script and mrtg config file are gpl'd. The proc stuff and iptables-usage are linux-specific, and the regexp matches will definitely need adjustment for your environment.
- cleansed mrtg config
- bigstat script
- read-bigstat script
