Rebuilding nagios at SGI
At work (SGI) I'm currently building a new
monitoring infrastructure using nagios.
It's coming along nicely. While working on it I sorted out a problem
trying to correctly check an LDAPS server and emailed someone who I'd
seen asking about the same problem (but not getting a reply). I let him
know what I'd found and in turn he mentioned that they were using a tool
called PNP4Nagios instead of cacti. Well, I'm not about to abandon Cacti
yet - it's just too darn good. But I did add PNP4Nagios to our new nagios
Briefly, it allows you to tell nagios to take the data a host/service
check returns and to hand it to some scripts which then log it into an
RRD file. There are "action_url" settings in nagios which you can use to
display the graphs generated on the fly from these RRD files. So things
like ping times, latencies on service checks like http, smtp, and ldap, can
all be tracked and graphed. Some service checks, obviously, don't lend
themselves to this such as the checks for RAID volumes (there's no numeric
data returned from the service check to graph!) but it's a nice addition
to nagios. It scales nicely too.
On that note, it looks like ndomod and nagvis, even though I like them,
don't scale quite as nicely. I'm beginning to see a slight cpu load on the
nagios server now that I've hit about 2800 service checks. I'll probably do
a bit of tuning on the mysql server to try to improve that some, but on a
four cpu core system I'm not too worried about it yet.
I decided to ingore the handful of web admin
interfaces I found for nagios. The only ones I found that looked like they had
an active developer community and also had decent flexibility were every
bit as complex as just editing the config files manually. So why bother?
The latest version of nagios is great! I've simplified our nagios
setup quite a lot and by making use of things like "multiple inheritance"
I've been able to make our nagios configs on the new server much
simpler and easier to manage.
Best of all, I think I'll be able to, eventually, auto-generate at least
part of our nagios config based on a simple LDAP query. We use a tool
internally known as DCSi (for historical reasons) to track assets - what
servers we have, where they are, what apps they run, who owns the hardware
and who owns the apps, who the emergency contacts are, any useful notes
about the server/device/app that someone on-call might need, etc. Basically,
it's really nothing more than a custom schema and an openLDAP server with
Apache DirectoryStudio. But it works great. And if I can have nagios
automagically start monitoring whatever servers/devices we document in
LDAP, that'd be pretty cool.
Anyway, that's what I'm shooting for in the long run, provided I can make
it simple enough that anyone in the group can feel comfortable maintaining
what I set up. What I'm envisioning is having most of the meat of the nagios
config setup in regular text config files, but then having a script/program
that would query LDAP for servers/devices and generate a config file for
each one using the appropriate templates (which would be specified along
with hostgroup membership in the DCSi LDAP record). One-off service checks
could be defined as children objects in LDAP in a similar fashion (defining
what templates to use and minimal nagios config info).