Stop Using Email Notifications In Cron Jobs

The more I work with different IT infrastructures, as a new employee taking over for previous admins, after acquiring another company and adopting some of their infrastructure or being acquired and having to migrate/retire our infrastructure (yeah, I've been involved in all of those), the more I've come to truly detest email notifications for cron/scheduler jobs and email notifications in general. Why? I find they can be grouped into several categories of bad.

The first is the "All is well" email notifications - you know, the ones you get at least once a day, that everyone has setup an email filter for that nobody looks at and are either filtered into a separate, giant folder or just get deleted. What's the point of an email notification that nobody looks at for cryin' out loud! And then every now and then, mixed in with thousands of ignored "All is well" emails, there will be one "All is NOT well" emails that nobody has looked at because they're all filtered off to a folder the recipient doesn't look at. Oops.

Then there's the email notifications sent to an undeliverable alias or mailing list or a person who hasn't been with the company in a decade. How could this be? Picture, IT geek Fred. He's been with the company since it's founding a decade earlier when the entire IT department was just him. Now the company is bigger and has a half dozen IT geeks. Hardware has come 'n gone and Fred leaves the company. Any bets as to how many devices have email notifications pointing to Any bets as to how many will be found and fixed? These only send an email when something is NOT well, so it's not like one can just look in the email logs to see what devices are trying to send email to Fred. Someone has to look at every single device. When people in IT are operating at a dead-run already, that's just not gonna happen. Typically, nobody notices an old piece of hardware desperately calling Fred for help until some an email or security admin happens to notice hundreds of bounced email to Fred. Or more often, the device will finally have enough failures that no redundancy is redundant enough and it fails spectacularly and management starts asking "why did nobody know that this device had 2 bad fans, one bad power supply and three failed hard drives?" Oops.

Another category is the cron/sechudler jobs that Fred setup over the years. Inevitably, these are critical pieces of automation in the IT infrastructure that nobody even knows exist since they mostly just quietly do their job... until one breaks for some reason and sends an email to Fred saying "Help! Fix me!". Then someone in IT has to do IT archaeology figuring out why some database transaction log is filling up now or why support tickets aren't magically getting sync'd to the support FTP server or whatever. Not fun.

So now that I've gotten that off my chest... Yeah, email notifications suck. They either wind up not being seen or going to the wrong place or otherwise wind up being more of an annoyance than a solution. So what are we to do?

At my last job, I inherited a metric crap-ton of systems which had been setup by people who'd long-since left the company running apps with no support contract, setup by other people who'd been gone a decade or more, and each with critical cron jobs that nobody in the IT or Apps departments knew anything about. Bummer! So any time I found one of these, I dug in my heels and found an owner for whatever process the job was supposed to accomplish and setup a passive nagios service for that cron job. The I edited it searching for any exit conditions. Any time the script exited, it would run a command to notify nagios about the success or failure of this job.

Why is this better? Well, because now the script can run as often as needed and report "all is well" to nagios who will dutifully note it without filling up anyone's mailboxes. Plus if it ever does exit with some fault, it notifies nagios of that fault - it doesn't go unnoticed. Now, it's also critical that nagios (or whatever monitoring server you use) be kept up to date with correct mailing addresses. But that's MUCH more likely to be kept up to date and much easier to keep up to date than to have to go edit a half a jillion cron jobs and hardware configs all over the world.

The more I did this, the more I fell in love with passive service checks in nagios. Any time we setup a new job the first thing after making sure the cron job worked was to setup a passive check for it to send notification to. For instance, here's the nagios checks for a database server:

Yeah, I'm a big believer in monitoring. (grin) If it can be monitored, we probably monitor it. Many of these service checks were just to gather performance data to render on a dashboard like ethernet or SAN switch traffic stats or CPU usages. But notice the last two service checks? They're the ones that have an icon that says PASV with two arrows pointing down. One of those is for a cron job that quiesces the database, then takes a snapshot of the filesystem the data is on, then releases the database and takes a snapshot of the filesystem the logs are on. The other check is for a cron job that uses these snapshots to create volumes we can then backup.

So if either job fails it exits with a status telling nagios that all is not well and giving us some sort of useful error as to what happened. But what if the cron job just doesn't run at all so no error conditions are reported? Nagios (and any decent monitoring system) can take care of this too. In nagios you just set the freshness_interval and setup the check so most of the time nagios doesn't actively check this service, it just sits passively listening for an update. If it doesn't get one within the freshness interval, it then runs whatever you've specified for the active check, and you specify a script that always returns a critical or unknown status (whatever you'd prefer) So if it gets an update from the cron job, great. If nothing is heard within your freshness interval (say, 25 hours for a job that runs once a day and takes around 30 mins to run), this service check will revert to critical or whatever you've specified for the active check.

Here's what my nagios config looks like for this service check:

# my-server create snapshot check
# Go critical if no notification received within 25 hours
# the stalking_options are so that nagios will log all checks, not
# just transitions from one state to a different one
define service {
	use			local-service
	host_name		my-server
	service_description	create snapshot
	check_command		check_dummy_crit
	flap_detection_enabled	0
	max_check_attempts	1
	active_checks_enabled	0
	passive_checks_enabled	1
	check_freshness		1
	freshness_threshold	90000
	stalking_options	o,w,c

See the check_dummy_crit command on the check_command line? That runs a script that just always returns a CRITICAL status to nagios. And you can see that passive checks are enabled and active checks are disabled. I also set the flap_detection off because I don't want it to squelch alerts if the cron job fails X days in a row. If memory serves, I set the stalking options because I wanted nagios to log all the successes (which includes the names of the created snapshots) as well as any failures. In other words, even if the last notification was a success and the most recent notification was also a success, I wanted nagios to log it, not just the changes in status. It just made it more convenient for the on-call if there was a problem and he/she needed to roll back a filesystem to a previous snapshot - they were all logged in nagios.

One thing to be careful with (at least in nagios) is to not get in the habit of just disabling and re-enabling all checks for all services on a host. Get in the habit of acknowledging alerts or disabling notifications instead of disabling service checks. This is because if you use the link to disable all checks (for instance) fix a problem and then enable all checks for a host you will wind up re-enabling active checks for your passive services also and then nagios will run the check_dummy_crit call rather than waiting for the next passive update from the script.

So what does it look like from the script's perspective? It turns out to be very easy. I'm using nscd and send_nsca but I encourage you to look for newer tools. I think there's a newer tool that allows for multi-line passive updates for instance. But here's how it works with send_nsca and nscd. You just configure nscd on your nagios server (consult the man pages - it's easy), then in your script you run send_nsca. In my case, my cron job is a perl script so I added a function to run send_nsca which I could call from any point within the script:

# This is a snippet from a cron job that creates snapshots of our database

# my notify_nagios perl function:
sub notify_nagios ($$$$) {
   my $naghost = shift;
   my $nagservice = shift; 
   my $nagstatus = shift;
   my $nagmsg = shift;

   my $NAGIOS = "/usr/local/nagios/nrpe/bin/send_nsca -H my-nagios-server -c /usr/local/nagios/nrpe/etc/send_nsca.cfg >& /dev/null";

   open(NAGIOS,"| $NAGIOS");
   print NAGIOS "$naghost\t$nagservice\t$nagstatus\t$nagmsg\n";

# ... a bunch of geeky DB and storage stuff ...

# At this point, $result is a numeric result code like
# 0 = OK

# And $resstr is a text string containing a textual status like
# "CRITICAL: Failed to create a snopshot for $volname"   or...
# "OK: Created snapshot for $volume on controller $ctrlr"

# Tell nagios the status
notify_nagios ($hostname,"create snapshot",$result,$resstr);

And from a bash script it can be as easy as running echo with some tab separated parameters and piping that to send_nsca. Easy. So let nagios log success and failures for you and let it sort out who should be notified when under which conditions. You'll save yourself no end of headaches in the future as people in your IT department come and go.