Icinga

Service and Host Freshness Checks

Introduction

Icinga supports a feature that does "freshness" checking on the results of host and service checks. The purpose of freshness checking is to ensure that host and service checks are being provided passively by external applications on a regular basis.

Freshness checking is useful when you want to ensure that passive checks are being received as frequently as you want. This can be very useful in distributed and failover monitoring environments.

How Does Freshness Checking Work?

Icinga periodically checks the freshness of the results for all hosts services that have freshness checking enabled.

[Tip] Tip

An active check is executed even if active checks are disabled on a program-wide or host- or service-specific basis.

For example, if you have a freshness threshold of 60 for one of your services, Icinga will consider that service to be stale if its last check result is older than 60 seconds.

Enabling Freshness Checking

Here's what you need to do to enable freshness checking...

[Tip] Tip

If you do not specify a host- or service-specific freshness_threshold value (or you set it to zero), Icinga will automatically calculate a threshold automatically, based on a how often you monitor that particular host or service. We would recommend that you explicitly specify a freshness threshold, rather than let Icinga pick one for you.

Example

An example of a service that might require freshness checking might be one that reports the status of your nightly backup jobs. Perhaps you have a external script that submit the results of the backup job to Icinga once the backup is completed. In this case, all of the checks/results for the service are provided by an external application using passive checks. In order to ensure that the status of the backup job gets reported every day, you may want to enable freshness checking for the service. If the external script doesn't submit the results of the backup job, you can have Icinga fake a critical result by doing something like this...

Here's what the definition for the service might look like (some required options are omitted)...

 define service{
        host_name               backup-server
        service_description     ArcServe Backup Job
        active_checks_enabled   0               ; active checks are NOT enabled
        passive_checks_enabled  1               ; passive checks are enabled (this is how results are reported)
        check_freshness         1
        freshness_threshold     93600           ; 26 hour threshold, since backups may not always finish at the same time
        check_command           no-backup-report        ; this command is run only if the service results are "stale"
        ...other options...
        }

Notice that active checks are disabled for the service. This is because the results for the service are only made by an external application using passive checks. Freshness checking is enabled and the freshness threshold has been set to 26 hours. This is a bit longer than 24 hours because backup jobs sometimes run late from day to day (depending on how much data there is to backup, how much network traffic is present, etc.). The no-backup-report command is executed only if the results of the service are determined to be stale. The definition of the no-backup-report command might look like this...

 define command{
        command_name    no-backup-report
        command_line    /usr/local/icinga/libexec/check_dummy 2 "CRITICAL: Results of backup job were not reported!"
        }

If Icinga detects that the service results are stale, it will run the no-backup-report command as an active service check. This causes the check_dummy plugin to be executed, which returns a critical state to Icinga. The service will then go into to a critical state (if it isn't already there) and someone will probably get notified of the problem.