The basic workings of service checks are described here...
Services are checked by the Icinga daemon:
On-demand checks are performed as part of the predictive service dependency check logic. These checks help ensure that the dependency logic is as accurate as possible. If you don't make use of service dependencies, Icinga won't perform any on-demand service checks.
The performance of on-demand service checks can be significantly improved by implementing the use of cached checks, which allow Icinga to forgo executing a service check if it determines a relatively recent check result will do instead. Cached checks will only provide a performance increase if you are making use of service dependencies. More information on cached checks can be found here.
You can define service execution dependencies that prevent Icinga from checking the status of a service depending on the state of one or more other services. More information on dependencies can be found here.
Scheduled service checks are run in parallel. When Icinga needs to run a scheduled service check, it will initiate the service check and then return to doing other work (running host checks, etc). The service check runs in a child process that was fork()ed from the main Icinga daemon. When the service check has completed, the child process will inform the main Icinga process (its parent) of the check results. The main Icinga process then handles the check results and takes appropriate action (running event handlers, sending notifications, etc.).
On-demand service checks are also run in parallel if needed. As mentioned earlier, Icinga can forgo the actual execution of an on-demand service check if it can use the cached results from a relatively recent service check.
Services that are checked can be in one of four different states:
Service checks are performed by plugins, which can return a state of OK, WARNING, UNKNOWN, or CRITICAL. These plugin states directly translate to service states. For example, a plugin which returns a WARNING state will cause a service to have a WARNING state.
When Icinga checks the status of services, it will be able to detect when a service changes between OK, WARNING, UNKNOWN, and CRITICAL states and take appropriate action. These state changes result in different state types (HARD or SOFT), which can trigger event handlers to be run and notifications to be sent out. Service state changes can also trigger on-demand host checks. Detecting and dealing with state changes is what Icinga is all about.
When services change state too frequently they are considered to be "flapping". Icinga can detect when services start flapping, and can suppress notifications until flapping stops and the service's state stabilizes. More information on the flap detection logic can be found here.
© 2009-2013 Icinga Development Team, http://www.icinga.org