1. Home
  2. Integrations
  3. Nagios Integration Guide – VictorOps

Nagios Integration Guide – VictorOps

Requirements

Versions Supported: Nagios 4.x and below

VictorOps Version Required: Standard or Enterprise

What you need to know Routing incidents from Nagios to multiple teams in VictorOps requires additional configuration.  (See Routing Incidents section of this article).  For Nagios environments secluded behind a firewall, see the Nagios Alerts Via Email section.

Nagios Plugin

VictorOps alert processing is implemented as a Nagios contact that is added to a contact group (often ‘admins’, but that will depend on your individual configuration).

The contact mechanism for the VictorOps contact is a simple shell script that spools the alert details to a file on disk. When an alert is fired, and Nagios invokes the contact script, the details wind up in /var/nagios and return to Nagios.

There is a long-running bash script that monitors /var/nagios for new files and posts the data in those files to VictorOps over HTTPS. This forwarding script is monitored by Nagios itself, and if it stops for any reason, the Nagios service check will attempt to restart it.

In the event that this forwarding script is unable to successfully send alerts to VictorOps for a time, it will fall back to sending an email version of the alert. The target address for this fallback alert is configurable (see below).

If you prefer not to install the plugin, see the section below on sending Nagios alerts via email.


Layout

The plugin files are installed to /opt/victorops/nagios_plugin. There is a Nagios configuration file called victorops.cfg in /opt/victorops/nagios_plugin/nagios_conf. This file contains all configuration for the plugin. After you’ve added your company ID, API key and Nagios host name, copy it to your Nagios configuration directory.


Installation

apt-get install
  1. Add the following lines to /etc/apt/sources.list:
    1
    2
    
    # VictorOps:
    deb http://software.victorops.com/apt precise main
  2. Execute
    1
    
    sudo apt-get update
  3. Execute
    1
    
    sudo apt-get install victorops-nagios

Note that we do not yet sign our deb package, so you will have to answer “Y” to the “Install these packages without verification” prompt.

yum install
  1. Download http://software.victorops.com/VictorOps-Public.repo and place it in the following directory.
    1
    
    /etc/yum.repos.d/
  2. Execute
    1
    
    sudo yum install victorops-nagios

Note that we do not yet sign our rpm package, so there is no GPG key to import at this time.


Configuration

If you install from the DEB or RPM packages, the installer will put the plugin files in their location in /opt/victorops/nagios_plugin and create the logging and alert directories.

After installation, modify the victorops.cfg file and copy it to your Nagios configuration directory. You’ll then need to modify your Nagios contacts configuration to add “VictorOps” to the contact groups (or individual services) for alerts you wish to be processed by VictorOps. We recommend that you send all alerts to VictorOps, but your individual may vary.

The victorops.cfg defines the VictorOps contact. The VictorOps contact to be added as a member of a Nagios “contact_group” for alerts to be sent to VictorOps. You may add the VictorOps contact to as many “contact_groups” as you would like, and we will use the “contact_group” name as the routing key (More info in Routing Incidents  section below).
Note: you may also add the VictorOps contact to specific services.

Sending alerts to VO is done via a shell script that requires the Nagios/Icinga environment macros. To enable this Nagios functionality, find the enable_environment_macros directive in /etc/nagios/nagios.cfg (actual path may vary) and make sure it is set to enable_environment_macros=1. If this directive does not exist, add it to the config file.

The Nagios plugin also requires the Nagios envronment macros, so Nagios must be configured to enable them. Add the following to your nagios.cfg file (or icinga.cfg): enable_environment_macros=1

Configure the following values in victorops.cfg:

  • _VO_ORGANIZATION_ID (~line 24 & 44) (case sensitive)
    • Can be found by accessing VictorOps Timeline and then looking at the URL.  The _VO_ORGANIZATION_ID will be the string that appears after ‘/client/’.  An example can be found below, where ‘my-company’ is the _VO_ORGANIZATION_ID

  • _VO_ORGANIZATION_KEY (~line 25 & 26)
    • Can be found by following Settings >> Alert Behavior >> Integrations >> Nagios.  It will be listed as ‘Service API Key’

These values are both the VictorOps_Contact_Settings (~line 20) contact and VictorOps_Service_Settings (~line 40) service object definitions. The values required in these fields are provided to you in the VictorOps Nagios integration settings page.

In VictorOps, select Settings >> Alert Behavior >> Integrations >> Nagios.

If the integration has not yet been enabled, click the “Enable Integration” button to generate your configuration values as seen here:

Also required:

(~Line 51)

This value is in the VictorOps_Service_Settings (line 40) service object definition. It is the name of your Nagios host, as defined to Nagios. It enables the heartbeat and command check services discussed below.

Additional configuration options:

_VO_MONITOR_NAME (~line 24 & 46)

This identifies the Nagios instance to VictorOps and may be left blank. If you are using multiple Nagios servers in your architecture, you should distinguish them with unique IDs in this field.

_VO_CONTACTEMAIL (~line 32)

A backup email address to send alerts to. If for any reason the plugin is unable to relay alerts to VictorOps, an alert email will be sent to this address. We recommend including an email-SMS gateway in this list. You may configure multiple addresses by separating them with spaces and enclosing the whole thing in single quotes:

‘me@mydomain.com you@mydomain.com him@mydomain.com 3035551212@vtext.com’

_VO_MAX_SEND_DELAY (~line 36)

The maximum amount of time (in seconds) that alerts will be allowed to remain in the queue before the alert is sent to the contact address above.


Additional services

These four services will appear on the Nagios server in the Nagios dashboard. By default, notifications for these services are disabled. If you wish to enable alerts for them, simply edit their service definitions in victorops.cfg.

VictorOps Alert Forwarder:

This is a process check for the long-running script described above. If this service goes critical, it will raise an alert via email (since normal alert forwarding can’t work when this service is down).

Heartbeat:

The victorops.cfg file defines a service to send heartbeat info to VictorOps. This service is enabled by default, and can be helpful in determining whether your plugin is working correctly, even if there are no alerts being generated by Nagios. Though today this service is just collecting info, it will eventually be used to generate alerts at VictorOps if your Nagios server seems to be malfunctioning or down.

Ack-Back Command Poll:

This service will poll VictorOps for commands to execute on your Nagios server. This service is disabled by default. The purpose is to allow commands issued at VictorOps to be relayed to your Nagios monitor. At this time, the only commands allowed by this service are host and service acknowledgements. Learn more.

Status Resync:

This service can send a complete Nagios status to VictorOps. It can be used in the event that VictorOps gets out of sync with your Nagios system. This might happen, for example, if you had notifications disabled in Nagios for a time. It requires cURL be installed on the Nagios host. There are two flavors, manual and auto. As you might guess, the manual flavor can only be invoked manually (via the Nagios console).

The auto version will run automatically, but is disabled (and commented out) by default. At this time, this is something of an experimental feature, so automatic execution is not recommended.


Verifying the Installation

After installing and configuring the plugin, you can verify functionality by using Nagios to send a custom notification for some service you have defined. The alert should be received by VictorOps and appear in your company timeline.

The contact script and alert forwarder write logs in /var/log/victorops. If the plugin does not seem to be working correctly, check these logs for errors.


Routing Incidents:

With the Nagios/Icinga plugin for VictorOps, the routing key sent to VictorOps is the name of whatever contact group contains the VictorOps contact.  If you would like Nagios to be able to route various incidents to multiple teams in VictorOps, you will need to create a unique contact, and unique contact group (with the one contact as the sole member) for each routing key you wish to use in VictorOps.  (Routing keys can be set up in VictorOps by clicking Settings>>Alert Behavior>>Route Keys).

In the below example, assume there are 3 teams in VictorOps that will be receiving incidents from Nagios. (DevOps, SRE, & Database)

First, define a contact for each team in your nagios configuration file, using the VictorOps_Contact settings that is defined in victorops.cfg:

DevOps Team Contact

1
2
3
4
5
6
define contact{
      use            VictorOps_Contact
      name           VictorOps_devops
      contact_name   VictorOps_devops
      alias          VictorOps_devops
      }

SRE Team Contact

1
2
3
4
5
6
define contact{
      use            VictorOps_Contact
      name           VictorOps_sre
      contact_name   VictorOps_sre
      alias          VictorOps_sre
      }

Database Team Contact

1
2
3
4
5
6
define contact{
      use            VictorOps_Contact
      name           VictorOps_database
      contact_name   VictorOps_database
      alias          VictorOps_database
      }

Next, define a unique contact group for each of the contacts defined above and add those contacts as the sole member, respectively.  The routing_key value used in the alert to VictorOps is derived from the contactgroup_name, so make sure that these names match the the values you wish to  use in VictorOps (or change the routing_keys in VictorOps to match the names you define here)

DevOps Contact Group (routing_key = devops)

1
2
3
4
5
define contactgroup{
      contactgroup_name     devops
      alias                 VictorOps DevOps contact group
      members               VictorOps_devops
      }

SRE Contact Group (routing_key = sre)

1
2
3
4
5
define contactgroup{
      contactgroup_name     sre
      alias                 VictorOps SRE contact group
      members               VictorOps_sre
      }

Database Contact Group (routing_key = database)

1
2
3
4
5
define contactgroup{
      contactgroup_name     database
      alias                 VictorOps Database contact group
      members               VictorOps_database
      }

Finally, add the contact groups to their appropriate check commands, and they will arrive with the correct routing key (contactgroup_name)


Nagios Alerts Via Email:

If your Nagios environment is restricted behind a firewall or if you simply would rather not install the plugin on your Nagios hosts, you can still send Nagios alerts to VictorOps via email. These alerts will show up in your timeline in a more limited format without on the extended functionality provided by the plugin.

To send Nagios alerts to VictorOps, simply create a Nagios contact using the sample configuration shown below, and add that contact to one of the Nagios contact groups that normally receives alerts from your system.

In the sample configuration given, the organization ID and organization key allow us to validate the alerts and route them to your timeline. The values can be found under the Integrations section of the VictorOps web app. The mail command in the configuration will format the alert details into the alert email appropriately.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
##------------------------------------------------------------------------------------------
## These Nagios contact and service definitions are used to pass configurable values to the email command.
##
## Contact settings:
## _VO_ORGANIZATION_ID
## _VO_ORGANIZATION_KEY
## These identify your alerts to VictorOps. The values for these fields are assigned to you by VictorOps.
##
## _VO_MONITOR_NAME
## VictorOps supports multiple Nagios instances per organization. This configuration value identifies the instance to
## VictorOps. It can be set to something you choose (such as the name of this Nagios host).
##
##------------------------------------------------------------------------------------------
 
define contact{
    contact_name    VictorOps_Email
    ## Configure these values as described above
    _VO_ORGANIZATION_ID    xxxxxxxxxxxxx
    _VO_ORGANIZATION_KEY    xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    _VO_MONITOR_NAME
 
    alias    VictorOps_Email
    service_notification_period    24x7
    host_notification_period    24x7
    service_notification_options    w,u,c,r
    host_notification_options    d,r
    service_notification_commands    notify-victorops-by-email
    host_notification_commands    notify-victorops-by-email
    register    1
    _VO_ALERT_DOMAIN    alert.victorops.com
}
 
define command{
    command_name     notify-victorops-by-email
    command_line /usr/bin/printf "%b" "\nVO_ORGANIZATION_ID=$_CONTACTVO_ORGANIZATION_ID$\nVO_ ORGANIZATION_KEY=$_CONTACTVO_ORGANIZATION_KEY$\n_CONTACTVO_ORGANIZATION_KEY=$_CONTACTVO_ORGANIZATION_KEY$\nVO_MONITOR_NAME=$_CONTACTVO_MONITOR_NAME$\n_CONTACTVO_MONITOR_NAME=$_CONTACTVO_MONITOR_NAME$\nTIMET=$TIMET$\nDATE=$DATE$\nTIME=$TIME$\nHOSTNAME=$HOSTNAME$\nHOSTALIAS=$HOSTALIAS$\nHOSTDISPLAYNAME=$HOSTDISPLAYNAME$\nHOSTSTATE=$HOSTSTATE$\nLASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$\nHOSTOUTPUT=$HOSTOUTPUT$\nHOSTPERFDATA=$HOSTPERFDATA$\nHOSTGROUPALIAS=$HOSTGROUPALIAS$\nHOSTGROUPNAME=$HOSTGROUPNAME$\nHOSTGROUPMEMBERS=$HOSTGROUPMEMBERS$\nHOSTGROUPNAMES=$HOSTGROUPNAMES$\nSERVICEDESC=$SERVICEDESC$\nSERVICEDISPLAYNAME=$SERVICEDISPLAYNAME$\nSERVICESTATE=$SERVICESTATE$\nLASTSERVICESTATECHANGE=$LASTSERVICESTATECHANGE$\nSERVICEOUTPUT=$SERVICEOUTPUT$\nSERVICECHECKCOMMAND=$SERVICECHECKCOMMAND$\nCONTACTGROUPNAME=$CONTACTGROUPNAME$\nNOTIFICATIONTYPE=$NOTIFICATIONTYPE$\nNOTIFICATIONAUTHOR=$NOTIFICATIONAUTHOR$\nNOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$\n" | /usr/bin/mail -s "$_CONTACTVO_ORGANIZATION_ID$:$_CONTACTVO_ORGANIZATION_KEY$:$_CONTACTVO_MONITOR_NAME$" $_CONTACTVO_ORGANIZATION_KEY$@$_CONTACTVO_ALERT_DOMAIN$
}

Centos 5 Timeouts:

To avoid timeouts when using Centos 5, you will need to link the timeout command to a directory that’s in the path.
First, create the symlink:

1
ln -s /usr/share/doc/bash-3.2/scripts/timeout /usr/bin/timeout

Next, make it executable:

1
chmod 755 /usr/share/doc/bash-3.2/scripts/timeout

 

Updated on November 7, 2017

Was this article helpful?

Related Articles