1. Home
  2. Integrations
  3. Zabbix Integration Guide – VictorOps

Zabbix Integration Guide – VictorOps

Requirements

Versions Supported: Zabbix 3.x and below, PHP version >= 5.3.0, CURL

VictorOps Version Required: Standard or Enterprise

What you need to know We offer two ways to integrate with Zabbix. The plugin method is recommended, but you may also integrate using our REST Endpoint API (requires manual configuration).

Zabbix is an enterprise-class open source distributed monitoring solution for networks and applications. We recommend you use the plugin, but you may also manually configure alerts using the REST Endpoint.


Configuration in VictorOps

In VictorOps please go to Settings -> Alert Behavior -> Integrations. Our integrations are arranged in alphabetical order.

Please scroll down and select the Zabbix icon from the list of available integrations.

Once selected, please Enable Integration.

From here you may copy the Service API Endpoint.

 


The Plugin

VictorOps’ Zabbix plugin is a set of scripts that queue up trigger notifications and forward them to VictorOps servers for processing. The queuing up of notifications is done in-band as Zabbix runs the notification script, and forwarding is done out-of-band by a long-running process.

The forwarding process can retry sending the notification, and if the send fails for any reason, for an extended period of time (1 minute by default), it will fall back to sending an email to a configured address.


Layout

The main package files are installed to /opt/victorops/zabbix_plugin

Notifications are queued in /var/victorops/zabbix-alerts

Logging is written in /var/log/victorops

Symbolic links are created in the Zabbix alert scripts and external scripts directories (probably /usr/lib/zabbix/alertscripts/ and /usr/lib/zabbix/externalscripts/) for some of the files in /opt/victorops/zabbix_plugin/bin.


Installation

The package is available as an RPM, DEB or tarball, and can be installed via yum or apt-get by adding VictorOps to your package sources.

apt-get

Add the following lines to /etc/apt/sources.list

1
2
# VictorOps:
add-apt-repository 'deb http://software.victorops.com/apt precise main'

Execute:

1
sudo apt-get update

Execute:

1
sudo apt-get install victorops-zabbix

Note that we do not yet sign our deb package, so you will have to answer “Y” to the “Install these packages without verification” prompt.

yum

Download http://software.victorops.com/VictorOps-Public.repo and place it in your /etc/yum.repos.d/ directory.

Execute:

1
sudo yum install victorops-zabbix

Note that we do not yet sign our rpm package, so there is no GPG key to import at this time.


Configuration

If you install from the DEB or RPM packages, the installer will put the plugin files in their location in /opt/victorops/zabbix_plugin. Additionally, two directories will be created,/var/victorops/zabbix-alerts and /var/log/victorops, for notification queuing and logging, respectively.

Once installed, the package will need configuration by running the /opt/victorops/zabbix_plugin/bin/configure.sh script. The configure script will connect to the Zabbix server, so to use it you will need to provide a Zabbix admin user ID and password. The script will prompt you for these.

Additionally, you will need to provide your organization “slug” and Zabbix alert key to the configure.sh script. These are found on your Zabbix integration page in VictorOps.  (Settings>>Alert Behavior>>Integrations>>Zabbix)

You may also provide an email address as the fallback contact in the unlikely event that sending a notification to VictorOps fails.


Configuration Elements

The configure script connects to the Zabbix server and, using the Zabbix API, adds several elements required to forward notifications to VictorOps. These are:

  • A Zabbix user “VictorOps Alert
  • A Zabbix user group “VictorOps Users
  • A Zabbix media type “Notify VictorOps
  • A Zabbix action “Notify VictorOps“. The action includes the specific message content required for processing and is invoked by Zabbix for any trigger in problem status, including recovery notifications.
  • A Zabbix item “VictorOps forwarder” to the host “Zabbix Server” for the VictorOps forwarder process. This item is an “external check” type. When Zabbix runs a check of this item, it will start the forwarder process if necessary.

Configure Ack-Back to Zabbix

To enable the ack-back feature for Zabbix, do the following:

Copy /opt/victorops/zabbix_plugin/conf/local.zabbix.conf.example to /opt/victorops/zabbix_plugin/conf/local.zabbix.conf

Open /opt/victorops/zabbix_plugin/conf/local.zabbix.conf in an editor and set a user and password allowed to log-in to the Zabbix console with “acknowledge” permissions

ZABBIX_USER=admin
ZABBIX_PASSWD=fooblyboo

You may wish to change the owner of the file to the Zabbix user, and set permissions to 600.

On the Zabbix console, navigate to the “Items” list for your Zabbix server and enable the VictorOps Command Poll item.


Configure Multiple Zabbix Servers:

If you will be using the VictorOps integration on multiple Zabbix servers, you will need to update the Zabbix Action so that it includes the MONITOR_NAME field. You must add this to both the Default and Recovery message bodies.

zabbixold3

The MONITOR_NAME field should be unique for each Zabbix server that has the plugin installed.

The matching MONITOR_NAME value needs to be added to the configuration file found at /opt/victorops/zabbix_plugin/conf/local.zabbix.conf

These changes will, unfortunately, also have the effect of disconnecting the items on the Zabbix side from the VictorOps side, meaning that if there are any currently open incidents in VictorOps they will have to be manually resolved in VictorOps.

Finally, you will need to enable the VictorOps Command Poll item, just like in the previous section Configure Ack-Back to Zabbix.


Routing Incidents in VictorOps

Any incidents created by notifications sent to VictorOps will be processed on the default routing rule (Settings>>Alert Behavior>>Routing). If you prefer to use another routing key in VictorOps, you can modify the media for the VictorOps Alert user and put your VictorOps routing key in the Send to field.

Access the VictorOps Alert user:

zabbixold1

Change the Send To to the routing key you wish to use.

zabbixold2

To create multiple or complex routing schemes, you may need to create new Zabbix actions, users and or media types for the routing keys involved.


Notification Handling

The action to notify VictorOps saves the notification details to a file on disk. Within a couple seconds, the forwarder process will send the details to VictorOps, where the normal alert processing will be performed.

If forwarding the details is unsuccessful, the timestamp of the details is checked, and if the details are old (1 minute by default), the details are instead sent to the backup email address you provided at configuration time.

NOTE: If you want to shorten the payload (Default Message) sent from Zabbix to VictorOps, you must leave the following minimum required fields:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
VO_ORGANIZATION_ID= YOUR_ORG_SLUG_HERE
CONTACTEMAIL= YOUR_EMAIL_HERE
ESC.HISTORY={ESC.HISTORY}
EVENT.ACK.HISTORY={EVENT.ACK.HISTORY}
EVENT.ACK.STATUS={EVENT.ACK.STATUS}
EVENT.DATE={EVENT.DATE}
EVENT.TIME={EVENT.TIME}
HOSTNAME={HOSTNAME}
HOST.NAME={HOST.NAME}
HOST.NAME1={HOST.NAME1}
TRIGGER.KEY={TRIGGER.KEY}
TIME={TIME}
TRIGGER.ID={TRIGGER.ID}
TRIGGER.NAME={TRIGGER.NAME}
TRIGGER.NSEVERITY={TRIGGER.NSEVERITY}
TRIGGER.SEVERITY={TRIGGER.SEVERITY}
TRIGGER.STATUS={TRIGGER.STATUS}

REST Endpoint (Manual Configuration)

If you prefer not to use the recommended plugin, you can The following describes how to set up the Zabbix integration using our Generic REST Endpoint to send alerts from Zabbix into the VictorOps timeline.

 

1.Copy the following script into the /usr/lib/zabbix/alertscripts directory:

1
2
3
4
#!/bin/bash
api_key=$1
message=$3
/usr/bin/curl -vv --data-binary "$message" http://alert.victorops.com/integrations/generic/20131114/alert/YOUR_API_KEY/YOUR_ROUTING_KEY > /tmp/zabbix.out 2>&1

2. Give it a name (for this example zabbix-critical.sh)

3. From the Zabbix UI make a new media type

zabbixold4
  • Administration → Media Types → Create Media Types
  • Name → (“VO Alert” for example)
  • Type → Script
  • Script name → zabbix-critical.sh
  • Enabled → Check
zabbixold5

 

4.Make a VictorOps User and user group

  • Administration → Users → Create User

zabbixold6

 

5. Configure User Media

  • Media Type → Vo Alert
  • Send to → Your VictorOps API Key Here
  • Select the severity you want to alert VictorOps of.
  • Status → Enabled

zabbixold7

 

6. Set User Permissions

  • Select the host group(s) and host(s) that you want VictorOps to have access to
  • Save User
  • Create a VictorOps user group
  • Add the VictorOps user to the group
  • Set permissions

 

7. Create an Action

  • Configuration → Actions → Create Action
  • Select the recovery message box
  • Give both a name and subject line
zabbixold8

 

 

8. Set the Default messages

zabbixold9

 

In the first box (alert message) place the following as the default message:

1
2
3
4
5
6
7
8
{
"message_type": "critical",
"state_message": "Zabbix Alert:{HOSTNAME}{TRIGGER.NAME}",
"entity_display_name": "{TRIGGER.NAME}",
"entity_id": "{TRIGGER.ID}",
"host_name":"{HOSTNAME}",
"monitoring_tool":"Zabbix"
}

In the second box (recovery message) place the following as the default message:

1
2
3
4
5
6
7
8
{
"message_type": "recovery",
"state_message": "Zabbix Alert:{HOSTNAME}{TRIGGER.NAME}",
"entity_display_name": "{TRIGGER.NAME}",
"entity_id": "{TRIGGER.ID}",
"host_name":"{HOSTNAME}",
"monitoring_tool":"Zabbix"
}

Select Enabled

9. Configure the Conditions

 

10. Configure the Operations

Alert/Recover operation:

  • Operation type → Send message
  • Send to User Groups → VictorOps User group
  • Send to User → VictorOps User
  • Send only to → VO Alert (in this example)
  • Default Message → selected
  • Update Operation
zabbixold11

 

 

11. Save your Action and you are done. Check by looking at a recent Event that would set off a trigger and look for a successful action.

Updated on June 7, 2017

Was this article helpful?

Related Articles