1. Home
  2. Integrations
  3. Splunking VictorOps Data

Splunking VictorOps Data

These reports can provide real-time visibility across multiple VictorOps instances and offer highly granular and customizable reporting.

In order to leverage this capability, there are two ways to ingest data.
1. Splunk VictorOps Add-on which leverages GET requests to the VictorOps public API.
2. Custom Outgoing Webhooks sent to Splunk from VictorOps

VictorOps Add-On

The VictorOps add-on for Splunk is a downloadable add-on (similar to an app) that will ingest VictorOps data into Splunk using the VictorOps public API.

The add-on is installed on a heavy forwarder and will play nice with any other add-ons also installed. It should work with either python 2 or 3 (version 7 to version 8). The add-on will create an input data source for users, teams, oncall and incidents. The polling interval can be defined for each data source and data sources can be selected or deselected depending on the data desired.

For each type of data the script will check to see if the API response contains duplicate data, and if so, then the data is not indexed. For example, all users will be polled on the interval, however, if for some user A data looks the same, then it won’t be indexed; if the the user updated their paging policy then the data will be indexed. This is important because it will ensure that the VictorOps data is a very low amount.

There are not yet built in dashboards in the VictorOps app, but this creates a standard for the data format such that prebuilt dashboard are far easier to build and maintain.

Set Up Instructions

After downloading the add-on from the Splunk base here, it needs to be installed. Navigate to Apps > Manage Apps >> Install App From File and import the .tar.gz file downloaded previously.

The Splunk add-on for VictorOps should now be visible as an app in Splunk, navigate to the app. Under Inputs, select Create New Input and choose a type of data you would like Splunk to ingest from VictorOps. For all data types the input configuration options will look like below:

  • Name — this is a unique name for the data input. As a best practice, choose a name that accurately represents the input. For example, use something like vo_users_<org_name>.
  • Interval — this is the polling interval, in seconds, at which the VictorOps API will be polled. Keep in mind the time scale that is desired to see changes reflected in Splunk, the rate at which updates happen in VictorOps, and the resource consumption of running the polling scripts when selecting this number. For users, teams and on-call the polling interval will likely end up being much longer than the interval used for incidents.
  • Index — select any Splunk index where the data should be available
  • Organization ID — Note which VictorOps organization this data is coming from. This of even more importance if collecting data from multiple organizations in VIctorOps
  • API ID — This value can be found in VictorOps under Integrations >> API (admin or alert admin required).
  • API Key — THis value can be found in VictorOps under INtegrations >> API (admin or alert admin required).

Input Details

There are four types of inputs collected: users, oncall, teams (which includes routing keys) and incidents. Each input can be selected individually and independently of other inputs. In other words, users have the option to decide what exactly would be indexed per organization. Below are the inputs and their respective attributes in a sample JSON format.

Users (type=user)

  • Info
    • Names (first, last, username)
    • Created date
    • Date created
    • Date password updated
    • Verified
  • Contact Methods – Name, verification status (phone only) and value of all contact methods.
  • Paging Policy
  • Organization

On-Call (type=oncall, events are split per team)

  • Organization
  • Team name, slug
  • Escalation Policy
    • Oncall user(s) at time of index

Teams (type=team)

  • Info
    • Number of members, verified members
    • Team name, slug
  • Members
    • Username, first name, last name
    • Verified
  • Organization
  • Policies
    • Name, slug

Routing Keys (type=routingkey)

  • Default routing key status (true/false)
  • Organization
  • Name
  • Target escalation policies
    • Escalation policy name, slug
    • Team name, slug

Incidents (source=victorops_incidents)

  • Paged Users, Teams
  • State changes (ack, resolve)
  • All Metadata
  • Index timestamp is set to the startTime field
  • Alert Count

Troubleshooting

Things to verify, generally in order, if encountering problems

  1. Check that the API credentials are correct. Note, this is not the ‘Splunk API key’ this is the public API key and id found under Integrations >> API.
  2. Is the environment permitted to access the outside web? Ensure that from the host you can reach the VictorOps API. Try running ‘ping api.victorops.com’ to confirm connection.
  3. You can investigate further by inspecting the logs in $SPLUNK_HOME/var/log/splunk/ta_splunk_add_on_for_victorops_victorops_<INSERT_INPUT_TYPE_HERE>.log.
  4. If polling incidents in an organization with more than 60 incidents in the past seven days, the incident poll can take some time to run due to VictorOps API rate limits. If the input has been configured correctly and incident data is still not appearing, check the above log path for the incidents log (i.e. tail -f ta_splunk_add_on_for_victorops_victorops_incidents.log), if the last log entry is similar to “Waiting 59.985822999999996 seconds”, the script is waiting on rate limits to finish collecting and indexing the data. If this issue persists, consider reducing the polling interval.

Webhooks

Ingesting Data

VictorOps will send data to Splunk using an HTTP Endpoint Collector (HEC) depending upon your deployment a heavy forwarder may also be needed. To ensure communication from VictorOps to Splunk, VictorOps’ range of IP addresses should be whitelisted.

Creating the Webhooks

Four outgoing webhooks should be created, one for each event type. See below for each configuration. While the url will be the same for each webhook, keep in mind that the url will vary with different deployments of Splunk.

Splunk Version Url
On-Prem Instance https://<host>:8088/services/collector
Self-Service Splunk Cloud Instance https://input-<host>:8088/services/collector
All Other Splunk Cloud Instances https://http-inputs-<host>:8088/services/collector

The header will be the same for all webhooks and Splunk deployments. Be sure to replace <token> with the appropriate value for the HEC.

Key Value
Authorization Splunk <token>

The Content Type field should be set to application/json

The body of each webhook will vary according to the event-type. Be sure to replace your org slug (organization id found in the url of victorops, e.g. https://portal.victorops.com/dash/<org_slug>/outgoing-webhooks) in all instance of <org_slug>.


Event Type: Any Incidents

Body:

{
 "sourcetype": "_json",
 "event":
 {
 "slug": "<org_slug>",
 "link": "https://portal.victorops.com/client/<org_slug>/popoutIncident?incidentName=${{STATE.INCIDENT_NAME}}",
 "type": "incident",
 "alertService": "${{ALERT.service}}",
 "hostName": "${{ALERT.host_name}}",
 "service": "${{ALERT.service}}",
 "ENTITY_TYPE": "${{INCIDENT.ENTITY_TYPE}}",
 "SERVICESTATE": "${{ALERT.SERVICESTATE}}",
 "VO_ALERT_RCV_TIME": "${{ALERT.VO_ALERT_RCV_TIME}}",
 "alert_url": "${{ALERT.alert_url}}",
 "entity_display_name": "${{ALERT.entity_display_name}}",
 "entity_state": "${{ALERT.entity_state}}",
 "message_type": "${{ALERT.message_type}}",
 "monitor_name": "${{ALERT.monitor_name}}",
 "monitoring_tool": "${{ALERT.monitoring_tool}}",
 "routing_key": "${{ALERT.routing_key}}",
 "alert_timestamp": "${{ALERT.timestamp}}",
 "ACK_MSG": "${{STATE.ACK_MSG}}",
 "ACK_USER": "${{STATE.ACK_USER}}",
 "ACK_TIMESTAMP": "${{STATE.ACK_TIMESTAMP}}",
 "ALERT_COUNT": "${{STATE.ALERT_COUNT}}",
 "CURRENT_ALERT_PHASE": "${{STATE.CURRENT_ALERT_PHASE}}",
 "CURRENT_STATE": "${{STATE.CURRENT_STATE}}",
 "ENTITY_ID": "${{STATE.ENTITY_ID}}",
 "IncidentNum": "${{STATE.INCIDENT_NAME}}",
 "INCIDENT_TIMESTAMP": "${{STATE.INCIDENT_TIMESTAMP}}",
 "LAST_TIMESTAMP": "${{STATE.LAST_TIMESTAMP}}",
 "MONITOR_TYPE": "${{STATE.MONITOR_TYPE}}",
 "stateService": "${{STATE.SERVICE}}",
 "alert_uuid": "${{ALERT.VO_UUID}}"
 }
}

Event Type: Any-Paging

Body:

{
 "sourcetype": "_json",
 "event":{ 
 "slug":"<org_slug>",
 "type":"paging",
 "user": "${{PAGE.USER_ID}}",
 "started":"${{PAGE.STARTED}}",
 "page_id": "${{PAGE.ID}}",
 "attempt_num": "${{PAGE.ATTEMPT_NUMBER}}",
 "method_type": "${{PAGE.METHODS.0.TYPE}}",
 "method_label": "${{PAGE.METHODS.0.LABEL}}",
 "cancellation": "${{PAGE.CANCELLATION}}"
 }
}

Event-type: Any-On-Call

Body:

{
 "sourcetype": "_json",
 "event":{ 
 "slug":"<org_slug>",
 "type":"oncall",
 "user":"${{ONCALL.USER_ID}}",
 "state":"${{ONCALL.STATE}}",
 "team":"${{ONCALL.TEAM_NAME}}",
 "group":"${{ONCALL.GROUP_ID}}",
 }
}

Event-type: All-Chats

Body:

{
 "sourcetype": "_json",
 "event":{ 
 "slug":"<org_slug>",
 "type":"chat",
 "user": "${{CHAT.USER_ID}}",
 "text": "${{CHAT.TEXT}}",
 "is_robot": "${{CHAT.IS_ROBOT}}"
 }
}
Updated on April 22, 2020

Was this article helpful?

Related Articles