1. Home
  2. Integrations
  3. Splunk Integration Guide – VictorOps

Splunk Integration Guide – VictorOps

This is the latest documentation for the current Splunk integration with VictorOps. The VictorOps and Splunk integration allows teams to schedule queries or alerts in Splunk to monitor system health. The VictorOps integration with Splunk can be leveraged to collect data about the overall release toolchain and deployment success to allow teams to collaborate around that information in the timeline.

Here you’ll find step by step guides for integrating VictorOps with Splunk Cloud, Splunk Enterprise, Splunk ITSI and Splunk SII products.

Helpful Links:

Download the Splunk App for VictorOps

Download the VictorOps App for Android

Download the VictorOps App for iOS

About Splunk and VictorOps

Splunk Intelligence with VictorOps Alerting

While the out-of-the-box notification methods in Splunk are great for reports and non-critical information, there’s next-level functionality gained when teams integrate Splunk with the VictorOps platform.

Advance Beyond Basic Alerting

The VictorOps and Splunk Integration Promotes End-to-End, Data-Based Incident Management

  • Deliver contextual alerts to the right person at the right time (via our scheduling and routing)
  • Leverage escalation workflows in VictorOps and collaborate in the timeline to see actionable incidents detected by Splunk
  • Establish custom routing workflows based on data generated in Splunk and utilize ChatOps capabilities to improve collaboration
    • For example, queries or searches against database logs include the word “database.” The VictorOps transmogrifier can then selectively route these specific alerts to a particular database team

Integrating Splunk Enterprise and Splunk Cloud

Versions Supported

Splunk Enterprise (on-prem)
Splunk Cloud 7.0, 7.1, 7.2

  • VictorOps Subscription Required: Getting Started, Essentials, or Full-Stack
  • Installation: location single instance Splunk or Splunk search head
  • If you would like to proceed with installation with SHC (search head clustering) or with a proxy setup please contact support for specific instructions.
  • Additionally, on-prem customers will need to open port 443 for outgoing https communication with VictorOps. The full url used will be https://alert.victorops.com/integrations/generic/20131114/alert/$your_api_key/$your_routing_key.

Splunk transforms machine-generated data into valuable insights that can help make your business more productive, profitable and secure.  The following guide will walk you through integrating VictorOps with action alerts from searches in Splunk Enterprise and cloud.

You will need an active VictorOps instance before you begin. Click here to start a free 14-day trial.

In VictorOps (SCP and Enterprise)

From the VictorOps web portal, select Settings, then Alert Behavior, and Integrations. Click the Splunk, Inc integration option.

selecting splunk victorops integration

Copy the API key from the “Service API Endpoint” field to the clipboard (This is the section after “…/alert/” and before “/$routing_key”).

splunk victorops service API endpoint

In Splunk

From the Splunk Base, search for VictorOps, or follow this link. Click the Download button and accept the license agreements by checking the boxes and clicking Agree to Download.

VictorOps in Splunk base

Start Splunk and open the web UI in a browser. From the top navigation bar, expand the drop down menu and select Manage Apps. Next, click the button Install app from file.

installing splunk app in victorops

Choose the VictorOps for Splunk app .tgz file downloaded earlier, check Upgrade app box to ensure your application is updated to the latest version. Click Upload then finish the process by restarting Splunk.

upload victorops for splunk file

Once Splunk has restarted, return to the Manage Apps page and click Set Up next to the VictorOps app. On the set up page, paste the api key copied earlier, along with any routing key from your account.

splunk victorops API and routing key prompt

VictorOps can now be used as an Alert Action.

Verify the Integration

From the Search app in Splunk, you can directly type

| sendalert victorops param.message_type="INFO"

to send a test alert directly to your VictorOps timeline. To create an incident, simply change INFO to CRITICAL. If you have any questions, please contact VictorOps support.

Sample Alert

Here is an example of setting up a new alert based on a search. From a new search select Save As, then select Alert.

victorops splunk sample alert

Give the alert a title, description and permissions as well configure the check schedule. Under + Add Actions, select VictorOps.

victorops splunk save alert

Select the desired message type, and use the state message field to add a brief description of what this particular alert indicates. You may overwrite the default values for the routing key and entity_id if desired, however, you should understand how best to use these fields. Additionally, you can dynamically reference Splunk fields within these assignments using tokens.

reference Splunk fields within these assignments using tokens

Once the specified conditions are met, you should see an alert appear in your VictorOps timeline.

splunk alert in victorops timeline

Troubleshooting

Note that if you receive an error while updating you API key “Error while posting to url=/servicesNS/nobody/victorops_apps/storage/passwords/” please follow the steps below to update your API key.

  1. Move the $SPLUNK_HOME/etc/apps/victorops_app/local/passwords.conf file to passwords.bak
  2. Hit the /debug/refresh endpoint on the SH (http://host:8000/en-US/debug/refresh)
  3. Select “Refresh”
  4. On the “Entity refresh control page” look for: “Refreshing admin/passwords… OK”
  5. Go back to the Splunk UI (back button works fine) and select “Manage Apps”
  6. Find the victorops_app in the list a choose “Set up”
  7. Place in the API Key and Routing Key (if needed) and Save
  8. Verify there’s a new passwords.conf in $SPLUNK_HOME/etc/apps/victorops_app/local/

You can compare hash values in the passwords.conf and .bak to determine if there’s a changed API key.

 

Integrating Splunk ITSI

This is documentation for integrating Splunk ITSI (IT Service Intelligence) with your VictorOps account. Splunk ITSI is a powerful AI-powered monitoring and analytics solution that allows for incident prediction and prevention.

Requirements

Versions Supported: Splunk ITSI 4.0 or newer

VictorOps Subscription Required: Full-Stack

VictorOps for Splunk: available in the splunkbase HERE.

Local Machine: Java 8 or newer

For more information check out the splunkbase documentation

In VictorOps

From the VictorOps web portal, select Settings, then Alert Behavior, then Integrations. Find and select the Splunk ITSI integration option.

splunk itsi integration in victorops

Copy the API key from the “Service API Endpoint” field to the clipboard (This is the section after “…/alert/” and before “/$routing_key”).

splunk itsi victorops service API endpoint key

Next, create a transmogrifier rule to link the ITSI Notable Events View to the VictorOps incident. Navigate to Settings, then Alert Behavior, and Transmogrifier. Create the following rule:
splunk itsi victorops transmogrifier rule

In Splunk ITSI

Navigate to Configure >> Notable Events Aggregation Policies >> then click the name of the Aggregation Policy you want to alert VictorOps.

spunk itsi victorops notable event aggregation policies

In the Action Rules tab, set your trigger conditions then choose VictorOps and configure your alert accordingly.

splunk itsi victorops Action Rules tab

Keep the Alert Entity ID consistent for all Message Types across related actions. VictorOps uses this field to identify incidents and correlate subsequent alerts with the original incident.

Once configured correctly, ITSI will pipe directly into your VictorOps Timeline.

splunk itsi in victorops timeline

To Create a VictorOps Incident

Navigate to the Action Rules tab for the desired Aggregation Policy. For an action to create an incident in VictorOps, set the conditions to if the following event occurs: severity greater than Normal then select VictorOps and click Configure.

splunk itsi victorops Navigate to the Action Rules tab

Configure the action using the following values:

  • Message Type : CRITICAL
  • Monitoring Tool: splunk-itsi
  • Alert Entity ID: $result.itsi_group_id$
  • Alert Entity Display Name: $result.itsi_group_title$
  • State Message: $result.itsi_group_title$
  • Routing Key: the VictorOps routing key you want to use

spunk itsi victorops rules configuration values

 

To Resolve a VictorOps Incident

Within the same Aggregation Policy, navigate to the Action Rules tab. To resolve the episode in ITSI, select change status to Resolved. To resolve the corresponding incident in VictorOps, set the conditions to if the episode is broken, then VictorOps and click Configure.

splunk itsi resolve victorops incident

When configuring the Action, check that the Alert Entity ID is the same as the initial alert so that VictorOps resolves the corresponding incident. Configure the action using the following values:

  • Message Type : RECOVERY
  • Monitoring Tool: splunk-itsi
  • Alert Entity ID: $result.itsi_group_id$
  • Alert Entity Display Name: $result.itsi_group_title$
  • State Message: $result.itsi_group_title$
  • Routing Key: the VictorOps routing key you want to use

splunk itsi victorops configure action rules

To Acknowledge a VictorOps Incident Manually

Navigate to Episode Review then click the desired episode, Actions, and select VictorOps.

splunk itsi acknowledge victorops incident manually

When configuring the Action, check that the Alert Entity ID is the same as the initial alert so that VictorOps resolves the corresponding incident. Configure the action using the following values:

  • Message Type : ACKNOWLEDGEMENT
  • Monitoring Tool: splunk-itsi
  • Alert Entity ID: $result.itsi_group_id$
  • Alert Entity Display Name: $result.itsi_group_title$
  • State Message: $result.itsi_group_title$
  • Routing Key: the VictorOps routing key you want to use

splunk itsi vicotrops configure action manually

Integrating Splunk SII

This documentation will integrate Splunk SII (Insights for Infrastructure) with your VictorOps account. Splunk SII is an infrastructure monitoring product that bridges monitoring and troubleshooting by collecting and analyzing both metrics and logs.

In VictorOps

From the VictorOps web portal, select Settings, then Alert Behavior, then Integrations. Select the Splunk, Inc integration option.

splunk sii configuration in victorops

Copy the API key from the “Service API Endpoint” field to the clipboard (This is the section after “…/alert/” and before “/$routing_key”).

splunk sii victorops API key example

In Splunk SII

Navigate to Settings >> Notifications and paste your API key and and a routing key from your VictorOps account into the respective fields. Click Save Credentials.

setting up victorops in splunk sii

Now the VictorOps notification can be attached to alert. Under the Investigate page, select an entity.

select an entity splunk sii victorops

Navigate to the Analysis tab and choose an alert graph, click the three dots and select Create Alert.

victorops splunk sii create alert

From the alert creation, scroll to the bottom of the modal and choose how under what conditions the alert should fire and then for the notification method select VictorOps. Click Submit.

splunk sii victorops submit alert

Updated on April 25, 2019

Was this article helpful?

Related Articles