NOTE: If you are new to Splunk On-Call a great place to start is with our User Training, found here. Once you a familiar with your User permissions, come back to this article to learn about your increased responsibilities as an Alert Admin.
Your Role as an Alert Admin
As an alert admin, you are responsible for managing alert configuration, integrations, and their workflow.
Your Permissions as an Alert Admin
As an Alert Admin, your permissions are organization-wide and come with a lot of responsibility. Proper management and upkeep of integrations are essential to your alert workflow. Alert Admins have permission to take the following actions:
| Permissions specific to an Alert Admin |
| Integration Configuration |
| Management of Routing Keys & Rules |
| Creation & Upkeep of Webhooks |
| Maintenance Mode |
View all User Roles and Permissions!
Your Resources as an Alert Admin
Knowledge Base: Splunk On-Call has an extensive Knowledge Base that is always a good place to start if you are unsure how something works or are in need of some tips! There is even an entire Integrations section for you to check out.
Contact Us: All users have the ability to reach out to Splunk On-Call support at any time with any questions!
1. Live Chat: If you are logged into your Splunk On-Call instance, you will have the ability to Live Chat with the Splunk On-Call Support team.
2. Splunk Support Portal: You can open a Splunk On-Call support case in the Splunk Support Portal:https://login.splunk.com/
If you are facing any issues when trying to contact us please have a look HERE!
Recommendations to be a Successful Alert Admin
- Create Routing Keys: Routing Keys are responsible for directing the alerts to the correct escalation policy in order to page the correct on-call user. Think of these as the “postage” of each alert.
♦ Best Practice Tip ♦ Reach out to Team Admins to assist with naming conventions for Routing Keys in order to ensure that escalation policies and routing key names are in sync and simple to identify.
- Enable & Configure Integrations using the Knowledge Base Guides: Search for the integrations you need on the integrations page. If you do not see an integration listed, you can always use the Generic Rest Endpoint or Email integration based on the capabilities of your tools. Use the Knowledge Base Integration Guides to configure your integrations.
♦ Best Practice Tip ♦ Make sure you are only sending critical, actionable alerts to Splunk On-Call to avoid alert fatigue and confusion.
- Confirm alerts are directed to the corresponding teams: After configuring your integrations, make sure that incidents are routing and behaving properly by sending test alerts.
- Create Rules Engine Rules: You can modify fields, add annotations, and redirect alerts based on certain matching conditions. The Rules Engine even has regex capabilities to parse out portions of fields or create time-based rules. Quick video on the Alert Rule Engine!
- Configure Custom Outgoing Webhooks: Webhooks allow you to pass information outside of Splunk On-Call based on actions taken within Splunk On-Call such as a triggered incident or a chat. When combined with the Rules Engine, they can be configured to conditionally fire. Quick video on Custom Outgoing Webhooks!
- Maintenance mode: If you need to perform maintenance for one of your integrations, you can turn on maintenance mode for a specific routing key or all routing keys. Maintenance mode will mute paging for the given period of time, and resume paging once ended. NOTE: Maintenance Mode does not stop the alerts from coming into Splunk On-Call, just from paging the on-call users when they do come in.
Alert Admin Checklist:
