What is VictorOps
VictorOps is incident management software that allows teams to maintain a culture of high availability without slowing down the innovation process.
Here, we’ll provide the steps you need to take to get your instance ready. If at any point you find yourself needing additional help, please contact support (email@example.com). Our 24/7 support team is on-call when you are.
Let’s get started!
Add a Few Users
The most important first step of setting up VictorOps is adding users.
To add new users:
- Leverage their email address in the portal (Settings -> Users -> Add User)
- Utilize our API (your API ID and Key can be found here)
*Uploading a lot of users? Reach out to us, and we’ll help you out!
Once you have a few users in the system, you can create teams! Teams hold user lists, on-call shifts, and escalation policies.
To create a team, head to the “Settings” section from the top nav bar. Click the “Add Team” button. Then choose a name.
We recommend standardizing your team names to clearly delineate across teams. You can choose team names based on service, internal team name, etc.—whatever makes sense to your organization, just aim for consistency.
Invite Users and Declare Admins
Once you’ve built a few teams, the next step is to add people (duh…). You can add invited users. Then, establish a hierarchy of users based on user roles, e.g, Admins, Users, and Team Admins.
You can learn more about user types and permission structure here.
Rotations are your recurring on-call schedules—basically groups of on-call shifts. A shift is shared across a number of people.
You find a step-by-step walkthrough of how to set up a rotation here. You can also reach out to us for help setting up your custom rotation.
Note: A scheduled rotation doesn’t automatically mean you’re on-call; rotations need to be tied to an escalation policy.
Create Escalation Policies
Escalation policies determine which incidents are routed, to whom they are routed, and how they are escalated. Essentially, an escalation policy is how VictorOps escalates a triggered event.
Best practice for setting up your escalation policy is to establish a minimum of three escalation paths: on-duty user, previous/next user in a rotation, and manager/team lead.
Here are the basics:
When an incident is routed to the following Escalation Team, its escalation policy will:
- Immediately notify the on-call user in the Dev_Rotation. It will also send an email to this team’s email address.
- The system will wait 10 minutes, and if the incident has not been acknowledged, @ngrieb’s personal notification policy will be executed
- If the incident is still un-acked after an additional 30 minutes, the Database team’s escalation policy will be executed.
- Finally, if the incident is still un-acked after an additional 60 minutes (cumulative time of 10 + 30 + 60 = 1 hour 40 minutes after initial trigger), a webhook will be executed to reboot the server.
Need more help? Reach out to us and we can help configure your on-call rotation!
Configure Routing Keys
Routing keys tie the alerts from your monitoring tools to the specific team (or escalation policy) in VictorOps—this helps get the right person on the problem and reduce alert noise for those unrelated to a specific incident.
You can navigate to Routing Keys via the “Settings” section of your nav bar. From settings go to “Alert Behavior” and then “Route Keys”
A quick note on naming conventions for your routing keys: Keep them simple! Use the name of the team/policy that is handling the alerts, the service/host for the alert, monitoring tool the alert is coming from:
- Matching team name: CloudOps (team) = cloudops (routing key)
- Matching monitoring tool: Splunk (tool) = splunk (routing key)
- Need something more specific? Do a combo of the two! CloudOps (team) Splunk (tool) = splunk_cloudops
And, although route keys are case sensitive, we recommend using all lowercase letters to prevent alerts from going to the default routing team.
Integrate All the Things!
The final piece is to set up your custom integrations. We integrate with a large number of tools. For a full list of integrations—plus more information on how to set them up—check here. Can’t find what you’re looking for? Check out our generic email or REST endpoints!