A virtually limitless number of scheduling options can be configured in VictorOps. This article will walk through how to configure some of the most popular ones.
Note: Only Global and Team Admins in VictorOps have the ability to create and edit schedules.
Scenario #1
Multiple people On-Call during Business Hours, One person takes turns doing Nights/Weekends, One person takes turns doing 24×7 Secondary coverage
Large Points:
- Create a Rotation called “Business Hours” with one Partial Day type shift for each user you would like on-call during Business Hours
- Create a Rotation called “Nights and Weekends” with one Partial Day type shift for the weeknights portion, and one Multi-day type shift for the Weekend
- Create a Rotation called “Secondary” with a 24/7 type shift
- Make 3 Escalation Policies that use these rotations with “Notify the on duty users in rotation” actions in Step 1. Set Step 2 of the Nights and Weekends and Business Hours escalation policies to execute the Secondary escalation policy
Process:
Business Hours rotation
- To start, navigate to the team you’d like to implement this schedule on and add all of the appropriate users using the ‘Invite User’ button
- Next, go to the Rotations tab and click “Add Rotation”
- Name the Rotation “Business Hours” and select “Partial Day” for the shift type
- Name the first shift either after the user who will be occupying it (i.e. Mike’s Shift) or just call it “Shift 1” and set hours
- Click “Save Shift”
- Note: We’re able to ignore the “Handoff happens every X week” and “The next handoff happens X” fields because this shift is being created with the intent to not handoff
- Next, add the appropriate user to this shift by clicking the Manage Members button and then “Select a User to add…”
- After adding the appropriate user, click “Add Another Shift”
- Again, select “Partial Day” and repeat the process from the previous steps. You will end up creating one shift for each user you would like on-call simultaneously during the business day (in my example 4 users will be on-call simultaneously)
- When finished, click “I’m Done, Save this Rotation”
Nights and Weekends Rotation
- We’ll now create the “Nights and Weekends” rotation by clicking “Add a Rotation”
- Call it “Nights and Weekends” and select “Partial Day” for the first shift type
- Call the first shift “Weeknights”
- Click on where it says Monday through Friday in blue lettering and un-check Friday. Also, change the hours to cover all week-hours that are not covered by the Business Hours rotation (i.e. if your business hours are 8 am – 5 pm, your Weeknights shift should be 5 pm – 8 am. If your Business Hours are 9 am – 4 pm, your Weeknights shift should be 4 pm – 9 am)
- Change the “The next handoff happens” date to the next upcoming Monday
- Finally, click “Save Shift”
- Next, add the appropriate users to this shift by clicking the Manage Members button and then “Select a User to add…”
- The order that you add the users in will dictate the week they end up on-duty (the first user you add will do the current week, the next user the week after, the third user the third week…). You can re-order them after adding by dragging them around on the “Members” menu
- After adding the appropriate users, click “Add Another Shift” and select “Multi-day” for the type
- Call this shift “Weekends” and adjust the hours to those desired
- Click “Save Shift” when done
- Again, click the Manage Members button and add in your users, recognizing that the order you add them will dictate the week they are on-duty. You can re-order these users after adding by dragging them around within this menu.
- Finally, click “I’m Done, Save this Rotation”
Secondary Rotation
- The next part of the configuration process will involve creating a “Secondary” rotation to serve as a backup in case either the Business Hours or Nights and Weekends user(s) doesn’t respond in a set amount of time
- To start, click “Add a Rotation”
- Call the rotation “Secondary” and select 24/7 for the shift type
- Call the shift “Secondary Shift”, change the handoff time to the appropriate one (typically when business hours begin for your organization), and change the “The next handoff happens” to the next upcoming Monda
- When done, click “Save Shift”
- Next, add your users to this shift by clicking the Manage Members button and then “Select a user to add”, noting that the order you add them in will dictate the order they are on-duty (i.e. the first user you add will do the current week, next user you add will do the next week, the third user you add will do the third week…)
- When finished, click “I’m Done, Save this Rotation”
Escalation Policies
- Next, navigate to the Escalation Policies tab and click “Add Escalation Policy”
- Name this one “Secondary” and set the first step to Immediately “Notify the on-duty users in rotation: Secondary”
- After clicking “Save”, refresh your page. This will allow the just-created escalation policy to be available for selection in other escalation policies
- Again, click “Add Escalation Policy” and this time, call the Policy Name “Business Hours”
- Set the first step to Immediately “Notify the on-duty users in rotation: Business Hours”
- Click “Add Step” and set Step 2 to “If still unacked after 15 minutes”, “Execute Policy: <Team Name>: Secondary”
- This will link the Business Hours and Secondary escalation policies together, so to speak, such that if the Business Hours users don’t respond after 15 minutes, the Secondary user will be paged
- After clicking “Save”, again click “Add Escalation Policy” and call this one “Nights and Weekends”
- The format of this escalation policy will be extremely similar to the Business Hours escalation policy, just with the Nights and Weekends rotation specified in Step 1 instead of the Business Hours rotation
- Similarly, this links the Nights and Weekends and Secondary escalation policies together so that if the Nights and Weekends on-call user doesn’t respond in 15 minutes, the Secondary user will be escalated to
- Next, click “Save”
Routing Keys
The final part of this configuration will involve assigning routing keys to the Business Hours and Nights and Weekends escalation policies
- To navigate here, click Settings >> Routing Keys
- From this screen, you can assign an existing routing key to the escalation policies, assign them to the Default Routing Policy, and/or create a new routing key and assign the escalation policies to it.
- To assign the escalation policies to an existing routing key, hover your mouse over the key until a pencil icon shows on the far right, and then click the pencil icon
- After clicking the pencil icon, locate the Business Hours and Nights and Weekends escalation policies (categorized under the team you’ve created them on) and select them
- When finished, click the blue checkbox to save your changes
- To assign the escalation policies to the Default Routing Policy, you’ll very similarly hover your mouse over the row until a pencil icon shows, and then select the escalation policies
Scenario #2
Only get alerted for certain alerts during business hours
Large Points:
- Create a Rotation called “Business Hours” with one Partial Day type shift for each user you would like on-call during Business Hours
- Specify this Rotation in the first step of an Escalation Policy
- Create a Routing Key called “bus-hours-only” and direct it at that Escalation Policy
- (Optional) Create Rules Engine rules to have this Routing Key applied to the appropriate alerts
Process:
Business Hours rotation
- To start, navigate to the team you’d like to implement this schedule on and add all of the appropriate users using the ‘Add User’ button.
- Next, go to the Rotations tab and click “Add Rotation
- Name the Rotation “Business Hours” and select “Partial Day” for the shift type
- Name the first shift either after the user who will be occupying it (i.e. Mike’s Shift) or just call it “Shift 1”
- Set the hours to the appropriate ones
- Click “Save Shift
- Note: We’re able to ignore the “Handoff happens every X week” and “The next handoff happens X” fields because this shift is being created with the intent to not handoff
- Next, add the appropriate user to this shift by clicking the Manage Members button and then “Select a User to add…
- After adding the appropriate user, click “Add Another Shift”
- Again, select “Partial Day” and repeat the process from the previous steps. You will end up creating one shift for each user you would like on-call simultaneously during the business day (in my example 4 users will be on-call simultaneously
- When finished, click “I’m Done, Save this Rotation”
Escalation Policies
- Next, navigate to the Escalation Policies tab and click “Add Escalation Policy”
- Name this one “Business Hours Only” and set the first step to Immediately “Notify the on duty users in rotation: Business Hours
- Click “Save”
Routing Keys
The next part of this configuration will involve assigning a routing key to the Business Hours Only escalation Policy
- To navigate here, click Settings >> Alert Rules Engine
- Click “Add Key” and name the routing key “bus-hours-only”
- In the dropdown menu locate the Business Hours Only escalation policy (categorized under the team you’ve created it on) and select it
- Click the blue Check box to Save
- Once you’ve created this routing key, you’ll likely want to go into your monitoring tools and assign this routing key to the alerts you only want to page during business hours (or optionally follow the rules engine steps below). You should be able to locate instructions on where to edit the routing key in your specific monitoring tool(s) by finding that tool’s integration guide in this Knowledge Base repository, but if you have any questions, feel free to shoot an email to victorops-support@splunk.com
