While “multiple escalation policies” may not sound sexy, this feature is essential to streamlining your incident management process as your organization grows and scales. Think of it as the ability to have multiple (or unique) alert escalation behaviors using the same users and rotations within a single team.
Here are a few tips and tricks to help you optimize your account and workflow.
Tip 1: Reduce the Number of Teams in Your Org by at Least 50%
On average, VictorOps customers create three teams, all comprised of the same humans, to handle alert behavior for the same group at their company. Transform your unwieldy list of teams into an organized directory by converting a set of related teams into one team with nested escalation policies.
In the example below, we identified three teams related to IT Ops and collapsed them into a single team, IT Operations, creating three policies within that team. This higher level of organization helps you find who is on-call faster and reduces scheduling headaches.
Tip 2: Create ‘Waiting Rooms’ to Eliminate Middle-Of-The-Night Alerts from Incidents Known to Auto-Resolve
Many of our customers have alerts that auto-resolve within a few minutes, so why wake up at 2 a.m.? In this case, if an incident has not resolved after a period of time, only then should VictorOps page you.
Here’s a trick to fix this:
- Create a second escalation policy within your team called Waiting Room. Create a routing key (used to send alerts to the Waiting Room) and append it to the policy.
- Next, designate the amount of time you feel comfortable with the alert existing prior to paging a human—set this amount of time as the first step in the Waiting Room policy. We picked 15 minutes as shown below.
- Lastly, once that amount of time has passed, reroute the alert to the Primary Policy. This is achieved by selecting Execute Policy as the action and designating the Primary Policy for the reroute.
Alerts that are routed to the Waiting Room policy will delay paging for 15 minutes, allowing ample time for them to auto-resolve before waking you up.
Tip 3: Surface Secondary On-Call Schedules in Your On-Call Calendar
VictorOps makes it very easy to designate previous or next user in a rotation as a backup, but the second step of an escalation policy will not appear on your on-call calendar nor can overrides be scheduled for these shifts. If your company treats secondary on-call as ‘on-call,’ here’s a tip to get VictorOps to surface secondary in the on-call calendar and enable scheduled overrides for these responsibilities.
- Create a secondary escalation policy in your team for users who will be backup (i.e., on-call for the second step). This policy will have one step immediately notifying the rotation you have designated for backup users.
- In the second step of your Primary Policy, select the Execute Policy action and reference the Secondary Policy.
Tip 4: Schedule an Override for a Single Escalation Policy
Requesting coverage has never been easier. Scheduled Overrides are on an Escalation Policy basis, and you may now request coverage for a single policy. When you schedule an override, VictorOps will centralize and surface all the policies that would result in a page to you directly—no hunting through your teams to ensure coverage. If you don’t require coverage for a particular policy, simply don’t assign anyone to cover that specific policy, and VictorOps will continue to alert you.