Incidents in your VictorOps timeline function like a simple table with two columns: the field name, and the value of that field. Field names are either predefined automatically by VictorOps, defined by the integrated monitoring tool or created by a Rules Engine rule. This makes an exhaustive list of all potential fields nearly impossible. However, certain fields are always present, and this article defines and explains those fields, how their values affect the behavior of an incident, and how the Rules Engine can be used to manipulate those fields.
Anatomy of an Incident:
When viewing an incident in the timeline, it appears as an abbreviated version, displaying only a few fields that summarize the event (It is not possible to configure which fields are displayed here):
By clicking on the “Alert Payload” button, you can view the full payload (all fields) of the incident:
Note that any field that has been impacted by a Rules Engine rule will show the Transformation symbol:
Required / Important Fields:
The message_type field is the one required field in VictorOps (all other fields would be filled in automatically). message_type is used to determine the behavior of the alert when it arrives.
- CRITICAL – Opens a new incident, which will set off an escalation policy and result in users being paged.
- WARNING – May open a new incident depending on configuration in Settings>>Alert Behavior>>Configure Incidents>>”Create incidents for entities in [xxxxxxx] state”. Otherwise, it will post information to the timeline without creating an incident or triggering any escalation policies.
- INFO – Displays an entry in the timeline, without opening an incident (cannot trigger escalation or paging)
- ACKNOWLEDGEMENT – Moves the incident from a triggered to acknowledged state (stops the escalation and paging)
- RECOVERY /or/ OK – Resolves the incident (also stops escalation and paging if still active)
This field serves as the central identity of an incident. It is used to recognize related events and must remain consistent throughout the life-cycle of the incident. This field is how the VictorOps platform knows that a particular recovery message applies to a particular open incident. When an incident is unresolved (in a triggered or acknowledged state), and another critical message arrives with the same entity_id, the new message will be rolled up into the existing incident without creating a new incident. This works great for preventing duplicate notifications for the same problem, but users must also be careful not to leave an incident unresolved for too long or they risk missing a separate occurrence of the same problem. If not provided, this field will be auto-filled with a random String value.
User / Monitor Defined Fields:
This field controls the routing of incidents to specific teams. Routing keys can be created and assigned to a team, or teams, at the bottom of the Settings>>Integrations page. An incident can only have one single routing_key associated with it.
Often times, the entity_id of an incident can be long and full of jargon. Setting the entity_display_name will change how the incident appears in the timeline because it is the serves as the title of the incident. This field is also read aloud during phone call notifications, which gives users an opportunity to simplify and customize the message without affecting the life-cycle of the incident.
The state_message field is meant to contain a more verbose description of the problem. It can also contain URL links. When using an email endpoint integration, the body of the email will become the state_message field.
If there is a hostname field with a value in the payload, we will display it after the entity_display_name in the incident card:
Users can add as many custom fields (with custom names) to an incident as they wish. This can be done by manually adding the fields to the HTTP POST request, or by using the Rules Engine to create a new field.
Glossary of Fields:
|Field Name||Possible Values||Purpose/Behavior||Common Transmogrifier
|ack_author||Username||Displays the user who has acknowledged this incident.|
Remains blank if incident is unacknowledged.
|Not for use
|ack_message||Acknowledgement method||Displays the method used to acknowledge or is left blank||Not for use
|agent||Any||Field for specific legacy integrations.||Not for use
|alert_type||Any||Field for specific legacy integrations.||Not for use
|api_key||Long String value||Displays the REST Endpoint key your organization uses to reach VictorOps (each org only has 1)||Should not be altered
with the Transmog,
but can be used for a
rule that matches all integrations using
the REST endpoint.
|entity_display_name||Any||More succinct, intuitive name for incident that does not affect the entity_id.|
Defaults to entity_id if not explicitly defined.
*This field is read aloud during phone call notifications.
* This field is displayed in email, SMS, and push notifications (Push and SMS truncated for length)
|Can be changed to
make the name of
the incident more succinct
and intuitive without
affecting the behavior
of the incident.
|entity_id||Any||Central identifier for incident.||Can be altered to
combine or separate
|entity_is_host||Boolean||Indicates whether the entity reporting the issue is also the host||Not for use
|entity_state||Same as message_type||Current state of monitored entity (May be different from message_type with certain integrations)||Not for use
|eventType||Any||Field for specific legacy integrations.||Not for use
|host_name||Any||Displays the affected host||Match on this field to control incidents related to a specific host (Change the routing_key to the team responsible for this host or quiet alerts matching this host by transforming the message_type field to "INFO" etc.)|
|message_type||CRITICAL||Opens a new incident||Change field to this value to always open an incident (Very useful with legacy email integrations)|
|^||WARNING||May open a new incident depending on configuration (Settings>>Integrations)||Behavior controlled by
options chosen in
>>Create incidents for
entities in [ ] state
|^||ACKNOWLEDGEMENT||Moves incident from Triggered to Acknowledged (stops escalation and paging)||Change field to this
value prevent paging,
send incident straight
to acknowledged state.
|^||INFO||Posts info to timeline without creating a new incident.||Change field to this
value to quiet a noisy
alert (prevent it from
opening a new incident
|^||RECOVERY / OK||Resolves incident (stops escalation and paging)||Change field to this
value to resolve an
incident. (Very useful
with legacy email
|monitor_name||Any||Name of specific monitor (if multiple) or message sender (email)||Match on this field
to control alerts
from a specific monitor
|monitoring_tool||Any||Displays the monitoring tool that triggered the incident.||Match on this field
to control all alerts
from a specific
|NOTIFICATIONTYPE||String||Legacy field created for Nagios integrations||Not for use
|routing_key||Any (defined by user)||Used to direct incidents to a specific team.||Use a transformation
to alter the routing key
and send the incident
to a different team.
|sender||Any||Field for specific legacy integrations.||Not for use
|SERVICESTATE||Any||Field for specific legacy integrations.||Not for use
|state_message||Any||Large field used for passing verbose information about the incident.|
*This field is displayed in email notifications (full) and SMS notificatoions (truncated for length)
|Pull values from
other fields to add
more useful information
to the message users
receive when they are
notified of a new
|state_start_time||Date / Time||Indicates the date and time that the problem began on the monitored host/service||Not for use with Transmogrifier|
|subject||Any||Field for specific legacy integrations.||Not for use
|timestamp||Date / Time||When monitoring tool detected an anomoly on monitored host / service (sent by monitoring tool, or defaults to VO_ALERT_RCV_TIME if not defined)||Not for use with
*Actual data is in Unix time
format and cannot be
used for time based rules
|VO_ALERT_RCV_TIME||Date and time||When message was received by VictorOps endpoint.||Not for use
|VO_ALERT_TYPE ||String||Index of alert types for internal use only.||Not for use
|VO_MONITOR_TYPE||Integer||Index of monitor types for internal use only.||Not for use
|VO_ORGANIZATION_ID||org slug||Slugified version of your organization's name used internally to identify your account.||Not for use
|VO_UUID||Random String||Used internally by VictorOps for logging||Not for use