The VictorOps generic email endpoint is a basic email ingestion interface that allows you to send emails to a specially crafted VictorOps address in order to create, acknowledge, or resolve incidents in your timeline. Simply send an email from any monitoring tool or email service provider to your assigned email endpoint address with VictorOps to receive a result.
(Note: You must be able to modify the subject line of the email. If, for any reason, you are unable to customize the subject or body of the email, please contact email@example.com for assistance using our Transmogrifier feature for custom handling of email messages.)
Log into the VictorOps web portal.
Please select Settings >> Alert Behavior >> Integrations >> Legacy Email. If the integration has not already been enabled, click the green Enable button to generate your email endpoint address.
Enabling this integration will generate the generic email endpoint address (partially obscured below).
Email Endpoint Address
Your VictorOps email endpoint address consists of three parts:
The Email Endpoint Key (The long String of numbers, characters, and dashes).The Endpoint Key is unique to your organization in VictorOps, and, although you may revoke a key and generate a new one, only one endpoint key will be available to you at a time.
The Routing Key (+$routing_key) The Routing Key can be used to route an email endpoint initiated incident to a specific team or teams within VictorOps. For example, you’ve established routing key named database. The phrase “$routing_key” would be replaced with “database” to form the address as follows:
Please note, a routing key is not required and can be omitted from the email address altogether. Below please find the same address without a routing key (notice that there is no “+” symbol):
The last part of the email endpoint address is email domain: @alert.victorops.com.
For more information on setting up routing keys, see our Knowledge Base article on Routing Keys.
Formatting Emails and Handling Incidents
When using the email endpoint, the resulting behavior of the VictorOps platform will depend on the use of predefined keywords in the subject line of the email as follows:
- CRITICAL – Either of these keywords will open a new incident, thus triggering whatever escalation policy has been configured for the team receiving the incident.
- WARNING – This keyword will add an entry to the timeline, but not open a new incident.
- INFO – This keyword will post an informational event in the timeline, without creating an incident. (Nobody gets paged)
- ACKNOWLEDGEMENT – This keyword, though rarely used, will acknowledge an incident. The platform will stop paging users.
- RECOVERY – Either of these keywords will resolve an open incident. The platform will stop paging users. (It is not necessary for an incident to be acknowledged before it can be resolved)
When an email is ingested by VictorOps, the subject line is parsed and the above keywords are removed. Any remaining text in the subject line will become the title and main identity of the resulting incident (entity_id field). The body of the message will be included as text in the state_message field of the incident. Best practice is to include the keyword at the end to avoid issues with spaces in the title of the incident.
If an email does not contain any of these keywords, it is not parsable.
Example Incident using Email Endpoint
The following example email will result in the creation of a new incident that will be routed to the team E-team-5 (routing key = E-team-5)Below please find the resulting incident with expanded payload in VictorOps
The above incident may be acknowledged by sending the same email but replacing the keyword CRITICAL with the keyword ACKNOWLEDGEMENT. It can also be resolved by replacing the keyword CRITICAL with the keywords RESOLVED or OK.
Make sure that the subject line of the email is the same for all emails related to a given incident (excluding the keyword, obviously). In other words, if you were to send an email with the subject line “Database server DB6 is down CRITICAL” and then you tried sending an email to resolve the incident with the subject line “Database server DB6 is up RECOVERY”, the VictorOps platform would not recognize that the second email is related to the incident opened by the first (because the entity_id for the first alert contains the word down, while the entity_id for the resolution message is different, containing the word up instead.)
Legacy Email Systems
Some legacy monitoring tools (and some not so “legacy”) do not permit users to alter the content of the subject line of their email notifications. In this case, it may be possible to use our Transmogrifier tool (Enterprise only) to control the workflow of incidents generated by that tool. Contact our support team at firstname.lastname@example.org for help with this configuration.
Recommended Transmogrifier Rules
Sending alerts through the email integration can be limited depending on the flexibility you have over the email Subject. If you have limited to no flexibility to modify the Subject line, and do not want to create Critical alerts for all emails, please see an example Transmogrifier rule that could be used, below.
With this Transmogrifier rule, we are looking for a keyword or phrase in the Email Subject line using wildcard matching, denoted by the asterisks. If the keyword or phrase is present in the Subject line, then change the message_type to critical (this can be replaced with any of the parsable fields listed in the “Formatting Emails and Handling Incidents” section).