The VictorOps generic email endpoint is a basic email ingestion interface that allows you to send emails to a specially crafted VictorOps address in order to create, acknowledge, or resolve incidents in your time-line. Simply send an email, from any monitoring tool or email service provider, to your assigned email endpoint address with VictorOps.
(Note: You must be able to modify the subject line of the email. If, for any reason, you are unable to customize the subject or body of the email, please contact email@example.com for assistance using our Transmogrifier feature for custom handling of email messages.)
Log into the VictorOps web portal.
Click on Settings >> Alert Behavior >> Integrations >> Legacy Email. If the integration has not already been enabled, click the green Enable button to generate you email endpoint address.
This will generate the generic email endpoint address to which you will send emails (partially obscured in photo):
Your Email Endpoint Address:
Your VictorOps email endpoint address consists of 3 parts:
- The email endpoint key (The long String of numbers, characters, and dashes)
- The routing key preceded by a “+” sign – ($routing_key) (If you are unable to use a “+” sign with your Gmail configuration, simply replace the “+” sign with a period.)
- The email domain – (@alert.victorops.com)
The endpoint key is unique to your organization in VictorOps, and though you can revoke a key and generate a new one, you will only have one single endpoint key. The routing key can be used to route the resulting incident to a specific team or teams in VictorOps. (for more information on setting up routing keys, see our knowledge-base article on routing) A routing key is not required, and can be omitted from the email address. However, if you are including a routing key, note that the “+” symbol must be present between the endpoint key and routing key. The rest of the text before the @ symbol must be replaced with the actual routing key.
For example, let’s say you have an established routing key of database. The phrase “$routing_key” would be replaced with “database” to form the address as follows (bold added for emphasis only):
Same address with no routing key (notice that there is no “+” symbol):
Formatting Emails and Handling Incidents:
When using the email endpoint, the resulting behavior of the VictorOps platform will depend on the use of predefined keywords in the subject line of the email as follows (keywords are not case sensitive but are shown here in all caps):
- CRITICAL – Either of these keywords will open a new incident, thus triggering whatever escalation policy has been configured for the team receiving the incident.
- WARNING – This keyword will add an entry to the timeline, but not open a new incident.
- INFO – This keyword will post an informational event in the timeline, without creating an incident. (Nobody gets paged)
- ACKNOWLEDGEMENT – This keyword, though rarely used, will acknowledge an incident. The platform will stop paging users.
- RECOVERY – Either of these keywords will resolve an open incident. The platform will stop paging users. (It is not necessary for an incident to be acknowledged before it can be resolved)
When the email is ingested by VictorOps, the subject is parsed and the above keywords are removed. Any remaining text in the subject line will become the title and main identity of the resulting incident (entity_id field). The body of the message will be included as text in the state_message field of the incident. Best practice is to include the keyword at the end to avoid issues with spaces in the title of the incident.
If an email does not contain any of these keywords, it is not parsable. You can determine how VictorOps will respond in such a case within the settings page for the email integration:
Example Incident using Email Endpoint:
The following example email will result in the creation of a new incident that will be routed to the operations team (routing key = opsteam):
Resulting incident in VictorOps:
Expanded view (by clicking “More Info” to view all fields):
The above incident can be acknowledged by sending the same email, but replacing the keyword CRITICAL with the keyword ACKNOWLEDGEMENT. It can be resolved by replacing the keyword CRITICAL with the keywords RESOLVED or OK.
Make sure that the subject line of the the email is the same for all emails related to the a given incident (excluding the keyword, obviously). In other words, if you were to send an email with the subject line “Database server DB6 is down CRITICAL” and then you tried sending an email to resolve the incident with the subject line “Database server DB6 is up RECOVERY”, the VictorOps platform would not recognize that the second email is related to the incident opened by the first (because the entity_id for the first alert contains the word down, while the entity_id for the resolution message is different, containing the word up instead.)
Legacy Email Systems:
Some legacy monitoring tools (and some not so “legacy”) do not permit users to alter the content of the subject line of their email notifications. In this case, it may be possible to user our Transmogrifier tool (Enterprise only) to control the workflow of incidents generated by that tool. Contact our support team at firstname.lastname@example.org for help with this configuration.