1. Home
  2. Alert Behavior
  3. Transmogrifier
  4. Getting Started with the Transmogrifier

Getting Started with the Transmogrifier

What is the Transmogrifier?

The Transmogrifier is a powerful tool that allows you to deliver helpful information and useful resources to your users at the same moment they are notified of the problem, as well as customize the behavior of your incidents.  It is, essentially, a rules engine that allows you to set certain conditions, and trigger custom actions when those conditions are met.

Getting paged for a non-critical issue?  You can use the Transmogrifier to make sure that nobody gets paged for alerts that are not actionable.

Want to attach remediation documentation directly to notifications so your users can immediately start solving the problem without having to dig around?  The Transmogrifier allows you to attach image links and other URLs directly to your notifications.

Here is an example of a typical incident, untouched by the Transmogrifier:

typicalAlertNoTransmog

Here is an example of an incident that has been annotated with useful information and resources using the Transmogrifier:

HeavyTransmogAction

Notice the addition of a graphical representation of the problem, links to run-book documentation, triage notes, the post mortem report for the last time this problem occurred, and a one-touch option to open a new ticket in Jira.

In other words, the Transmogrifier is designed to give you the information and tools you need, when you really need them.


Getting Started

To get started, click Settings >> Alert Behavior >> Transmogrifier.

SettingsAlertBehaviorTransmog

This is where you will create all of your Transmogrifier rules. To create a new rule select “Add a Rule”.
addTransmogRule

This brings you to the Transmog rule creation window below. The top portion sets the matching condition (When should this rule be applied), while everything below defines the actions to be taken.

TransmogSetupWindow


Matching Conditions

The matching condition will determine when this rule should be applied.  You can choose any field that exists within the payload of an alerts and match on a specific value for that field using a direct match or wildcard matching.

TransmogMatchingRule

When viewing an incident in the timeline, field names are on the left and values are on the right:

fieldAndValue

In the above example, the field of interest is the entity_id field and the value that matters is the phrase “disk space”.  The matching condition therefore is the following (wildcard matching used in this example, hence the * symbol).

matchingCondition2

Wildcard Matching

Rules can match on an alert field value using a simplified wildcard syntax to match some or all of the string. The “*” character matches 0 or more characters and the “?” character matches exactly one character. They can be used anywhere in the match pattern, as many times as needed.

Wildcard Examples:
PhraseMatchesDoes Not Match
db?.mydomain.tlddb2.mydomain.tld
dbx.mydomain.tld
db0.mydomain.tld
db.mydomain.tld
db14.mydomain.tld
db-main.mydomain.tld
*.mydomain.tldwww.mydomain.tld
www.subdomain.mydomain.tld
db778.mydomain.tld
mydomain.tld
x.mydomain.tld/with/a/long/path/suffix.html
db-???.*db-123.foobar.baz
db-abc.bazfoo.bar
db-abc123.foobaz.bar
db000.barfoo.baz

Annotations

An annotation is simply bit of information or link to a resource you can add to the payload of any incident that meets the matching condition for this rule. It can be a URL, an image URL, or a Note (just a blob of plain text). You can add multiple annotations to a single rule by clicking + Add an Annotation.

Select the drop-down menu for the annotation:

AnnotationTypeDropDown

URLs

When annotating a URL, you must add a “label”, which is the text that your users will see in the payload of the incident.  Clicking on the annotation will take the user directly to the URL.

If the payload form your monitoring tool includes a URL link in a field, you can use variable expansion (explained below) to import the value of that field into the annotation as you see here.

URLwithVariableExpansion

Image URLs

Any image hosted at a URL can be rendered as an image in the payload of an incident.  Please note that VictorOps does not accept attachments and we do not store images.  Any images annotated to incidents must be hosted and available via URL link.

If the payload from your monitoring tool includes an image URL in a field, you can use variable expansion (explained below) to import the value of that field into the annotation as you see here.
TransmogImageURL

Notes

Annotating a note into your incidents allows you to deliver a specific message to your users in plain text.


Transformations

A transformation is a way to change alert data before it arrives at your VictorOps timeline. Typing the name of an existing field into the Transform’s ‘alert field’ box, allows you to overwrite that field with a new value of your choosing.

Transformation actions can also add entirely new fields to an alert.  This can be accomplished by simply typing the desired name of the field into the alert field section and assigning a value.

Transformation Examples

  • Change the routing key of a particular set of alerts that need to create incidents for a different team.  Assuming you set up an integration that sends all alerts to your Database team, but you want a particular subset of incidents related to a specific host (db03) to go to the Development team (routing_key = devs)

TransformExample1

  • Add a new unique field to an alert.

TransmogTransform2


Stop Flag

At the bottom of any Transmogrifier rule setup window, there is an option to stop processing after the rule has been applied.

TransmogStopAfter

Every alert sent to VictorOps runs through the list of Transmogrifier rules from top to bottom before reaching the timeline.  (More on managing and ordering rules below)  This check box allows you to stop an alert from continuing to process through subsequent rules.  This has performance advantages (speeds up processing of the alert) and allows you to prevent subsequent rules from overwriting the current rule after it has acted upon the alert.


Variable Expansion

The Transmogrifier can pull the content of an alert field into the rule, thus allowing users to dynamically update any annotation or transformation with data from the alert using the following syntax: ${{field_name}}

Variable Expansion Examples:

  • Pull the name of the affected host from the alert and add it into the URL link for your wiki documentation. (The field in the alert containing this information is “host_name”)

VariableExpansion1

  • Turn an image link provided by the monitoring tool into an annotation.

VariableExpansion2

  • Combine multiple fields into the state_message so your users get more information in their notifications, without losing the original information in that field.  (Assuming the field you want to include is “error_message”)

TransformExample3


Managing Rules

All alerts are processed through the Transmogrifier before reaching your timeline in descending order (top to bottom).  This means that the order of the rules is very important.  From the main Transmogrifier page, you can click and drag individual rules to re-order them.  By default, any new rules you create will appear at the top of the list.

It is a good general practice to add a detailed description when creating a rule so your fellow admins will understand its purpose.

Click on the symbols at the right side of the bar to edit or view a rule.

TransmogRuleEditOrView

Options include the ability to enable / disable (rules are enabled by default), delete, and preview.

Preview

To use the preview window, set your matching condition and then click the edit symbol (three horizontal lines) and select Preview.  The preview option opens a small window on the right side of your rules that will display any recent events in your timeline which meet the matching condition you have set.

This allows you to view the content of your alerts without having to switch back to the timeline.  One useful trick is to set a very broad matching condition (use a in the pattern section), then open the preview and find the alert you want to manipulate.  Clicking More Info on the alert shows you the full payload.  Hover over any field and click to automatically populate the matching condition of your rule to match that selection.

TransmogPreviewWindow

To close the preview window, click on the edit symbol again and choose Close Preview.

Scope Limiting

You may encounter situations where wildcard matching is required for a commonly occurring phrase (“down” or “database” for example).  Matching on such a common phrase can cause your rule to be unintentionally applied to alerts that you do not wish to alter.  This problem can be solved by using a sequence of sequential rules to limit the scope of the wildcard matching condition.

For example, lets say we want to catch the word “staging” in the entity_id field and convert those alerts to INFO alerts so we don’t create an incident and notify anyone at 3:00 AM for a problem in a staging environment.  However, there are some cases where you do want to create an incident for the staging environment, so creating a single rule to match on the phrase “staging” could prevent that from happening.  In this case we only want to affect alerts from one particular monitoring tool (New Relic).

First, we need a rule to match on the monitoring tool value.

TransmogScopeLimitExample1

The matching condition for this rule catches all alerts from New Relic.  It then uses a transformation action to declare a new field (new_relic_staging) and uses variable expansion to import the value of the entity_id field into that new field.  This means that all alerts from New Relic now have a new field that is unique to New Relic only.  The subsequent rule will now use wildcard matching on that new field.

TransmogScopeLimitExample1

This second rule (MUST BE POSITIONED BELOW THE FIRST RULE!) matches on the new field created by the first rule, using wildcard matching to catch the phrase “staging”, and then takes the appropriate action.  This limits the scope of the wildcard matching rule to only alerts from New Relic.

AND / OR Logic

OR logic can be achieved by simply replicating a rule with a different matching condition.

Using a set of sequential rules, when ordered correctly, can achieve basic AND logic in the Transmogrifier.  As with scope limiting rules, the first rule must create a new field which can be acted upon by a subsequent rule.

AND Logic Example

Let’s say you want to catch the phrase “disk space” from the entity_id  field AND the name “stage-db-26” from the host_name  field to convert these alerts to INFO events only when both these conditions are met.

The matching condition for the first rule will catch the first desired phrase and use variable expansion to import the value of the second field into a newly declared field.

TransmogScopeLimit2

The matching condition for the second rule (MUST BE POSITIONED BELOW THE FIRST RULE!) checks the newly declared field for the value “stage-db-26” and takes the appropriate action.

TransmogAndLogic2


Integrations and the Transmogrifier

For information on the recommended Transmogrifier rules for specific integrated monitoring tools, select the appropriate tool below:

Datadog

Updated on July 7, 2017

Was this article helpful?

Related Articles