Email Action

Email action delivers messages to email subscribers based on window status events.

Notifications can be triggered by numerous events such as when a window changes status or maintains the REPEAT status as the expression continues to evaluate to true for a defined interval.

Email Client

Messages generated by rule engine are sent via built-in mail client which is configurable on the Settings > Mail Client.

Custom mail client formatting options such as header and footer are applied to all outgoing messages.

Enable Notifications

Open the Email Notifications tab.

Set Enabled status and enter one or more email addresses separated by comma, semi-colon, or whitespace.

Set toggle to Yes to subscribe to notifications initiated by OPEN, REPEAT, or CANCEL triggers.

For repeat notifications, adjust the notification frequency using Repeat Interval, for example every 6 hours or every 10 events.

If necessary, clear the Same as On Open checkbox to configure message content independently.

Settings

Name Description
Enabled Enable or disable email delivery.
Name User-defined email configuration name.
Each rule can have multiple independently executed configurations.
Recipients One or more email subscribers which receive messages.
Use a comma, semi-colon, or whitespace to separate multiple addresses.
Priority Message priority to classify messages in commonly used email clients: Low, Normal, High.
Merge Messages Merge multiple notifications from different rules into one email message for the same subscriber to prevent excessive emails from arriving within a short time span.
The subject line of merged messages is generated automatically and contains rule names, metric names, and the total number of merged notifications in the given message.
For example: Alerts(2)-cpu_busy: statistical-time, nmon_cpu.
Minimum Notification Interval Minimum interval between messages to prevent excessive email messages generated by a rule.
The rule engine discards messages that are generated by the rule before the specified interval from the last sent message expires.

Configure Message Content

Complete message Subject and Text.

Both fields can include any text as well as placeholders to customize outgoing messages based on alert details.

Warning! Rule ${rule} for server ${entity} is active.

Placeholders, escaped with $ and wrapped in curly brackets ${expression}, are evaluated and replaced with actual values when the notification is sent.

Warning! Rule JVM Memory Low for server nurswgvml007 is active.

Subject Field

The Subject field can include plain text, HTML entity characters, and emoji. HTML markup is not supported.

Note that subjects which exceed the 78 character limit according to RFC 2822, Section 2.1.1 are truncated or rejected by mail servers. The actual limit is typically higher, but is implementation-specific.

If placeholders in the subject potentially evaluate to long strings, apply truncate or abbreviate functions to ensure that the subject length remains within the limit.

${entity} received message '${abbreviate(tags.notification, 50)}'

Text Field

The Text field can include any text including emoji and HTML entity characters as well as placeholders.

Unlike the Subject field, Text is not constrained by a length limit and supports HTML markup.

Database Error.
Code: <b>${tags.code}</b>
<pre>
${message}
</pre>

Message text can include output of scripts, SQL queries, and API calls formatted as HTML tables.

Top-10 running containers by CPU:
${addTable(executeSqlQuery(query), 'html', true)}

In addition, the Text field can invoke attachment functions to include portal screenshots as inline images or CSV files as attachments.

${addPortal('AWS Route53 Health Check Detail', aws_entity)}

The Text field supports control flow syntax which supports the customization of content based on alert details.

@if{tags.payload.type != 'cron'}
    ${detailsTable('html')}
@end{}

Trigger Settings

Setting Description
Delay on Open Delay interval for sending notification for OPEN status.
If the window changes to CANCEL status within the specified delay interval, no OPEN status email is sent.
Set this interval to prevent emails on momentary spikes.
Repeat Interval Interval for sending REPEAT status notifications.
If Repeat Interval is set in time units, the exact interval can vary because the REPEAT notifications are triggered by incoming data.
In particular, REPEAT notifications are not sent if the incoming data flow ceases.
Subject Custom subject text, specified for each status separately.
The subject can include placeholders with built-in fields, functions, and expressions, for example: ${round(avg())}.
Note that the maximum allowed length of the subject is limited to several hundred characters in most email clients.
Text Custom message text, specified for each status separately.
The text can include placeholders with built-in fields, functions, and expressions, for example: ${round(avg())}.
Attach Details Include a summary table with window statistics and action links.
Attach Portals One or more portal screenshots attached to the message as inline images.
If a portal is a template, placeholders such as entity, metric, tags are resolved from the window fields.
Series Chart: attach a screenshot containing series monitored by this rule.

Grouping

Incoming data is grouped into windows by metric, entity, and command tags with each window generating emails separately from the others.

If a rule creates excessive windows, restrict the rule filter or add Override exceptions that disable alerting for a particular series.

The override table below contains rules that always return false for the matching series since the value cannot exceed 100%.

Message Composition

  • Subject
  • Content
    • Header
    • Text
    • Detail Table
      • Entity tags
      • Alert fields
      • Command/Event tags
      • User variables
    • Portal Screenshots (inline images in png format)
    • Footer
    • Attachments
      • Files (CSV, Excel, PDF, etc)

Subject

The subject can include placeholders with fields and expressions which are replaced by actual values when the message is sent. If a placeholder is not found, the placeholder is replaced with an empty string.

Sample subject:

[${status}] Rule ${rule} for ${entity} ${tags}

When using placeholders that can be replaced with text of arbitrary length, apply the truncate or abbreviate functions to limit the subject length.

[${status}] Rule ${rule} for ${entity}: ${truncate(tags.error, 100)}

Text

The message body can include placeholders.

Use the HTML tag <br> to split content into multiple lines.

Start Time: ${windowStartTime}<br>
Duration: ${(timestamp / 1000 - windowStartTime) * 1000} ms

Placeholders with link fields are automatically inlined.

Message text can include control flow statements for conditional processing.

[${upper(tags.status)}] ${entityLink} Ω ${getEntityLink(tags.docker-host)}
<span style='color: orange'>${marker}</span>
@if{is_launch}
  ${addTable(entity.tags, 'html')}
@end{}

A header and footer can be specified in both plaintext and HTML format in Email Client Settings and applied to all messages.

Header and footer do not support any placeholders.

<p style="color: #8db600; font-weight: bold; margin: 0px; padding: 0px;">Classification: UNCLASSIFIED</p>
<p style="color: #8db600;">END of MESSAGE</p>

Details Table

The details table is optional and is formatted with styles for enhanced readability in commonly used email client software.

Each table includes multiple parts which are compiled depending on the alert context.

Below the table, links are provided to view extended alert information, open charts, and export underlying data.

Decimal numbers are rounded to five significant digits for readability.

Portals

Web Driver

To attach screenshots, a web driver must be installed and configured on the ATSD server.

To attach a screenshot of the default portal for the given metric, entity and tags, enable Series Chart.

Charts can include multiple series depending on statistical functions referenced in the condition.

abs(forecast_deviation(median())) > 2 && (median() < 200 || median() > 600)

If a rule correlates multiple metrics using database functions or rules functions these metrics are included in the screenshot on the right axis.

avg() > 10 && db_last('memfree') < 500000

To attach additional portals, select them from the Additional Portal drop-down list.

If the additional portal is a template portal, placeholders such as entity, metric, tags are set based on the current window fields.

Monitoring

Monitor the number of messages sent per minute with the email_notifications_per_minute metric collected by ATSD.

https://atsd_hostname:8443/portals/series?entity=atsd&metric=email_notifications_per_minute