Alerts

Introduction#

Alerts can be setup in DBA Dash to provide instant notification for issues that require your immediate attention. Alerts need to be configured before you start receiving alerts. The rules, thresholds and notification schedules can be tailored to your environment.

Rules#

An alert rule defines a condition that you want to be notified about. To create a rule:

• Click the Configure button on the Alerts tab in the GUI.

• Click the *Add Rule button on the Alert Configuration window.

• Select a rule type. The following rule types are currently available:

  • AG Health
  • Collection Dates
  • Counter. e.g.
    • Memory Manager\Memory Grants Pending - queries queued waiting for a memory grant
  • CPU
  • Drive Space
  • Wait. e.g.
    • RESOURCE_SEMAPHORE - queries queued waiting for a memory grant
    • RESOURCE_SEMAPHORE_QUERY_COMPILE - queries queued waiting to compile
    • THREADPOOL - queries waiting for an available worker thread
    • LCK% - queries blocked
  • Agent Job.
  • Offline - connection to the instance has failed.
  • Restart - instance has recently come back online following a restart.

  Receive a notification when a job fails. The rule can be filtered by job name or category (with LIKE syntax support)

  The agent job rule can be used for custom notifications by creating a job that will fail based on your condition

• Configure the rule

  • Apply To (Instance) - enter the name of the instance you want the alert to apply to OR leave blank to apply to all instances.
  • Apply To (Tag) - Select a tag to apply the alert to a group of instances.

• IsActive - Set to false to disable the alert.
• Notes - Option to add notes to describe the purpose of the alert.
• Priority - The priority determines the status icon for the alert and it can also be used to filter alerts for notification purposes (Alerts can be generated without sending a notification). The broad categories are Critical, High (1-10), Medium (1-10), Low (1-10) and Information (1-10).
• Depending on the rule type selected, different properties are available to configure.

• Click Save

Notification Channels#

A notification channel defines where to send the alert notification. To create a channel:

• Click Add Channel on the alert configuration window. Select from the menu options:

  • Webhook - For Google chat and generic webhooks. Google chat supports threaded conversations.
  • Slack - Slack can also be configured as a webhook, but this option uses the API and supports threaded conversations.
  • Teams - Post alerts to a Microsoft Teams channel using a Teams Workflow. See Microsoft Teams.
  • Email
  • PagerDuty. Select the service in PagerDuty. Click Integrations. Click to add an Events API V2 integration. This provides the integration key required when setting up the notification channel in DBA Dash

• Configure the channel.

  • Alert Message Consolidation Threshold - The number of notifications that will trigger a single consolidated alert notification to reduce noise. Default 5.
  • Channel name - How you want to identify the channel
  • Disable From/To - option to disable a notification channel. e.g. Holidays.
  • Schedules - By default, a notification channel is available 24x7 for alerts of Medium priority and higher. You can edit the schedule and add additional schedules as required. For example, you can have a schedule for medium priority alerts that send notifications Mon-Fri, 9am-5pm. An additional schedule can be added for high priority alerts that will send notifications 24x7. If you have teams in different timezones you can configure notification channels and schedules to align with when the teams are available. Schedules can also be applied to a specific tag. So the 24x7 high priority schedule could be filtered for tag Role:Production. Or if you have different teams responsible for different instances a tag could be used to filter instances applicable to the team associated with the notification channel. Note: It’s OK to have overlapping schedules. A channel will receive a single notification. If multiple channels have overlapping schedules, each channel will be notified.

  You can apply schedule filters to notification channels to control which alerts a channel receives based on:

  • Time of day
  • Day of week
  • Alert priority
  • Instance tag (for example: Prod, DEV)

Schedules have an associated re-trigger threshold which is the minimum delay required before sending an updated alert notification. By default you will receive updates every 10 minutes and a maximum of 6 (configurable in Options\Repository Settings) notifications will be sent for each alert. For Slack and Google chat alerts, updates are sent to the previous conversation thread.

Microsoft Teams#

Alerts are posted to a Teams channel as an Adaptive Card using a Teams Workflow. To get the Workflow URL:

1. In Teams, go to the channel you want to receive alerts.
2. Click … (More options) next to the channel name and select Workflows.
3. Select the Send webhook alerts to a channel template. This is the name Microsoft gives this workflow template in Teams - it’s unrelated to the Message Template field described below, which is DBA Dash’s own setting.
4. Give the workflow a name, check the team and channel are correct, and click Add workflow.
5. Copy the URL that is displayed.

In DBA Dash, click Add Channel and select Teams. Paste the URL into Workflow Url and give the channel a name. Once saved, click the Test link for the channel to send a test notification.

Troubleshooting#

With the unmodified Send webhook alerts to a channel workflow, Teams accepts the request before the workflow runs, so DBA Dash can report that a notification was sent even if it doesn’t appear in Teams. If this happens, open the Workflows app in Teams, select the workflow and check the run history for failed runs. The details show why the message wasn’t posted.

The workflow runs as the user who created it. If that user leaves the organization or their account is disabled, the workflow will stop working. Consider creating the workflow with a service account.

Customizing the message#

DBA Dash’s own Message Template field (on the notification channel, separate from the Teams workflow template selected in step 3) is optional. Leave it blank to use the default card. To customize the message, provide JSON in this format. The Adaptive Cards Designer can help you design the card. Select Microsoft Teams as the host app to preview how it will look.

{
  "type": "message",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.adaptive",
      "contentUrl": null,
      "content": {
        "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
        "type": "AdaptiveCard",
        "version": "1.4",
        "body": [
          { "type": "TextBlock", "text": "{Emoji} {Title}", "weight": "Bolder", "size": "Medium", "wrap": true },
          { "type": "TextBlock", "text": "{Instance}", "isSubtle": true, "spacing": "None", "wrap": true },
          { "type": "TextBlock", "text": "{Text}", "wrap": true }
        ]
      }
    }
  ]
}

Available placeholders: {Title}, {Text}, {Instance}, {ConnectionID}, {InstanceAndConnectionID}, {AlertKey}, {Action}, {Icon}, {IconUrl}, {Emoji}, {Priority}, {PriorityBucket}, {TriggerDate}, {Now}, {CloudProvider}, {CloudResourceID}, {CloudRegion}, {CloudAccountID}.

Threaded replies#

With the unmodified Send webhook alerts to a channel workflow, each notification for an alert (including when it’s resolved) is posted as a new message. To post updates as replies to the original message, edit the workflow so it returns the ID of the message it posts. DBA Dash stores the ID and includes it as replyToMessageId in the body of later notifications for the alert.

Edit the workflow so it does the following after the When a Teams webhook request is received trigger:

1. Add a Condition that checks if replyToMessageId is empty. e.g. empty(triggerBody()?['replyToMessageId']) is equal to true.
2. True (new alert):
   • Post card in a chat or channel - Post in Channel and select the team and channel. For the Adaptive Card, use triggerBody()?['attachments'][0]['content'].
   • Response - Status code 200. Add a header named MessageId with the Message ID output from the previous step.
3. False (update for an existing alert):
   • Reply with an adaptive card in a channel - Select the team and channel. For the Message ID, use triggerBody()?['replyToMessageId']. For the Adaptive Card, use triggerBody()?['attachments'][0]['content'].
   • Response - Status code 200.

The MessageId header is preferred, but DBA Dash will also accept the message ID in the response body. Either as the body on its own, or as JSON: {"messageId": "..."}.

Notification channel groups#

Notification channel groups let you target alert rules to specific sets of notification channels (email, Google Chat, Slack, Teams, PagerDuty, etc.). A default group is created automatically and additional groups can be created if required.

How to create a notification channel group#

1. On the Alerts tab, click Configure.
2. Click Add Group, enter a name, and click OK.
3. Create or edit a notification channel and assign it to the group.

4. When creating or editing an alert rule, select the group to control where notifications are sent.

Desktop notifications#

DBA Dash can send desktop notifications (popup window) while you are using the GUI. To receive desktop notifications, check the Desktop Notifications from the options menu. If desktop notifications are disabled, there is still a visual cue in the application that you have alerts that require your attention in the menu area on the top right. This shows the number of active alerts and the number that require your attention (alerts that are not resolved or acknowledged).

Blackout Periods#

A blackout defines when alerts are not monitored - e.g. For maintenance. Alerts are not generated during the blackout. If an alert is active at the time of a blackout, it’s status is changed to indicate that it is now in blackout.

Disable alerts for the next hour#

If you want to disable alerts for the next hour, click Add Blackout on the Alert Configuration window. The start date and end date already default to the next hour - adjust these if required. The rest of the configuration can be left to the defaults - click OK to create the blackout. More advanced configuration is available, including scheduled blackouts and blackouts targeting specific alerts and instances.

Blackout configuration#

• Start Date - when the blackout period starts. Defaults to current date/time
• End Date - when the blackout period ends. Defaults to the next hour. For scheduled alerts, set to empty/null or a date in the distant future.
• Alert Key - By default the blackout period will apply to all alerts. To filter for a specific alert, enter the alert key to filter on (LIKE syntax is supported.). Alternatively, create the blackout using the Add Blackout link in the alert rules grid.
• Apply To (Instance) - option to filter the blackout period to a specific instance
• Apply To (Tag) - option to apply the blackout period to a specific tag. e.g. Role:DEV
• Notes - option to add notes to describe the purpose of the blackout.
• Time From/To - For scheduled alerts, enter a time when you want the blackout period to start & end. e.g. 23:00.
• TimeZone - The timezone used for Time From/To.
• Days - check which days of the week you want the blackout to apply.

Service Config Tool#

By default, the service config tool is already configured to process alerts. If you have multiple services, only one of the services will process alerts at a time. In some cases you might want to control which service processes alert notifications (maybe outbound connections are disabled preventing notifications). On the Options tab you have the following alert configuration options:

• Process Alerts - option to disable alert processing.
• Alert polling frequency - How often we check alert rules & process notifications.
• Alert startup delay - when the service starts you might want to introduce an additional delay, allowing collections time to run before generating alerts.

Handling new alerts#

An alert can be acknowledged by clicking the Acknowledge link in the grid or selecting Acknowledge alerts from the Actions menu. Once acknowledged, notifications for the alert will be muted. Notifications are also muted once the maximum number of notifications is reached (Configurable in Options\Repository Settings).

You can add notes to the alert to explain the root cause by clicking the link in the Notes column. This can be a simple line of text or a full RCA written in markdown. You can also add the same notes to multiple alerts by selecting them in the grid and using the option from the Actions menu.

Once an alert has been resolved it will be automatically closed after a period of time (Configurable in Options\Repository Settings). You can also Close alerts manually by clicking the Close link or the option from the Actions menu. Once closed, a new alert will be generated if the rule is triggered again. This will reset the notification count and start new threads in Slack and Google chat. To see Closed alerts, click the History button on the Alerts tab.

Other alert options#

SQL Server has built-in alert capabilities that can be used alongside DBA Dash. For example, you can configure alerts for high severity errors, corruption, performance conditions & job failures.