Alerting and Notifications from SkySQL

spacer

MariaDB SkySQL is a database-as-a-service (DBaaS) that delivers unmatched flexibility and massive scalability in the cloud. SkySQL offers a dedicated observability portal, SkySQL Monitoring, which serves as a single pane of glass for the customer’s performance metrics across all clouds.

In this blog, we’ll provide a walkthrough of our new “Alerts and Notifications” feature, which is in Technical Preview.

Alerts and Notifications

Outsourcing the tedious task of watching NOC screens to intelligent, reliable automated assistants is not new.  It can transform all the anomalies into notifications that remind us when something abnormal happens. This is the essence of the alerting feature.

For SkySQL Alerts and Notifications, we went even further, involving our own SkyDBA, Support, and SRE engineers to leverage their years of experience supporting MariaDB products. They produced fine-tuned, MariaDB tailored, recommendations in the form of  predefined alerting rules. They are ready to use out-of-the-box.

We also understand that every customer is unique, and we offer our customers the choice of how they want to use those canned rules. They can be tuned to customer-specific use cases and requirements.

We help customers organize and curate all events, in categories, removing repetitive notifications, allowing focus on the important notifications for each functional team.

We understand that the interests of different functional teams are different. Support, DBA, Marketing, and Sales focus on different planes and we provide the tools to organize the notifications for each team or even person.

How to Access

Once you are logged into SkySQL, SkySQL Monitoring can be accessed on the left navigation. You must launch a service first to have access to Monitoring. More details can be found in MariaDB SkySQL Documentation.

Once you are in SkySQL Monitoring, “Alerts” can be found on the left side menu.

SkySQL monitoring interface

SkySQL monitoring

Or you can use the direct link in one of your email notifications. 

SkySQL email alert

Notification email

 

Alerting Screen

Active is the default view. It presents a list of all active alerts sorted by the last activity date. Those are the alerts that are actively running at this moment.

SkySQL alerting: Active default view

Active alerts screen

History will show the log of all alerts that had been fired but are no longer firing. They can be filtered by date.

SkySQL alert history screen

Alert history screen

Users can consult those two views for more information about the active and historical alerts.

Readily Available

This release of Alerting and Monitoring comes with 15 predefined alerting rules. They are configured by our support experts and cover most of the standard scenarios. Predefined Alerts include server-unavailable, CPU/disk/memory utilization, long-running queries, max connections, network errors, and accounts without passwords.

SkySQL pre-canned alert rules

Pre-canned alert rules

By default, all predefined rules are enabled. If one of the rules is met, the notification will be sent only to the Active Alerts screen in the SkySQL Monitoring Portal.

To send an email or other notification when an alert is triggered, the customer needs to specify a rule and recipient.

Highly Configurable

Customers can change the canned alerting rules to produce custom rules that perfectly fit their specific needs or start from scratch and create their own.

On top of the alerting page, you can find a settings button. It looks like a small blue gear.

This opens the configuration screen for all components of the Alerts and Notifications.

By clicking the gear icon, you can create, edit, delete and disable specific rules.

 

(Alert) Rules

The Rules tab in settings provides a user interface for the management of all rules. Customers can change the parameters of the existing rules or create new rules from scratch.

SkySQL alerting rules configuration

Alerting rules configuration screen

They can also use this screen to edit, delete, and enable/disable the alerting rules. When the rule is disabled it is not evaluated and cannot trigger any notification.


Create new alerting rule

The tag section is essential, it allows the configuration of descriptive labels that can be used for filtering or choosing the notification that will take care of this alerting rule.

We added severity and a custom tag — Development for our new rule.

Notifications

Notifications are used to define who will be notified and on what conditions.

SkySQL notifications screen

Notifications screen

We already have several notifications defined. Customers can define their own using the same user interface.

Let’s open the notification named “Notify Ivan for Warning on Development Environment”.

SkySQL notifications edit screen

Edit Notification Screen

As the name of this Notification suggests, but also visible from the attached tags, this Notification will handle all Warnings on our Development environment.

In the verification window below, we can see our rule “New rule” displayed, this is the only rule right now that can trigger this notification.

I can replace “Development” with “Production” and the set of rules that will trigger this notification will be different.

SkySQL notification rules

This makes matching alerting rules to notifications very flexible and easy.

(Notification) Channels

Channels allow you to manage notification methods (i.e. email list, slack definitions, etc.).

SkySQL notification channels

Notification channel

The channels can be saved with user-friendly names and later attached as a notification channel to any Notification.

Let’s illustrate the process end-to-end.

Problem:

As a customer, I want to create a support ticket for every alert I consider a Severity 1. I only want this to be done for the Production environment. I don’t care about the other servers that run in my account.

For me, the default severity levels Info, Warning, and Critical are limiting. I want to use 5 levels of severity — SEV1-SEV5. I don’t want to create repeating events every minute.  A reminder once every 24h will be nice.

Solution:

I will go to the (Alerting) Rules and define rules on my own, fitting my specific requirements for thresholds and limits. Once I am happy with those, I will label them with tags according to my own scale of severity. SEV1-SEV5

SkySQL alerting severity settings

 

For all Severity 1 alerts, I want to send an email to sev1_ticket@mariadb.com to create an internal ticket.

I want also to notify the support directly on the_support@mariadb.com because this is high Severity.

For this, I can reuse the notification channel I created before.

SkySQL notification channel

Then I will link all existing rules of Severity 1 that happen to be on Production to my SEV1 Mail channel.

 

Here I can remove Production to create a ticket for all servers with SEV1. Or add more tags to filter the rules even more.

In all cases, the “RULES THAT WILL TRIGGER THIS NOTIFICATION” will represent the current rules that will match this Notification. I can check them before I save them.

The Channel name is pointed to SEV1 Mail. I can save and my setup is complete.

Now when “Production Server is Unavailable” or the CPU load is on 100% (CPUs are burning) starts firing for a specific server the system will send an email to the support email address designated and will create a ticket.

This will not be repeated in the next 24h — Because of the repeat interval defined in my Notification (24h). The same rule for another server will produce another notification.

Now my setup is complete and ready for some testing. I created a heavy log on those database servers.

Now in Active alerts, we can see our production servers that run on 100% CPU started generating some alerts.

 

We got a notification email as well.

SkySQL alert email

The notification groups those 4 events together because they are part of the same service transitions-pr and the alert fired about the same time.

Our alert notification works as requested.

If in the future, I need to create a new rule, the only thing I have to do is to add labels SEV1 and Production and the rule will be automatically added to the existing configuration.

Create new rule

The name of my SEV1 Notification in “NOTIFICATIONS TRIGGERED BY THIS RULE” will confirm the new rule will be treated the same way as the other SEV1, Production issues.

Simple as that.

SkySQL Alerts and Notifications are in Technical Preview. In the GA version, you can expect more notification channels such as Slack, ServiceNow, PagerDuty, etc.

Plus, even more predefined alerting rules and more metrics.

That was a very quick introduction of the Alerts and Notifications feature. Alerts & Notifications are part of SkySQL Monitoring — an expansive tool that provides all available observability tools for your cloud database services. The best way to find out more is to try it for yourself on SkySQL.

For More Information

 Sign up for SkySQL today and get a $500 credit toward up to three databases of any size and any type.

More details can be found in MariaDB SkySQL Documentation.