Remote Observability Service

Overview

The Remote Observability Service makes SkySQL observability features available for MariaDB Enterprise customers on-premises or in private clouds.

The MariaDB Agent Manager coordinates deployment of the agents needed to consolidate and push data to the Remote Observability Service.

Compatibility

MariaDB Products

  • MariaDB Enterprise ColumnStore 5

  • MariaDB Enterprise ColumnStore 6

  • MariaDB Enterprise Server 10.3

  • MariaDB Enterprise Server 10.4

  • MariaDB Enterprise Server 10.5

  • MariaDB Enterprise Server 10.6

  • MariaDB MaxScale 2.4

  • MariaDB MaxScale 2.5

  • MariaDB MaxScale 6

  • MariaDB Xpand 5.3

  • MariaDB Xpand 6

Operating Systems

  • CentOS 7

  • Red Hat Enterprise Linux 7

  • Red Hat Enterprise Linux 8

  • Rocky Linux 8

  • Ubuntu 18.04 LTS

  • Ubuntu 20.04 LTS

  • Ubuntu 22.04 LTS

Host Types

These instructions describe actions performed on two types of hosts:

  • Forwarding Hosts forward data to the Remote Observability Service. Forwarding Hosts run the following components:

    • A local Prometheus server

    • skysql_uni_exporter

    • MariaDB Agent Manager daemon

  • Monitored Hosts are servers that run MariaDB Enterprise Server, MariaDB Xpand, or MariaDB MaxScale. Monitored Hosts transmit monitoring data to Forwarding Hosts. Monitored Hosts run exporters to collect the monitoring data, such as:

    • mariadb_exporter

    • maxscale_exporter

    • node_exporter

Dedicated Forwarding Hosts are supported, but not required. A service can have one dedicated Forwarding Host.

A Monitored Host can serve as a combined Forwarding and Monitored Host. In non-production environments, particularly when only a single host is in use, it might be preferable for the host to serve both roles.

Introduction

This procedure shows how to install and configure MariaDB Agent Manager to send monitoring data to the Remote Observability Service. This procedure is intended to be followed linearly from the top to the bottom of the page.

Some steps in the procedure contain different instructions for MariaDB Enterprise Server and MariaDB Xpand. If the host is running MariaDB Enterprise ColumnStore or MariaDB Enterprise Cluster (powered by Galera), the instructions for MariaDB Enterprise Server should be used.

Some steps in the procedure contain different instructions for Forwarding Hosts and Monitored Hosts. Read the instructions carefully to determine which instructions apply to each host.

System Requirements

Memory Requirements

On each Forwarding Host, the expected memory footprint of MariaDB Agent Manager and its components is around 150 MB.

Storage Requirements

On each Forwarding Host, the Prometheus data directory must have sufficient storage capacity to cache service metrics. The service metrics typically require 100 MB or less. The specific capacity requirements depend on the number of services, the service types, and the Prometheus configuration.

For additional information about the specific directories that are used, see "MariaDB Agent Manager Directory Reference".

Prerequisites

Before deploying MariaDB Agent Manager, you must meet the following prerequisites:

  • Sign-up for a SkySQL Portal Account

  • Generate a SkySQL API Key with the desired key scopes, and save the API key to a plain-text file

    • For the Remote Observability Service, the API key must have the "Observability API: Write" key scope

    • When users are part of a team, the API key must be created by the team owner.

  • Enable the Remote Observability Service:

    1. Log in to the SkySQL Portal.

    2. Click on "Remote Observability service" in the left navigation menu.

    3. In the popup window, click the "Enable" button.

      If the "Enable" button is not present, then your account already has the Remote Observability Service enabled, so you can skip to the next step.

  • Consider creating a dedicated user account and group for MariaDB Agent Manager:

    • To create a user account named mariadbagent in a default group with the same name:

      $ groupadd mariadbagent
      $ useradd -g mariadbagent mariadbagent
      
    • The instructions below should be executed as the dedicated user account.

Step 1: Download Agent Manager

On each Forwarding Host and Monitored Host, MariaDB Agent Manager must be downloaded.

This step shows how to download MariaDB Agent Manager using your web browser. For alternative details on how to download MariaDB Agent Manager via command-line interface, see "Download Agent Manager via CLI".

Perform the following steps:

  1. Download the MariaDB Agent Manager binary tarball.

  2. Copy the binary tarball to each host.

  3. On each host, extract the files from the binary tarball:

    $ tar -xvzf mariadb-agent-manager-for-pdc-1.8.0.linux-amd64.tar.gz
    $ cd mariadb-agent-manager-for-pdc-1.8.0.linux-amd64
    

Step 2: Install MariaDB Agent Manager

On each Forwarding Host and Monitored Host, MariaDB Agent Manager must be installed by executing the mariadb_agent_manager install sub-command:

$ ./mariadb_agent_manager install

As part of the installation process, the MariaDB Agent Manager components are copied to the installation directories.

Step 3: Configure Firewalls

MariaDB Agent Manager communicates over TCP between the Forwarding Host and each Monitored Host. MariaDB Agent Manager also communicates over TCP between the Forwarding Host and the Remote Observability Service.

Firewall configuration must support this communication.

Forwarding Server ports

On each Forwarding Host, allow incoming traffic on the following ports:

Port

Source

Purpose

TCP/9303

localhost

API port (overridden by specifying the --api-port option when executing the mariadb_agent_manager start sub-command)

Enterprise Server ports

On each Monitored Host running MariaDB Enterprise Server, allow incoming traffic on the following ports:

Port

Source

Purpose

TCP/3306

Each Forwarding Host

Enterprise Server port

TCP/9100

Each Forwarding Host

node_exporter listening port

TCP/9104

Each Forwarding Host

mariadb_exporter listening port

In environments where non-default ports are used, the firewall configuration must be adapted for the specific ports in use.

Xpand ports

On each Monitored Host running MariaDB Xpand, allow incoming traffic on the following ports:

Port

Source

Purpose

TCP/3306

Each Forwarding Host

Xpand MYSQL_PORT

TCP/9100

Each Forwarding Host

node_exporter listening port

TCP/9104

Each Forwarding Host

mariadb_exporter listening port

In environments where non-default ports are used, the firewall configuration must be adapted for the specific ports in use.

MaxScale ports

On each Monitored Host running MariaDB MaxScale, allow incoming traffic on the following ports:

Port

Source

Purpose

MaxScale listener database port (TCP port varies)

Each Forwarding Host

See "MaxScale Listener Database port parameter

TCP/8989

Each Forwarding Host

MaxScale REST API port

TCP/9100

Each Forwarding Host

node_exporter listening port

TCP/9104

Each Forwarding Host

mariadb_exporter listening port

TCP/9105

Each Forwarding Host

maxscale_exporter listening port

In environments where non-default ports are used, the firewall configuration must be adapted for the specific ports in use.

Step 4: Configure MaxScale

If MaxScale is part of the service, the MaxScale node might require some additional configuration.

Configure REST API

MariaDB Agent Manager's maxscale_exporter communicates with MaxScale using the REST API.

If a MaxScale node is a Monitored Host, the node's MaxScale configuration might need to be adapted to allow REST API access from the Forwarding Hosts. The default configuration file is located at /etc/maxscale.cnf.

The following MaxScale parameters are relevant for MariaDB Agent Manager:

Parameter

Description

admin_enabled

The default value of true is required for maxscale_exporter. If this parameter has been set to false, maxscale_exporter will not function.

admin_host

The default value of 127.0.0.1 would not allow connections from a Forwarding Host located on a different host. This parameter should be changed to 0.0.0.0 to allow connections from other hosts.

admin_port

The default value of 8989 works for maxscale_exporter, but the port might need to be changed if the port is already being used.

admin_ssl_key

If TLS is required, then the path to the private key file must be configured.

admin_ssl_cert

If TLS is required, then the path to the certificate file must be configured.

admin_ssl_ca_cert

If TLS is required, then the path to the Certificate Authority (CA) chain file must be configured.

For example, if TLS is not required, a minimalist MaxScale configuration file would only need to set admin_host:

[maxscale]
threads          = auto
admin_host       = 0.0.0.0

Restart MaxScale

On the MaxScale host, if the MaxScale configuration was changed in the previous step, MaxScale should be restarted:

$ sudo systemctl restart maxscale

MaxScale User Accounts

MaxScale's default administrative user is named admin, and the default password is mariadb.

On the MaxScale host, MariaDB recommends changing the default password using the maxctrl alter user command:

$ maxctrl --user=admin --password='mariadb' \
   alter user admin 'NEW_PASSWORD_STRING'

MariaDB also recommends creating a dedicated user account for MariaDB Agent Manager using the maxctrl create user command:

$ maxctrl --user=admin --password='mariadb' \
   create user remote_obs 'PASSWORD_STRING'

Step 5: Create Database Accounts

MariaDB Agent Manager connects to database user accounts on Monitored Hosts.

A dedicated user account should be created for MariaDB Agent Manager.

Enterprise Server 10.5+ users

These instructions apply to Enterprise Server 10.5 and greater. Alternative instructions are provided for Enterprise Server 10.3 and 10.4.

On each MariaDB Enterprise Server host:

CREATE USER remote_obs@'localhost'
   IDENTIFIED BY 'passwd';

GRANT PROCESS, BINLOG MONITOR, REPLICA MONITOR
   ON *.*
   TO remote_obs@'localhost';

GRANT SELECT
   ON mysql.global_priv
   TO remote_obs@'localhost';

Enterprise Server 10.3/10.4 users

These instructions apply to Enterprise Server 10.3 and Enterprise Server 10.4. Alternative instructions are provided for Enterprise Server 10.5 and greater.

On each MariaDB Enterprise Server host:

CREATE USER remote_obs@'localhost'
   IDENTIFIED BY 'passwd';

GRANT PROCESS, REPLICATION CLIENT
   ON *.*
   TO remote_obs@'localhost';

GRANT SELECT
   ON mysql.global_priv
   TO remote_obs@'localhost';

Xpand users

On each MariaDB Xpand cluster:

CREATE USER remote_obs@'localhost'
   IDENTIFIED BY 'passwd';

GRANT PROCESS, REPLICATION CLIENT
   ON *.*
   TO remote_obs@'localhost';

GRANT SELECT
   ON system.*
   TO remote_obs@'localhost';

Step 6: Prepare credentials.json

On each Monitored Host, exporters authenticate using credentials that are stored locally. These credentials must initially be provided in a credentials file.

The MariaDB Agent Manager credentials file is in JSON format. These instructions assume that the credentials file is named credentials.json

On each Monitored Host, create a credentials file:

  • For Monitored Hosts running MariaDB Enterprise Server or MariaDB Xpand, create a mariadb section with the user account and password details from "Step 5: Create Database Accounts".

  • For Monitored Hosts running MariaDB MaxScale, create a maxscale section with the user account and password details from "MaxScale User Accounts".

Expand the sections below to see examples of credentials files.

Enterprise Server credentials

{
  "cred_store": {
    "mariadb": {
      "user": "remote_obs",
      "pwd": "passwd"
    }
  }
}

Xpand credentials

{
  "cred_store": {
    "mariadb": {
      "user": "remote_obs",
      "pwd": "passwd"
    }
  }
}

MaxScale credentials

{
  "cred_store": {
    "maxscale": {
      "user": "admin",
      "pwd": "mariadb"
    }
  }
}

Step 7: Encrypt Passwords

On each Monitored Host, the passwords in the credentials file must be encrypted by executing the mariadb_agent_manager encrypt-creds sub-command:

$ mariadb_agent_manager encrypt-creds -p ~/credentials.json
The following files have been successfully created:
Encrypted credentials:    /home/USER/.config/mariadb-agent-manager/cred.json
Secret key:               /home/USER/.config/mariadb-agent-manager/.key.txt
  • The -p option specifies the path to the credentials file. Alternatively, the REMOTE_AGENT_CREDENTIALS environment variable can be used.

MariaDB Agent Manager encrypts the credentials and saves the encrypted credentials to its configuration directory.

After the credentials have been encrypted, MariaDB Agent Manager deletes the credentials file. If you need to preserve the credentials file, ensure that you have a backup.

Step 8: Prepare configuration.json

MariaDB Agent Manager reads a configuration file to understand the database infrastructure. These instructions assume that the configuration file is named configuration.json. The MariaDB Agent Manager configuration file is in JSON format.

Create a configuration file:

  • Select the example configuration file for your topology and edit to reflect your database infrastructure and parameters.

  • Alternatively, a configuration file can be created using the configuration wizard by executing the mariadb_agent_manager init sub-command.

The configuration file must be copied to each Forwarding Host and each Monitoring Host.

Single Node Enterprise Server

{
  "datacenter": {
    "name": "Example Data Center",
    "location": "Oregon, US",
    "database_services": [
      {
        "name": "example-standalone",
        "type": "standalone",
        "servers": [
          {
            "name": "mdb1",
            "type": "server",
            "hostname": "mdb1.example.com"
          }
        ]
      }
    ]
  }
}

Replicated Transactions Topology

{
  "datacenter": {
    "name": "Example Data Center",
    "location": "Oregon, US",
    "database_services": [
      {
        "name": "example-replicated",
        "type": "replicated",
        "servers": [
          {
            "name": "mxs1",
            "type": "maxscale",
            "hostname": "mxs1.example.com"
          },
          {
            "name": "mdb1",
            "type": "server",
            "hostname": "mdb1.example.com"
          },
          {
            "name": "mdb2",
            "type": "server",
            "hostname": "mdb2.example.com"
          },
          {
            "name": "mdb3",
            "type": "server",
            "hostname": "mdb3.example.com"
          }
        ]
      }
    ]
  }
}

Distributed Transactions Topology

{
  "datacenter": {
    "name": "Example Data Center",
    "location": "Oregon, US",
    "database_services": [
      {
        "name": "example-distributed",
        "type": "distributed",
        "servers": [
          {
            "name": "mxs1",
            "type": "maxscale",
            "hostname": "mxs1.example.com"
          },
          {
            "name": "xpand1",
            "type": "xpand-backend",
            "hostname": "xpand1.example.com"
          },
          {
            "name": "xpand2",
            "type": "xpand-backend",
            "hostname": "xpand2.example.com"
          },
          {
            "name": "xpand3",
            "type": "xpand-backend",
            "hostname": "xpand3.example.com"
          }
        ]
      }
    ]
  }
}

Multi-Node Analytics Topology

{
  "datacenter": {
    "name": "Example Data Center",
    "location": "Oregon, US",
    "database_services": [
      {
        "name": "example-columnstore",
        "type": "columnstore",
        "servers": [
          {
            "name": "mxs1",
            "type": "maxscale",
            "hostname": "mxs1.example.com"
          },
          {
            "name": "cs_node1",
            "type": "server",
            "hostname": "mdb1.example.com"
          },
          {
            "name": "cs_node2",
            "type": "server",
            "hostname": "mdb2.example.com"
          },
          {
            "name": "cs_node3",
            "type": "server",
            "hostname": "mdb3.example.com"
          }
        ]
      }
    ]
  }
}

Galera Cluster Topology

{
  "datacenter": {
    "name": "Example Data Center",
    "location": "Oregon, US",
    "database_services": [
      {
        "name": "example-galera",
        "type": "galera",
        "servers": [
          {
            "name": "mxs1",
            "type": "maxscale",
            "hostname": "mxs1.example.com"
          },
          {
            "name": "mdb1",
            "type": "server",
            "hostname": "mdb1.example.com"
          },
          {
            "name": "mdb2",
            "type": "server",
            "hostname": "mdb2.example.com"
          },
          {
            "name": "mdb3",
            "type": "server",
            "hostname": "mdb3.example.com"
          }
        ]
      }
    ]
  }
}

Step 9: Start MariaDB Agent Manager

On each Forwarding Host and Monitored Host, MariaDB Agent Manager must be started by executing the mariadb_agent_manager start sub-command.

  • On each Forwarding Host, MariaDB Agent Manager can be started for the first time with the following options:

    $ mariadb_agent_manager start -c ~/configuration.json -k ~/skysql-api-key.txt -f
    
    • The -c option specifies the path to the configuration file. Alternatively, the REMOTE_AGENT_CONFIG environment variable can be used.

    • The -k option specifies the path to a file that contains your SkySQL API Key. Alternatively, the REMOTE_AGENT_API_KEY environment variable can be used. Setting the API key is only required the first time that the services are started on the Forwarding Host, because MariaDB Agent Manager saves the API key to a file. If the option or environment variable are not set, MariaDB Agent Manager prompts the user to enter the API key.

    • The -f option indicates that the node is a Forwarding Host. Alternatively, the FORWARDING_SERVER environment variable can be used. If the host is not defined in the configuration file, MariaDB Agent Manager assumes that it is a dedicated Forwarding Host. If the host is defined in the configuration file, MariaDB Agent Manager handles it as a combination Forwarding and Monitored Host.

  • On each Monitored Host, MariaDB Agent Manager can be started with the following options:

    $ mariadb_agent_manager start -c ~/configuration.json
    
    • The -c option specifies the path to the configuration file. Alternatively, the REMOTE_AGENT_CONFIG environment variable can be used.

After the service has been successfully started, MariaDB Agent Manager prints the status of the components running on the host:

STATUS of the SERVICES:
Service: Prometheus                               active
Service: MariaDB Agent Manager Daemon             active
Service: SkySql Uni Exporter                      active
Service: Node Exporter                            active
Service: Maxscale Exporter                        active

Step 10: Check SkySQL Monitoring

After MariaDB Agent Manager has been started on each Forwarding Host and Monitored Host, check SkySQL Monitoring.

Download Agent Manager via CLI

MariaDB Corporation customers can download MariaDB Agent Manager via command-line interface with their Customer Download Token.

Retrieve Customer Download Token

MariaDB Corporation customers can retrieve their Customer Download Token through the MariaDB Customer Portal.

Note

MariaDB Corporation's Customer Download Tokens are customer-specific. Protect the token as you would any security credential.

To retrieve the Customer Download Token for your account:

  1. Navigate to the Customer Download Token at the MariaDB Customer Portal.

  2. Log in using your MariaDB ID.

  3. Copy the Customer Download Token.

Download Binary Tarball via CLI

MariaDB Corporation provides an interface to download binary files using command-line tools or automation.

Binary files can be downloaded using command-line tools or automation from the MariaDB Download interface with the Customer Download Token. The URL is in the following format:

https://dlm.mariadb.com/CUSTOMER_DOWNLOAD_TOKEN/FILE

Download a binary file using the following procedure:

  1. In your web browser, visit the MariaDB Download interface for the MariaDB Agent Manager binary tarball.

  2. In your web browser, navigate to the binary file that you would like to download and copy the URL.

    For example, to download a binary tarball of MariaDB Agent Manager 1.8.0, the URL is:

    https://dlm.mariadb.com/FILE_ID/monitoring/agent-manager/mariadb-agent-manager-for-pdc-1.8.0.linux-amd64.tar.gz
    

    FILE_ID is an internal identifier that is different for each file.

  3. Extract the FILE path from the copied URL.

    For example, to download the file mentioned above, the FILE path is:

    FILE_ID/monitoring/agent-manager/mariadb-agent-manager-for-pdc-1.8.0.linux-amd64.tar.gz
    
  4. Use your Customer Download Token and the FILE path to construct your customer-specific URL to download the file using command-line tools or automation.

    For example, to download the file mentioned above, the customer-specific URL is:

    https://dlm.mariadb.com/CUSTOMER_DOWNLOAD_TOKEN/FILE_ID/monitoring/agent-manager/mariadb-agent-manager-for-pdc-1.8.0.linux-amd64.tar.gz
    
  5. Use your customer-specific URL to download the file using command-line tools or automation:

    For example, to download the file mentioned above using wget:

    $ wget https://dlm.mariadb.com/CUSTOMER_DOWNLOAD_TOKEN/FILE_ID/monitoring/agent-manager/mariadb-agent-manager-for-pdc-1.8.0.linux-amd64.tar.gz
    

    Or using curl:

    $ curl -LO https://dlm.mariadb.com/CUSTOMER_DOWNLOAD_TOKEN/FILE_ID/monitoring/agent-manager/mariadb-agent-manager-for-pdc-1.8.0.linux-amd64.tar.gz
    

Next Steps