General information and hints on how to automate MariaDB deployments and configuration with Puppet, an open source tool for deployment, configuration, and operations.
Puppet is a tool to automate servers configuration management. It is produced by Puppet Inc, and released under the terms of the Apache License, version 2.
It is entirely possible to use Ansible to automate MariaDB deployments and configuration. This page contains generic information for MariaDB users who want to learn, or evaluate, Puppet.
Puppet modules can be searched using Puppet Forge. Most of them are also published on GitHub with open source licenses. Puppet Forge allows filtering modules to only view the most reliable: supported by Puppet, supported by a Puppet partner, or approved.
For information about installing Puppet, see Installing and upgrading in Puppet documentation.
With Puppet, you write manifests that describe the resources you need to run on certain servers and their attributes.
Therefore manifests are declarative. You don't write the steps to achieve the desired result. Instead, you describe the desired result. When Puppet detects differences between your description and the current state of a server, it decides what to do to fix those differences.
Manifests are also idempotent. You don't need to worry about the effects of applying a manifest twice. This may happen (see Architecture below) but it won't have any side effects.
Here's an example of how to describe a resource in a manifest:
This block describes a resource. The resource type is file, while the resource itself is /etc/motd. The description consists of a set of attributes. The most important is ensure, which in this case states that the file must exist. It is also common to use this resource to indicate that a file (probably created by a previous version of the manifest) doesn't exist.
These classes of resource types exist:
Built-in resources, or Puppet core resources: Resources that are part of Puppet, maintained by the Puppet team.
Defined resources: Resources that are defined as a combination of other resources. They are written in the Puppet domain-specific language.
Custom resources: Resources that are written by users, in the Ruby language.
To obtain information about resources:
To group several resources in a reusable class:
There are several ways to include a class. For example:
Puppet has a main manifest that could be a site.pp file or a directory containing .pp files. For simple infrastructures, we can define the nodes here. For more complex infrastructures, we may prefer to import other files that define the nodes.
Nodes are defined in this way:
The resource type is node. Then we specify a hostname that is used to match this node to an existing host. This can also be a list of hostnames, a regular expression that matches multiple nodes, or the default keyword that matches all hosts. To use a regular expression:
The most important Puppet concepts are the following:
Depending on how the user decides to deploy changes, Puppet can use two different architectures:
An Agent-master architecture. This is the preferred way to use Puppet.
A standalone architecture, that is similar to .
A Puppet master stores a catalog for each target. There may be more than one Puppet master, for redundancy.
Each target runs a Puppet agent in background. Each Puppet agent periodically connects to the Puppet master, sending its facts. The Puppet master compiles the relevant manifest using the facts it receives, and send back a catalog. Note that it is also possible to store the catalogs in PuppetDB instead.
Once the Puppet agent receives the up-to-date catalog, it checks all resources and compares them with its current state. It applies the necessary changes to make sure that its state reflects the resources present in the catalog.
With this architecture, the targets run Puppet apply. This application usually runs as a Linux cron job or a Windows scheduled task, but it can also be manually invoked by the user.
When Puppet apply runs, it compiles the latest versions of manifests using the local facts. Then it checks every resource from the resulting catalogs and compares it to the state of the local system, applying changes where needed.
Newly created or modified manifests are normally deployed to the targets, so Puppet apply can read them from the local host. However it is possible to use PuppetDB instead.
PuppetDB is a Puppet node that runs a PostgreSQL database to store information that can be used by other nodes. PuppetDB can be used with both the Agent-master and the standalone architectures, but it is always optional. However it is necessary to use some advanced Puppet features.
PuppetDB stored the following information:
The latest facts from each target.
The latest catalogs, compiled by Puppet apply or a Puppet master.
Optionally, the recent history of each node activities.
With both architectures, it is possible to have a component called an External Node Classifier (ENC). This is a script or an executable written in any language that Puppet can call to determine the list of classes that should be applied to a certain target.
An ENC received a node name in input, and should return a list of classes, parameters, etc, as a YAML hash.
Bolt can be used in both architectures to run operations against a target or a set of targets. These operations can be commands passed manually to Bolt, scripts, Puppet tasks or plans. Bolt directly connects to targets via ssh and runs system commands.
See to get an idea of what you can do with Bolt.
hiera is a hierarchical configuration system that allows us to:
Store configuration in separate files;
Include the relevant configuration files for every server we automate with Puppet.
See for more information.
.
.
.
.
More information about the topics discussed in this page can be found in the Ansible documentation:
in Puppet documentation.
in Puppet documentation.
.
in Puppet documentation.
Content initially contributed by .
This page is licensed: CC BY-SA / Gnu FDL
Module
A set of manifests.
Resource
A minimal piece of description in a manifest describing system components like files or services.
Resource Type
Determines the class of a resource, e.g., file resource type with multiple resources describing different files.
Attribute
Characteristic of a resource, like file owner or mode.
Class
A group of resources reusable across several manifests.
Hiera in Puppet documentation.
Target
A host whose configuration is managed via Puppet.
Group
A logical group of targets. Example: mariadb group.
Facts
Information collected from targets (e.g., system name/version) using Facter. Can be core or custom facts.
Manifest
A description that can be applied to a target.
Catalog
A compiled manifest.
Apply
Modifying a target's state to reflect its manifest description.
file { '/etc/motd':
content => '',
ensure => present,
}# list existing resource types
puppet resource --typesupp
# print information about the file resource type
puppet describe fileclass ssh_server {
file { '/etc/motd':
content => '',
ensure => present,
}
file { '/etc/issue.net':
content => '',
ensure => present,
}
}include Class['ssh_server']node 'maria-1.example.com' {
include common
include mariadb
}node /^(maria|mysql)-[1-3]\.example\.com$/ {
include common
}hiera is part of Puppet. It is a hierarchical configuration system that allows us to:
Store configuration in separate files;
Include the relevant configuration files for every server we automate with Puppet.
Each hierarchy allows to one choose the proper configuration file for a resource, based on certain criteria. For example criteria may include node names, node groups, operating systems, or datacenters. Hierarchies are defined in a hiera.yaml file, which also defines a path for the files in each hierarchy.
Puppet facts are commonly used to select the proper files to use. For example, a path may be defined as "os/%{facts.os.name}.yaml". In this case, each resource will use a file named after the operating system it uses, in the os directory. You may need to use custom facts, for example to check which microservices will use a MariaDB server, or in which datacenter it runs.
We do not have to create a file for each possible value of a certain fact. We can define a default configuration file with settings that are reasonable for most resources. Other files, when included, will override some of the default settings.
A hiera configuration file will look like this:
This file would include the global files, the OS-specific files and the node-specific files. Each hierarchy will override settings from previous hierarchies.
We can actually have several hiera configuration files. hiera.yaml is the global file. But we will typically have additional hiera configuration files for each environment. So we can include the configuration files that apply to production, staging, etc, plus global configuration files that should be included for every environment.
Importantly, we can also have hiera configuration files for each module. So, for example, a separate mariadb/hiera.yaml file may defined the hierarchies for MariaDB servers. This allow us to define, for example, different configuration files for MariaDB and for MaxScale, as most of the needed settings are typically different.
You probably noticed that, in the previous example, we defined data_hash: yaml_data, which indicates that configuration files are written in YAML. Other allowed formats are JSON and HOCON. The data_hash setting is defined in defaults, but it can be overridden by hierarchies.
Content initially contributed by .
This page is licensed: CC BY-SA / Gnu FDL
This page contains links to Puppet modules that can be used to automate MariaDB deployment and configuration. The list is not meant to be exhaustive. Use it as a starting point, but then please do your own research.
Puppet Forge is the website to search for Puppet modules, maintained by the Puppet company. Modules are searched by the technology that needs to be automated, and the target operating system.
Search criteria include whether the modules are supported by Puppet or its partners, and whether a module is approved by Puppet. Approved modules are certified by Puppet based on their quality and maintenance standards.
Some modules that support the Puppet Development Kit allow some types of acceptance tests.
We can run a static analysis on a module's source code to find certain bad practices that are likely to be a source of bugs:
If a module's authors wrote unit tests, we can run them in this way:
At the time of writing, there are no supported or approved modules for MariaDB.
However there is a supported by Puppet, that supports the Puppet Development Kit. Though it doesn't support MariaDB-specific features, it works with MariaDB. Its documentation shows how to use the module to install MariaDB on certain operating systems.
Several unsupported and not approved modules exist for MariaDB and MaxScale.
website.
documentation.
in Puppet documentation.
in Puppet documentation.
Content initially contributed by .
This page is licensed: CC BY-SA / Gnu FDL
Puppet can also be used to manage Docker container upgrades and configuration changes. Docker has more specific tools for this purpose, but sometimes there are reasons to choose alternatives. See .
In this page you will find out what managing Docker with Puppet looks like. All the snippets in this page use the docker resource type, supported by the Puppet company.
Installing or upgrading Docker is simple:
In this example we are using our system's repositories instead of Docker official repositories, and we are specifying the desired version. To upgrade Docker later, all we need to do is to modify the version number. While specifying a version is not mandatory, it is a good idea because it makes our manifest more reproducible.
Puppet Supported Modules page in Puppet Forge.
To uninstall Docker:
Check the docker resource type documentation to find out how to use more features: for example you can use Docker Enterprise Edition, or bind the Docker daemon to a TCP port.
To pull an image from Dockerhub:
We specified the 10.0 tag to get the desired MariaDB version. If we don't, the image with the latest tag will be used. Note that this is not desirable in production, because it can lead to unexpected upgrades.
You can also write a Dockerfile yourself, and then build it to create a Docker image. To do so, you need to instruct Puppet to copy the Dockerfile to the target and then build it::
It is also possible to subscribe to Dockerfile changes, and automatically rebuild the image whenever a new file is found:
To remove an image that was possibly built or pulled:
To run a container:
mariadb-01 is the contained name. We specified the optional 10.5 tag, and we mapped the guest port 3306 to the host port 6606. In production, you normally don't map ports because you don't need to connect MariaDB clients from the host system to MariaDB servers in the containers. Third-party tools can be installed as separate containers.
docker resource type documentation, in Puppet documentation.
Content initially contributed by Vettabase Ltd.
This page is licensed: CC BY-SA / Gnu FDL
version: 5
defaults:
datadir: global
data_hash: yaml_data
hierarchy:
- name: "Node data"
path: "nodes/%{trusted.certname}.yaml"
- name: "OS data"
path: "os/%{facts.os.family}.yaml"
- name: "Per-datacenter business group data" # Uses custom facts.
path: "location/%{facts.whereami}/%{facts.group}.yaml"pdk validatepdk test unitclass { 'docker':
use_upstream_package_source => false,
version => '17.09.0~ce-0~debian',
}class { 'docker':
ensure => absent
}docker::image { 'mariadb:10.0': }file { '/path/to/remote/Dockerfile':
ensure => file,
source => 'puppet:///path/to/local/Dockerfile',
}
docker::image { 'image_name':
docker_file => '/path/to/remote/Dockerfile'
}docker::image { 'image_name':
docker_file => '/path/to/remote/Dockerfile'
subscribe => File['/path/to/remote/Dockerfile'],
}docker::image { 'mariadb':
ensure => absent
}docker::run { 'mariadb-01':
image => 'mariadb:10.5',
ports => ['3306:6606']
}This page shows some examples of what we can do with Bolt to administer a set of MariaDB servers. Bolt is a tool that is part of the ecosystem.
For information about installing Bolt, see in Bolt documentation.
The simplest way to call Bolt and instruct it to do something on some remote targets is the following:
However, for non-trivial setups it is usually better to use an inventory file. An example:
In this way, it will be possible to refer the target by name or alias.
We can also define groups, followed by the group members. For example:
With an inventory of this type, it will be possible to run Bolt actions against all the targets that are members of a group:
In the examples in the rest of the page, the --targets parameter will be indicated in this way, for simplicity: --targets <targets>.
The simplest way to run a command remotely is the following:
To copy a file or a whole directory to targets:
To copy a file or a whole directory from the targets to the local host:
We can use Bolt to run a local script on remote targets. Bolt will temporarily copy the script to the targets, run it, and delete it from the targets. This is convenient for scripts that are meant to only run once.
Puppet tasks are not always as powerful as custom scripts, but they are simpler and many of them are idempotent. The following task stops MariaDB replication:
It is also possible to apply whole manifests or portions of Puppet code (resources) on the targets.
To apply a manifest:
To apply a resource description:
Further information about the concepts explained in this page can be found in Bolt documentation:
Inventory Files in Bolt documentation.
Applying Puppet code in Bolt documentation.
Content initially contributed by Vettabase Ltd.
This page is licensed: CC BY-SA / Gnu FDL
bolt ... --nodes 100.100.100.100,200.200.200.200,300,300,300,300targets:
- uri: maria-1.example.com
name: maria_1
alias: mariadb_main
...groups:
- name: mariadb-staging
targets:
- uri: maria-1.example.com
name: maria_1
- uri: maria-2.example.com
name: maria_2
- name: mariadb-production
targets:
...
...bolt ... --nodes mariadb-stagingbolt command run 'mariadb-admin start-all-slaves' --targets <targets>bolt file upload /path/to/source /path/to/destination --targets <targets>bolt file download /path/to/source /path/to/destination --targets <targets>bolt script run rotate_logs.sh --targets <targets>bolt task run mysql::sql --targets <targets> sql="STOP REPLICA"bolt apply manifests/server.pp --targets <targets>bolt apply --execute "file { '/etc/mysql/my.cnf': ensure => present }" --targets <targets>