Amazon Web Services (AWS) Key Management Service (KMS) Encryption Plugin Setup Guide
MariaDB Enterprise 10.1 introduces robust, full instance, at-rest encryption. This feature uses a flexible plugin interface to allow actual encryption to be done using a key management approach that meets the customer's needs. MariaDB Enterprise includes a plugin that uses the Amazon Web Services (AWS) Key Management Service (KMS) to facilitate separation of responsibilities and remote logging & auditing of key access requests.
Rather than storing the encryption key in a local file, this plugin keeps the master key in AWS KMS. When you first start MariaDB, the AWS KMS plugin will connect to the AWS Key Management Service and ask it to generate a new key. MariaDB will store that key on-disk in an encrypted form. The key stored on-disk cannot be used to decrypt the data; rather, on each startup, MariaDB connects to AWS KMS and has the service decrypt the locally-stored key(s). The decrypted key is stored in-memory as long as the MariaDB server process is running, and that in-memory decrypted key is used to encrypt the local data.
This guide is based on CentOS 7, using systemd with SELinux enabled. Some steps will differ if you use other operating systems.
Sign up for Amazon Web Services
If you already have an AWS account, you can skip this section.
- Load http://aws.amazon.com/.
- Click "Create a Free Account" and complete the steps.
- You'll need to enter credit card information. Charges related only to your use of the AWS KMS service should be limited to about $1/month for the single master key we will create. If you use other services, additional charges may apply. Consult AWS Cloud Pricing Principles https://aws.amazon.com/pricing/ for more information about pricing of AWS services.
- You'll need to complete the AWS identify verification process.
Create an IAM user
After creating an account or logging in to an existing account, follow these steps to create an IAM user with restricted privileges that will use (but not administer) your master encryption key.
- Load the Identity and Access Management Console at https://console.aws.amazon.com/iam/.
- Click "Users" in the left-hand sidebar.
- Click the "Create New Users" button
- Enter a single User Name of your choosing. We'll use "MDBEnc" for this demonstration. Keep the "Generate an access key for each user" box checked.
- Click "Create".
- Click "Show User Security Credentials".
- Copy the Access Key ID and Secret Access Key. Optionally, you can click "Download Credentials". We will need these in order for local programs to interact with AWS using its API.
- Create a file on your computer to hold the credentials for this user. We'll use this file later. It should have this structure:
[default] aws_access_key_id = AKIAIG6IZ6TKF52FVV5A aws_secret_access_key = o7CEf7KhZfsVF9cS0a2roqqZNmuzXtIR869zpSBT
- Click "Close". If prompted because you did not Download Credentials, ensure that you've saved them somewhere, and click "Close".
Create a master encryption key
Now, we'll create a master encryption key. This key can never be retrieved by any application or user. This key is used only to encrypt the actual encryption keys that will be used by MariaDB.
- Click "Encryption Keys" in the left-hand sidebar.
- Click the "Get Started Now" button.
- Click the "Create Key" button.
- Enter an Alias and Description of your choosing.
- Click "Next Step".
- Do not check the box to make the MDBEnc user a Key Administrator.
- Click "Next Step" again.
- Check the box to give the MDBEnc user permissions to use this key.
- Click "Next Step".
- Click "Finish".
You should now see your key listed in the console:
Now, you'll need to copy the "ARN" (Amazon Resource Name) of the key.
- Click on the name of the key to view details.
- Copy and save the "ARN" value. We will use this to set the
aws-key-management-master-key-idparameter for the AWS KMS plugin.
We now have a master encryption key and an IAM user that has privileges to access it using access credentials. This is enough to begin using the AWS KMS plugin.
There are a number of ways to give the IAM credentials to the AWS KMS plugin. The plugin supports reading credentials from all standard locations used across the various AWS API clients. The easiest approach is to simply place the credentials in the MariaDB data directory. The AWS API client looks for a
credentials file in the
.aws subdirectory of the home directory of the user running the client process. In the case of MariaDB, its home directory is its
- Create a
credentialsfile that MariaDB can read. For example:
$ cat /var/lib/mysql/.aws/credentials [default] aws_access_key_id = AKIAIG6IZ6TKF52FVV5A aws_secret_access_key = o7CEf7KhZfsVF9cS0a2roqqZNmuzXtIR869zpSBT
- Create a new option file to tell MariaDB to enable encryption functionality to use the AWS KMS plugin. Create a new file under
/etc/my.cnf.d/(or wherever your OS may have you create such files) with contents like this:
[mariadb] plugin-load-add = aws_key_management.so innodb-encrypt-tables=FORCE innodb-encrypt-log aria-encrypt-tables encrypt-tmp-disk-tables encrypt-binlog encrypt-tmp-files ignore-db-dirs=.aws ignore-db-dirs=.pki aws-key-management aws-key-management-master-key-id = arn:aws:kms:us-west-2:551888187628:key/9da6b21e-e50f-43d4-95d5-a89947c2f75c
- Use the "ARN" you copied above as the value for the
Now, you have told MariaDB to use the AWS KMS plugin and you've put credentials for the plugin in a location where the plugin will find them.
When you start MariaDB, the AWS KMS plugin will connect to the AWS Key Management Service and ask it to generate a new key. MariaDB will store that key on-disk in an encrypted form. The key stored on-disk cannot be used to decrypt the data; rather, on each startup, MariaDB must connect to AWS KMS and have the service decrypt the locally-stored key. The decrypted version is stored in-memory as long as the MariaDB server process is running, and that in-memory decrypted key is used to encrypt the local data.
SELinux and outbound connections from MariaDB
Because MariaDB needs to connect to the AWS KMS service, you must ensure that the host has outbound network connectivity over port 443 to AWS and you must ensure that local policies allow the MariaDB server process to make those outbound connections. By default, SELinux restricts MariaDB from making such connections.
The most simple way to cause SELinux to allow outbound HTTPS connections from MariaDB is to enable to mysql_connect_any boolean, like this:
setsebool -P mysql_connect_any 1
There are more complex alternatives that have a more granular effect, but those are beyond the scope of this document.
Start MariaDB using the
systemctl start mariadb
If you do not use systemd, you may have to start MariaDB using some other mechanism.
You should see journal output similar to this:
# journalctl --no-pager -o cat -u mariadb.service [Note] /usr/sbin/mysqld (mysqld 10.1.9-MariaDB-enterprise-log) starting as process 19831 ... [Note] AWS KMS plugin: generated encrypted datakey for key id=1, version=1 [Note] AWS KMS plugin: loaded key 1, version 1, key length 128 bit [Note] InnoDB: Using mutexes to ref count buffer pool pages [Note] InnoDB: The InnoDB memory heap is disabled [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins [Note] InnoDB: Memory barrier is not used [Note] InnoDB: Compressed tables use zlib 1.2.7 [Note] InnoDB: Using CPU crc32 instructions [Note] InnoDB: Initializing buffer pool, size = 2.0G [Note] InnoDB: Completed initialization of buffer pool [Note] InnoDB: Highest supported file format is Barracuda. [Note] InnoDB: 128 rollback segment(s) are active. [Note] InnoDB: Waiting for purge to start [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1616819 [Note] InnoDB: Dumping buffer pool(s) not yet started [Note] Plugin 'FEEDBACK' is disabled. [Note] AWS KMS plugin: generated encrypted datakey for key id=2, version=1 [Note] AWS KMS plugin: loaded key 2, version 1, key length 128 bit [Note] Using encryption key id 2 for temporary files [Note] Server socket created on IP: '::'. [Note] Reading of all Master_info entries succeded [Note] Added new Master_info '' to hash table [Note] /usr/sbin/mysqld: ready for connections.
Note the several lines of output that refer explicitly to the "AWS KMS plugin". You can see that the plugin generates a "datakey", loads that data key, and then later generates and loads a second data key. The 2nd data key is used to encrypt temporary files and temporary tables.
You can see the encrypted keys stored on-disk in the datadir:
# ls -l /var/lib/mysql/aws* -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.1.1 -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.2.1
Note that those keys are not useful alone. They are encrypted. When MariaDB starts up, the AWS KMS plugin decrypts those keys by interacting with AWS KMS.
For maximum security, you should start from an empty datadir and run
mysql_install_db after configuring encryption. Then you should re-import your data so that it is fully encrypted. Use
sudo to run
mysql_install_db so that it finds your credentials file:
# sudo -u mysql mysql_install_db Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 2016-02-25 23:16:06 139731553998976 [Note] /usr/sbin/mysqld (mysqld 10.1.11-MariaDB-enterprise-log) starting as process 39551 ... 2016-02-25 23:16:07 139731553998976 [Note] AWS KMS plugin: generated encrypted datakey for key id=1, version=1 2016-02-25 23:16:07 139731553998976 [Note] AWS KMS plugin: loaded key 1, version 1, key length 128 bit ...
Create encrypted tables
innodb-encrypt-tables=ON, new InnoDB tables will be encrypted by default, using the key ID set in
innodb_default_encryption_key_id (default 1). With
innodb-encrypt-tables=FORCE enabled, it is not possible to manually bypass encryption when creating a table.
You can cause the AWS KMS plugin to create new encryption keys at-will by specifying a new ENCRYPTION_KEY_ID when creating a table:
MariaDB [test]> create table t1 (id serial, v varchar(32)) ENCRYPTION_KEY_ID=3; Query OK, 0 rows affected (0.91 sec)
[Note] AWS KMS plugin: generated encrypted datakey for key id=3, version=1 [Note] AWS KMS plugin: loaded key 3, version 1, key length 128 bit
# ls -l /var/lib/mysql/aws* -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.1.1 -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.2.1 -rw-rw----. 1 mysql mysql 188 Feb 25 19:10 /var/lib/mysql/aws-kms-key.3.1
Advanced security practices
Ultimately, keeping all the credentials required to read the key on a single host means that a user who has gained access to the host has enough information to read the encrypted files in the datadir, read the encrypted keys from the datadir, interact with AWS KMS to decrypt the encrypted keys, and then used the decrypted keys to decrypt the encrypted data.
Theoretically, a superuser can read the memory of the MariaDB server process to read the decrypted keys or restart MariaDB with password authentication disabled in order to dump data, or add new users to MariaDB in order to allow a user to connect and dump the data. Resolving these issues is beyond the scope of this document. A user who gains root access to your operating system or root access to your MariaDB server will have the ability to decrypt your data. Plan accordingly.
But there are some steps that can be taken to improve matters.
Putting the AWS credentials in a file inside the MariaDB home directory is not ideal. By default, any user with the FILE privilege can read any files that the MariaDB server has permission to read, which would include the credentials file.
But putting them in other locations requires passing additional data to the server, which in the case of CentOS 7 requires customizing the systemd startup procedure.
AWS credentials can be put directly into a "drop-in" systemd file that will be read when starting the MariaDB service:
# cat /etc/systemd/system/mariadb.service.d/aws-kms.conf [Service] Environment=AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A Environment=AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV
After creating that file, execute
systemctl daemon-reload and then start the service as usual.
However, any OS user can read this information from systemd, which could be considered a security risk. Another solution is to put the credentials in a separate file that is only readable by root and then refer to that file using an
EnvironmentFile directive in a drop-in systemd file.
# cat /etc/systemd/system/mariadb.service.d/aws-kms.env AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV # chown root /etc/systemd/system/mariadb.service.d/aws-kms.env # chmod 600 /etc/systemd/system/mariadb.service.d/aws-kms.env # cat /etc/systemd/system/mariadb.service.d/aws-kms.conf [Service] EnvironmentFile=/etc/systemd/system/mariadb.service.d/aws-kms.env
That has the advantage the the credentials can only be read directly by root. systemd adds those variables to the environment of the MariaDB server when starting it, and MariaDB can use the credentials to interact with AWS. Note, though, that any process running as the "mysql" user can still read the credentials from the proc filesystem on Linux.
$ whoami mysql $ cat /proc/$(pgrep mysqld)/environ | tr '\0' '\n' | grep AWS AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV
Separation of responsibilities
In environments where more security is needed, additional measures can be put into place. Note that these measures will make it difficult or impossible to start or restart the MariaDB service programmatically. That is, the OS itself will not be able to restart the service if it fails for some reason or if the host restarts.
The first option is to use the API to disable access to the master key and enable it only when a trusted administrator knows that the MariaDB service needs to be started. A specialized tool on a separate host could be used to enable the key for a very short period of time while the service starts and then quickly disable the key.
The second option is to modify the key policy for the master key so that MFA (Multi-Factor Authentication) is required in order to use the key. This is achieved with a wrapper that handles prompting the user for an MFA token, acquires temporary, limited-privilege credentials from the AWS Security Token Service (STS), and puts those credentials into the environment of the MariaDB server process. The credentials can expire after as little as 15 minutes.