Amazon Web Services (AWS) Key Management Service (KMS) Encryption Plugin Setup Guide
Contents
Overview
MariaDB Enterprise 10.1 introduces robust, full instance, at-rest encryption. This feature uses a flexible plugin interface to allow actual encryption to be done using a key management approach that meets the customer's needs. MariaDB Enterprise includes a plugin that uses the Amazon Web Services (AWS) Key Management Service (KMS) to facilitate separation of responsibilities and remote logging & auditing of key access requests.
Rather than storing the encryption key in a local file, this plugin keeps the master key in AWS KMS. When you first start MariaDB, the AWS KMS plugin will connect to the AWS Key Management Service and ask it to generate a new key. MariaDB will store that key on-disk in an encrypted form. The key stored on-disk cannot be used to decrypt the data; rather, on each startup, MariaDB connects to AWS KMS and has the service decrypt the locally-stored key(s). The decrypted key is stored in-memory as long as the MariaDB server process is running, and that in-memory decrypted key is used to encrypt the local data.
This guide is based on CentOS 7, using systemd with SELinux enabled. Some steps will differ if you use other operating systems.
Sign up for Amazon Web Services
If you already have an AWS account, you can skip this section.
- Load http://aws.amazon.com/.
- Click "Create a Free Account" and complete the steps.
- You'll need to enter credit card information. Charges related only to your use of the AWS KMS service should be limited to about $1/month for the single master key we will create. If you use other services, additional charges may apply. Consult AWS Cloud Pricing Principles https://aws.amazon.com/pricing/ for more information about pricing of AWS services.
- You'll need to complete the AWS identify verification process.
Create an IAM user
After creating an account or logging in to an existing account, follow these steps to create an IAM user with restricted privileges that will use (but not administer) your master encryption key.
- Load the Identity and Access Management Console at https://console.aws.amazon.com/iam/.
- Click "Users" in the left-hand sidebar.
- Click the "Create New Users" button
- Enter a single User Name of your choosing. We'll use "MDBEnc" for this demonstration. Keep the "Generate an access key for each user" box checked.
- Click "Create".
- Click "Show User Security Credentials".
- Copy the Access Key ID and Secret Access Key. Optionally, you can click "Download Credentials". We will need these in order for local programs to interact with AWS using its API.
- Create a file on your computer to hold the credentials for this user. We'll use this file later. It should have this structure:
[default] aws_access_key_id = AKIAIG6IZ6TKF52FVV5A aws_secret_access_key = o7CEf7KhZfsVF9cS0a2roqqZNmuzXtIR869zpSBT
- Click "Close". If prompted because you did not Download Credentials, ensure that you've saved them somewhere, and click "Close".
Create a master encryption key
Now, we'll create a master encryption key. This key can never be retrieved by any application or user. This key is used only to encrypt the actual encryption keys that will be used by MariaDB.
- Click "Encryption Keys" in the left-hand sidebar.
- Click the "Get Started Now" button.
- Click the "Create Key" button.
- Enter an Alias and Description of your choosing.
- Click "Next Step".
- Do not check the box to make the MDBEnc user a Key Administrator.
- Click "Next Step" again.
- Check the box to give the MDBEnc user permissions to use this key.
- Click "Next Step".
- Click "Finish".
You should now see your key listed in the console:
Now, you'll need to copy the "ARN" (Amazon Resource Name) of the key.
- Click on the name of the key to view details.
- Copy and save the "ARN" value. We will use this to set the
aws-key-management-master-key-id
parameter for the AWS KMS plugin.
We now have a master encryption key and an IAM user that has privileges to access it using access credentials. This is enough to begin using the AWS KMS plugin.
Configure MariaDB
There are a number of ways to give the IAM credentials to the AWS KMS plugin. The plugin supports reading credentials from all standard locations used across the various AWS API clients. The easiest approach is to simply place the credentials in the MariaDB data directory. The AWS API client looks for a credentials
file in the .aws
subdirectory of the home directory of the user running the client process. In the case of MariaDB, its home directory is its datadir
.
- Create a
credentials
file that MariaDB can read. For example:$ cat /var/lib/mysql/.aws/credentials [default] aws_access_key_id = AKIAIG6IZ6TKF52FVV5A aws_secret_access_key = o7CEf7KhZfsVF9cS0a2roqqZNmuzXtIR869zpSBT
- Create a new option file to tell MariaDB to enable encryption functionality to use the AWS KMS plugin. Create a new file under
/etc/my.cnf.d/
(or wherever your OS may have you create such files) with contents like this:[mariadb] plugin-load-add = aws_key_management.so innodb-encrypt-tables=FORCE innodb-encrypt-log aria-encrypt-tables encrypt-tmp-disk-tables encrypt-binlog encrypt-tmp-files ignore-db-dirs=.aws ignore-db-dirs=.pki secure-file-priv=/var/lib/mariadb-load-data aws-key-management aws-key-management-master-key-id = arn:aws:kms:us-west-2:551888187628:key/9da6b21e-e50f-43d4-95d5-a89947c2f75c
- Use the "ARN" you copied above as the value for the
aws-key-management-master-key-id
option.
Now, you have told MariaDB to use the AWS KMS plugin and you've put credentials for the plugin in a location where the plugin will find them.
When you start MariaDB, the AWS KMS plugin will connect to the AWS Key Management Service and ask it to generate a new key. MariaDB will store that key on-disk in an encrypted form. The key stored on-disk cannot be used to decrypt the data; rather, on each startup, MariaDB must connect to AWS KMS and have the service decrypt the locally-stored key. The decrypted version is stored in-memory as long as the MariaDB server process is running, and that in-memory decrypted key is used to encrypt the local data.
SELinux and outbound connections from MariaDB
Because MariaDB needs to connect to the AWS KMS service, you must ensure that the host has outbound network connectivity over port 443 to AWS and you must ensure that local policies allow the MariaDB server process to make those outbound connections. By default, SELinux restricts MariaDB from making such connections.
The most simple way to cause SELinux to allow outbound HTTPS connections from MariaDB is to enable to mysql_connect_any boolean, like this:
setsebool -P mysql_connect_any 1
There are more complex alternatives that have a more granular effect, but those are beyond the scope of this document.
Start MariaDB
Start MariaDB using the systemctl
tool:
systemctl start mariadb
If you do not use systemd, you may have to start MariaDB using some other mechanism.
You should see journal output similar to this:
# journalctl --no-pager -o cat -u mariadb.service [Note] /usr/sbin/mysqld (mysqld 10.1.9-MariaDB-enterprise-log) starting as process 19831 ... [Note] AWS KMS plugin: generated encrypted datakey for key id=1, version=1 [Note] AWS KMS plugin: loaded key 1, version 1, key length 128 bit [Note] InnoDB: Using mutexes to ref count buffer pool pages [Note] InnoDB: The InnoDB memory heap is disabled [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins [Note] InnoDB: Memory barrier is not used [Note] InnoDB: Compressed tables use zlib 1.2.7 [Note] InnoDB: Using CPU crc32 instructions [Note] InnoDB: Initializing buffer pool, size = 2.0G [Note] InnoDB: Completed initialization of buffer pool [Note] InnoDB: Highest supported file format is Barracuda. [Note] InnoDB: 128 rollback segment(s) are active. [Note] InnoDB: Waiting for purge to start [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1616819 [Note] InnoDB: Dumping buffer pool(s) not yet started [Note] Plugin 'FEEDBACK' is disabled. [Note] AWS KMS plugin: generated encrypted datakey for key id=2, version=1 [Note] AWS KMS plugin: loaded key 2, version 1, key length 128 bit [Note] Using encryption key id 2 for temporary files [Note] Server socket created on IP: '::'. [Note] Reading of all Master_info entries succeded [Note] Added new Master_info '' to hash table [Note] /usr/sbin/mysqld: ready for connections.
Note the several lines of output that refer explicitly to the "AWS KMS plugin". You can see that the plugin generates a "datakey", loads that data key, and then later generates and loads a second data key. The 2nd data key is used to encrypt temporary files and temporary tables.
You can see the encrypted keys stored on-disk in the datadir:
# ls -l /var/lib/mysql/aws* -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.1.1 -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.2.1
Note that those keys are not useful alone. They are encrypted. When MariaDB starts up, the AWS KMS plugin decrypts those keys by interacting with AWS KMS.
For maximum security, you should start from an empty datadir and run mysql_install_db
after configuring encryption. Then you should re-import your data so that it is fully encrypted. Use sudo
to run mysql_install_db
so that it finds your credentials file:
# sudo -u mysql mysql_install_db Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 2016-02-25 23:16:06 139731553998976 [Note] /usr/sbin/mysqld (mysqld 10.1.11-MariaDB-enterprise-log) starting as process 39551 ... 2016-02-25 23:16:07 139731553998976 [Note] AWS KMS plugin: generated encrypted datakey for key id=1, version=1 2016-02-25 23:16:07 139731553998976 [Note] AWS KMS plugin: loaded key 1, version 1, key length 128 bit ...
Create encrypted tables
With innodb-encrypt-tables=ON
, new InnoDB tables will be encrypted by default, using the key ID set in innodb_default_encryption_key_id
(default 1). With innodb-encrypt-tables=FORCE
enabled, it is not possible to manually bypass encryption when creating a table.
You can cause the AWS KMS plugin to create new encryption keys at-will by specifying a new ENCRYPTION_KEY_ID when creating a table:
MariaDB [test]> create table t1 (id serial, v varchar(32)) ENCRYPTION_KEY_ID=3; Query OK, 0 rows affected (0.91 sec)
[Note] AWS KMS plugin: generated encrypted datakey for key id=3, version=1 [Note] AWS KMS plugin: loaded key 3, version 1, key length 128 bit
# ls -l /var/lib/mysql/aws* -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.1.1 -rw-rw----. 1 mysql mysql 188 Feb 25 18:55 /var/lib/mysql/aws-kms-key.2.1 -rw-rw----. 1 mysql mysql 188 Feb 25 19:10 /var/lib/mysql/aws-kms-key.3.1
Advanced security practices
Ultimately, keeping all the credentials required to read the key on a single host means that a user who has gained access to the host has enough information to read the encrypted files in the datadir, read the encrypted keys from the datadir, interact with AWS KMS to decrypt the encrypted keys, and then used the decrypted keys to decrypt the encrypted data.
Theoretically, a superuser can read the memory of the MariaDB server process to read the decrypted keys or restart MariaDB with password authentication disabled in order to dump data, or add new users to MariaDB in order to allow a user to connect and dump the data. Resolving these issues is beyond the scope of this document. A user who gains root access to your operating system or root access to your MariaDB server will have the ability to decrypt your data. Plan accordingly.
AWS credentials
Putting the AWS credentials in a file inside the MariaDB home directory is not ideal. By default, any user with the FILE privilege can read any files that the MariaDB server has permission to read, which would include the credentials file. To protect against this, you should set secure_file_priv
to restrict the location the server will allow a user to read from when executing LOAD DATA INFILE
or the LOAD_FILE()
function.
But putting them in other locations requires passing additional data to the server, which in the case of CentOS 7 requires customizing the systemd startup procedure. This is most easily done by creating a "drop-in" file in /etc/systemd/system/mariadb.service.d/. The file should end in ".conf" but can otherwise be named whatever you like. After making any changes to systemd files, execute systemctl daemon-reload
and then start (or restart) the service as usual.
You can place the credentials file in a location of your choosing and then refer to that file by setting the AWS_CREDENTIAL_PROFILES_FILE
environment variable in the drop-in file:
[Service] Environment=AWS_CREDENTIAL_PROFILES_FILE=/etc/aws-kms-credentials
The credentials file will need to be readable by the "mysql" user, but it does not need to be readable by any other user.
AWS credentials can also be put directly into a "drop-in" systemd file that will be read when starting the MariaDB service:
# cat /etc/systemd/system/mariadb.service.d/aws-kms.conf [Service] Environment=AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A Environment=AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV
However, any OS user can read this information from systemd, which could be considered a security risk. Another solution is to put the credentials in a separate file that is only readable by root and then refer to that file using an EnvironmentFile
directive in a drop-in systemd file.
# cat /etc/systemd/system/mariadb.service.d/aws-kms.env AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV # chown root /etc/systemd/system/mariadb.service.d/aws-kms.env # chmod 600 /etc/systemd/system/mariadb.service.d/aws-kms.env # cat /etc/systemd/system/mariadb.service.d/aws-kms.conf [Service] EnvironmentFile=/etc/systemd/system/mariadb.service.d/aws-kms.env
That has the advantage the the credentials can only be read directly by root. systemd adds those variables to the environment of the MariaDB server when starting it, and MariaDB can use the credentials to interact with AWS. Note, though, that any process running as the "mysql" user can still read the credentials from the proc filesystem on Linux.
$ whoami mysql $ cat /proc/$(pgrep mysqld)/environ | tr '\0' '\n' | grep AWS AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV
Separation of responsibilities
In environments where more security is needed, additional measures can be put into place. Note that these measures will make it difficult or impossible to start or restart the MariaDB service programmatically. That is, the OS itself will not be able to restart the service if it fails for some reason or if the host restarts.
Using a Multi-Factor Authentication (MFA) device
One approach is to modify the key policy for the master key so that MFA (Multi-Factor Authentication) is required in order to use the key. This is achieved with a wrapper that handles prompting the user for an MFA token, acquires temporary, limited-privilege credentials from the AWS Security Token Service (STS), and puts those credentials into the environment of the MariaDB server process. The credentials can expire after as little as 15 minutes.
To require an MFA token for users of the key, you'll need to manually edit the key policy.
- Load the IAM console at https://console.aws.amazon.com/iam/.
- Click "Encryption Keys" in the left-hand sidebar.
- Click the name of your encryption key to view its details.
- Click the link labeled "Switch to policy view", to the right of the heading of the "Key Policy" section.
- Locate the section that contains
"Sid": "Allow use of the key"
. - Add this text below the
"Sid"
line:"Condition": { "Bool": { "aws:MultiFactorAuthPresent": "True" } },
- Click "Save Changes".
- Click "Proceed" if prompted with a warning about using the default view in the future.
Now, add an MFA device for your user. You'll need to have a hardware MFA device or an application such as Google Authenticator installed on your smartphone.
- Click "Users" in the left-hand sidebar.
- Click the name of your user.
- Click the "Security Credentials" tab.
- In the "Sign-In Credentials" section, click the "Manage MFA Device" button.
- Complete the steps to activate your MFA device.
- Copy the ARN for your MFA device. You will need to use this when configuring the wrapper program.
Now, set up the wrapper program.
- Copy the iam-kms-wrapper file to /usr/local/bin/, and ensure that it is executable.
- Create a drop-in systemd config file in
/etc/systemd/system/mariadb.service.d/
:[Service] EnvironmentFile=/etc/systemd/system/mariadb.service.d/aws-kms.env ExecStart= ExecStart=/usr/local/bin/iam-kms-wrapper --config=/etc/my.cnf.d/iam-kms-wrapper.config /usr/sbin/mysqld $MYSQLD_OPTS
- Execute
systemctl daemon-reload
. - Create a file at
/etc/my.cnf.d/iam-kms-wrapper.config
with these contents, using the ARN for your MFA device as the value forkms_mfa_id
:[kms] kms_mfa_id = arn:aws:iam::551888187628:mfa/MDBEnc kms_mfa_socket = /tmp/kms_mfa_socket
When you start the MariaDB service now, the wrapper will temporarily create a socket file at the location given by the kms_mfa_socket
option. The wrapper will read the MFA code from the socket and will use it to authenticate to KMS. To give the MFA code, simply write the digits to the socket file using echo
: echo 111676 > /tmp/kms_mfa_socket
.
The systemctl
command will block until MariaDB starts, so you will need to write the code to the socket file via a separate terminal.
Disabling keys when not needed
Another possibility is to use the API to disable access to the master key and enable it only when a trusted administrator knows that the MariaDB service needs to be started. A specialized tool on a separate host could be used to enable the key for a very short period of time while the service starts and then quickly disable the key.
To do this, you can create an extra IAM User that can only use the kms:EnableKey and kms:DisableKey API endpoints for your key. This user will not be able to encrypt or decrypt any data using the key.
First, create a new user.
- Load the IAM console at https://console.aws.amazon.com/iam/.
- Click "Users" in the left-hand sidebar.
- Click "Create New Users".
- Enter a new user name. (The examples will use "MDBEncAdmin".)
- Click "Show User Security Credentials".
- Copy the credentials and put them in a
credentials
file with this structure:[MDBEncAdmin] aws_access_key_id=AKIAJMPPNO7EBKABCDEF aws_secret_access_key=pVdGwbuK5/jG64aBK1oEJOXRlkdM0aAylgabCDef
- Click "Close".
- Click "Close" again if prompted.
- Click the name of your new user to open the details view.
- Copy the "User ARN" value for your user (for example "arn:aws:iam::551888181234:user/MDBEncAdmin"). You will need this for the next step.
Now, give the new user permission to perform API operations on your key.
- Click "Encryption Keys" in the left-hand sidebar.
- Click the name of your key to open the details view.
- Click "Switch to policy view" if it is not already open. (The "policy view" is a large text field that contains JSON describing the key policy.)
- Create a new item in the
Statement
array with this structure:{ "Sid": "Allow Enable and Disable of the key", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::551888181234:user/MDBEncAdmin" }, "Action": [ "kms:EnableKey", "kms:DisableKey" ] },
...so that your Key Policy looks like this:{ "Version": "2012-10-17", "Id": "key-consolepolicy-2", "Statement": [ { "Sid": "Allow Enable and Disable of the key", "Effect": "Allow", "Principal": { ...
- Click "Save Changes".
You've now added a new IAM user and you've given that user privileges to enable and disable your key. This user will be able to perform those operations using the AWS CLI or via a script or your own design using the AWS API. For example, using the AWS CLI:
$ cat ~/.aws/credentials [MDBEncAdmin] aws_access_key_id=AKIAJMPPNO7EBKABCDEF aws_secret_access_key=pVdGwbuK5/jG64aBK1oEJOXRlkdM0aAylgabCDef $ AWS_PROFILE=MDBEncAdmin aws --region us-east-1 kms disable-key --key-id arn:aws:kms:us-east-1:551888181234:key/abcdf8f6-084b-4cff-99ca-abcdef6c7907c