Comments - Data-at-Rest Encryption Overview

7 years, 5 months ago Johan Eriksson

Having support for data-at-rest encryption is awesome, but I have to admit I'm a bit puzzled on how to get a viable backup-strategy with encryption in place as xtrabackup wont work with it enabled.

Can someone advice on a good way to implement hot backups with encryption enabled ?

 
7 years, 5 months ago Bryan Alsdorf

We use LVM snapshots for our servers here at MariaDB.

https://mariadb.com/kb/en/mariadb-enterprise/mariadb-enterprise-backup-guide/#lvm-logical-volume-management

The other common option would be replicating to a backup server and where you could perform blocking backups.

 
7 years, 4 months ago Vlad Tokarskiy

Using LVM snapshots leaves the data encrypted which is a good thing. However, what happens if the key(either local or AWS) gets rotated?

How do you restore data on servers not connected to AWS?

 
7 years, 4 months ago Vladislav Vaintroub

This is the question about AWS Key Management plugin, I assume?

If AWS rotates its master key, it does not remove the old one. That means it is still able to decrypt old keys stored in the datadir.

Same is true about local keys. Rotation does not remove the old keys, they are kept in the datadir. They can be deleted when they are not used. But we don't do this. Keys are not used, after Innodb keys rotation completes, and, in case you encrypt binlogs, after binlogs using that key are removed.

There is no automatic way to restore data on servers not connected to AWS.

If you want to do this, you need to do what server does, namely decrypt keys to get plaintext, and for that

a) decrypt the aws-kms.x.y files inside the datadir yourself. You can do it with AWS KMS API called Decrypt, or even using aws-cli for example. You need the plaintext keys for every file

b) write your own aws_key_management plugin that would work with plaintext instead of sending files to AWS

Maybe, you can reuse file_key_management given plaintext keys, if you're not using rotation, but I have not checked that myself.

 
7 years, 4 months ago Kolbe Kegel

The LVM snapshot should be of the entire directory, which will include the encrypted data keys stored locally, so local key rotation is not a concern when restoring from a snapshot. You will still need access to the Customer Master Key in AWS in order to decrypt those data keys. Key Rotation in AWS KMS "currently retains all prior backing keys so that decryption of encrypted data can take place transparently" (see http://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html), so rotating your CMK is not a concern when restoring from a snapshot. If you *delete* your CMK, you will be unable to decrypt the data keys (and thus unable to decrypt your data). Your MariaDB instance must be able to connect to AWS KMS in order to decrypt the data; there is no way to decrypt the encrypted data keys without access to the correct CMK in AWS KMS.

 
7 years, 4 months ago Vlad Tokarskiy

Is recommended way to rotate keys using plugin aws_key_management_rotate_key option and configure appropriate permissions in AWS or setup AWS IAM to rotate the key. In galera cluster would it be better to use AWS IAM to rotate instead of individual servers rotating.

Also, once the key is rotated, will "create table" pick up current key or it must be specified?

Thank you for your feedback

 
7 years, 4 months ago Vladislav Vaintroub

Just enabling annual rotation on AWS does not have any effect until the local keys are rotated.

One approach would be to enable rotation of the key in AWS, and use aws_key_management_rotate_key once a year (so that you're sure that master key has rotated)

Another approach would be to use a different master_key_id for new version - create several master keys in AWS IAM, and change aws_key_management_master_key_id prior to rotating keys with aws_key_management_rotate_key

I'm not sure how all this relates to galera. Whenever new key or key version is created (implicitly, with CREATE/ALTER table if you specify encryption id), or via rotate_key, the new "data" key is created is stored locally in aws-kms-key.x.y . Those keys are not exchanged between servers via replication or galera.

CREATE TABLE always picks up the latest version of the key.

 
7 years, 4 months ago Vlad Tokarskiy

If I understand correctly, the plugin will start rotate keys process and use newly specified master key to create new local key to encrypt. However, since it's pointing to different master key, how does it decrypt previous local key so that tables can be decrypted prior to rotation. If they don't need to be decrypted, how are they decrypted in during database operation since previous local key can not be decrypted with previous master key id.

Does the aws-kms-key contain master key id?

 
7 years, 4 months ago Vladislav Vaintroub

Does the aws-kms-key contain master key id?

We don't record it ourselves, the contents of the file is just cipherblob from GenerateDataKey() response. But AWS knows which keys was used. The question "how does it know" interesting, I'll check the KMS documentation.

 
7 years, 4 months ago Vladislav Vaintroub

Ok, found a (slightly cryptic) explanation here, on slide 18, under 4.

http://www.slideshare.net/AmazonWebServices/encryption-and-key-management-in-aws

The cipherblob we get with GenerateDataKey and store on disk, actually contains encrypted master key id + datakey.

 
7 years, 4 months ago Vladislav Vaintroub

AWS KMS does not need or accept master key id for Decrypt, it only needs it for GenerateDataKey/Encrypt. http://docs.aws.amazon.com/kms/latest/APIReference/API_Decrypt.html

As you can see, Decrypt actually *returns* the master-key-id that was used for GenerateDataKey its response (how does it know it, I have no idea, never checked), in addition to Plaintext that we use.

Thus: - rotate does KMS-GenerateDataKey., It *needs* master key id. ditto CREATE/ALTER TABLE with new key ids, or - decrypt(at the startup time) does KMS-Decrypt(). It *does not need* master-key-id. It can decrypt multiple local keys, backed by different CMK, or by single CMK that was transparently rotated by AWS.

 
7 years, 4 months ago Kolbe Kegel

When a key is rotated in AWS KMS, it keeps the same key alias and earlier versions of the key are still available for decryption. Take a look at http://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html for more information. Local key rotation, done by MariaDB Server, causes the new version of the key to be written to a file in your datadir. Old versions of the key are kept as well, since they'll be needed to decrypt tables/pages/logs encrypted using earlier versions of the key. When you're certain that an older version of a key is no longer needed, you can delete the corresponding file from the datadir.

 
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.