Hashicorp Key Management

The Hashicorp Key Management Pugin is used to implement encryption using keys stored in the Hashicorp Vault KMS.

For more information about configuring the plugin as well as different capabilities, please check the documentation. This guide will cover a minimal example for configuring the plugin with the operator.

Configuring TDE in MariaDB Using Hashicorp Key Management Plugin

Transparent Data Encryption (TDE) can be configured in MariaDB leveraging the Hashicorp Key Management Plugin.

Requirements

Running and accessible Vault KMS setup with a valid SSL certificate.
Vault is unsealed and you've logged in to it with vault login $AUTH_TOKEN, where $AUTH_TOKEN is an authentication token given to you by an administrator
openssl for generating secrets

Steps

Creating A New Key-Value Store In Vault. Create a new key-value store and take note of the path. In our example we will use mariadb.
```
vault secrets enable -path /mariadb -version=2 kv
```
Adding necessary secrets. We will put 2 secrets with ids 1 and 2. 2 will be used for temporary files, while 1 will be used for everything else. It is not neccessary to create 2 of them and in that case, temporary files will use 1.
Note: Here you should use the path we chose in the previous step.
```
vault kv put /mariadb/1 data="$(openssl rand -hex 32)"
vault kv put /mariadb/2 data="$(openssl rand -hex 32)"
```

(Optional) Create An Authentication Token With Policy. This step can be skipped if you want to use your own token. Consult with a Vault administrator regarding this. Policies are Vault's way to restrict access to what you are allowed to do. The following is a policy that should be used by the token following the least permission principle.

cat <<'EOF' | vault policy write -non-interactive mariadb -
# Allow access to MariaDB secrets
path "mariadb/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Allow reading the mount configuration
path "sys/mounts/mariadb/tune" {
  capabilities = ["read"]
}
EOF

After which, we can create a new token with the given policy.

vault token create -policy mariadb

You will see output similar to:

Key                  Value
---                  -----
token                EXAMPLE_TOKEN
token_accessor       utFtmh98YAAJyYdxEVN3SFQA
token_duration       768h
token_renewable      true
token_policies       ["default" "mariadb"]
identity_policies    []
policies             ["default" "mariadb"]

Your new token is: EXAMPLE_TOKEN.

Create A Secret For the vault token. Now that you've either created a new token, or are using an existing one, we need to create a secret with it.
```
export TOKEN="EXAMPLE_TOKEN"
kubeclt create secret generic mariadb-vault-token --from-literal=token="$TOKEN"
```
Create a Secret for the Certificate Authority (CA) used to issue the Vault certificate. For further information, consult the docs If you have the certificate locally in a file called ca.crt you can run:
```
kubectl create secret generic vault-tls --from-file=./ca.crt
```

Create A MariaDB Custom Resource. The final step is creating a new MariaDB instance.

mariadb-vault.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: mariadb # Used to hold the mariadb and root user passwords
  labels:
    enterprise.mariadb.com/watch: ""
stringData:
  password: MariaDB11!
  root-password: MariaDB11!
---
apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  image: docker.mariadb.com/enterprise-server:11.4.7-4.3
  rootPasswordSecretKeyRef:
    name: mariadb
    key: password

  username: mariadb
  passwordSecretKeyRef:
    name: mariadb-password
    key: password
    generate: true
  database: mariadb

  port: 3306

  storage:
    size: 1Gi
    # storageClassName: csi-hostpath-sc

  myCnf: |
    [mariadb]
    bind-address=*
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    innodb_buffer_pool_size=800M
    max_allowed_packet=256M

    plugin_load_add = hashicorp_key_management
    hashicorp-key-management-vault-url=https://vault-0.vault-internal.default.svc.cluster.local:8200/v1/mariadb
    hashicorp-key-management-caching-enabled=ON
    hashicorp-key-management-vault-ca=/etc/vault/certs/ca.crt

    innodb_encrypt_tables = FORCE
    innodb_encrypt_log = ON
    innodb_encrypt_temporary_tables = ON
    encrypt_tmp_disk_tables = ON
    encrypt_tmp_files = ON
    encrypt_binlog = ON
    aria_encrypt_tables = ON

    innodb_encryption_threads = 4
    innodb_encryption_rotation_iops = 2000

  env:
    - name: VAULT_TOKEN # This is where our token is defined!
      valueFrom:
        secretKeyRef:
          name: mariadb-vault-token
          key: token

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      memory: 1Gi

  metrics:
    enabled: true

  volumes:
    - name: vault-certificates
      secret:
        secretName: vault-tls
        defaultMode: 0600
  volumeMounts:
    - name: vault-certificates
      mountPath: /etc/vault/certs/

kubectl apply -f mariadb-vault.yaml

Verify Encryption Works.

kubectl run mariadb-connect --rm -it --image=mariadb:11.4 -- bash -c "mariadb -u root -p'MariaDB11!' --ssl=false -h mariadb"

You should see something along the lines of:

If you don't see a command prompt, try pressing enter.
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 95
Server version: 11.4.7-4-MariaDB-enterprise MariaDB Enterprise Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

At this point, you can check the encryption status:

SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION;

If you create a new database and then table, the above query should return additional information about them. Something like:

MariaDB [my_db]> SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION;
+-----------------+-------------------+-----------------+---------------------+----------------+----------------------+
| NAME            | ENCRYPTION_SCHEME | MIN_KEY_VERSION | CURRENT_KEY_VERSION | CURRENT_KEY_ID | ROTATING_OR_FLUSHING |
+-----------------+-------------------+-----------------+---------------------+----------------+----------------------+
| innodb_system   |                 1 |               1 |                   1 |              1 |                    0 |
| innodb_undo001  |                 1 |               1 |                   1 |              1 |                    0 |
| innodb_undo002  |                 1 |               1 |                   1 |              1 |                    0 |
| innodb_undo003  |                 1 |               1 |                   1 |              1 |                    0 |
| mysql/innodb_ta |                 1 |               1 |                   1 |              1 |                    0 |
| mysql/innodb_in |                 1 |               1 |                   1 |              1 |                    0 |
| mysql/gtid_slav |                 1 |               1 |                   1 |              1 |                    0 |
| mysql/transacti |                 1 |               1 |                   1 |              1 |                    0 |
| my_db/people    |                 1 |               1 |                   1 |              1 |                    0 |
+-----------------+-------------------+-----------------+---------------------+----------------+----------------------+

Note: The above query is truncated. In reality, you will see a few more columns.

Day-2 Operations

Rotating Secrets

Put A New Secret In Vault. After logging in to vault, you can run again:

vault kv put /mariadb/1 data="$(openssl rand -hex 32)"
vault kv put /mariadb/2 data="$(openssl rand -hex 32)"

This will start re-encrypting data.

Monitor Re-Encryption.

kubectl run mariadb-connect --rm -it --image=mariadb:11.4 -- bash -c "mariadb -u root -p'MariaDB11!' --ssl=false -h mariadb"

If you check the encrpytion status again:

SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION;

You should see CURRENT_KEY_VERSION column start getting updated to point to the new key version.

MariaDB [my_db]> SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION;
+-----------------+-------------------+-----------------+---------------------+----------------+----------------------+
| NAME            | ENCRYPTION_SCHEME | MIN_KEY_VERSION | CURRENT_KEY_VERSION | CURRENT_KEY_ID | ROTATING_OR_FLUSHING |
+-----------------+-------------------+-----------------+---------------------+----------------+----------------------+
| innodb_system   |                 1 |               1 |                   2 |              1 |                    0 |
| innodb_undo001  |                 1 |               1 |                   2 |              1 |                    0 |
| innodb_undo002  |                 1 |               1 |                   2 |              1 |                    0 |
| innodb_undo003  |                 1 |               1 |                   2 |              1 |                    0 |
| mysql/innodb_ta |                 1 |               1 |                   2 |              1 |                    0 |
| mysql/innodb_in |                 1 |               1 |                   2 |              1 |                    0 |
| mysql/gtid_slav |                 1 |               1 |                   2 |              1 |                    0 |
| mysql/transacti |                 1 |               1 |                   2 |              1 |                    0 |
| my_db/people    |                 1 |               1 |                   2 |              1 |                    0 |
+-----------------+-------------------+-----------------+---------------------+----------------+----------------------+

Rotating Token

Make sure when rotating the token, to do so in advance of the token expiring.

Acquire a new token and update the secret.

export TOKEN="EXAMPLE_TOKEN"
kubeclt create secret generic mariadb-vault-token --from-literal=token="$TOKEN"

Restart The MariaDB StatefulSet. MariaDB will continue using the old token until the StatefulSet is restarted. If the name of the MariaDB CR is mariadb (as per the example above), then:
```
kubectl rollout restart statefulset mariadb
```

Known Issues/Limitations

Vault Not Being Accessible Will Result In MariaDB Not Working

As MariaDB uses Vault to fetch it's decryption key, in case that Vault becomes unavailable, it will result in MariaDB not being able to fetch the decryption key and hence stop working. While the Hashicorp plugin has a configurable cache, that should be set and will result in MariaDB still working for a few seconds to minutes, depending on configuration, the cache is not reliable as it's ephemeral and short lived.

Deleting The Decryption Key Will Make Your Data Inaccessible.

It is recommended to back up the decryption key so accidental deletions will not result in issues.

Decryption Key Must Be Hexadecimal

Use the following to generate correct decryption keys.

openssl rand -hex 32

Rotating The Decryption Key Before A Previous Re-Encryption Has Finished, Will Result In Data Corruption.

To check the re-encryption progress, you can run:

SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION;

Look for the CURRENT_KEY_VERSION and make sure they are in sync with the latest version you have in Vault.

PreviousPAM NextAPI Reference

Last updated 2 days ago

Was this helpful?