Skip to main content

Database Cert Authority Rotation

Teleport connects to self-hosted databases using mutual TLS, or mTLS for short. To enable mTLS connections to your self-hosted database via Teleport, you have to configure your database and Teleport so that they can complete an mTLS handshake: essentially, they need to trust each other. To that end, your Teleport cluster manages its own internal database certificate authorities. This guide will explain how to rotate your Teleport cluster's database CAs.

Prerequisites​

  • A running Teleport cluster version 15.4.22 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl admin tool and tsh client tool.

    On Teleport Enterprise, you must use the Enterprise version of tctl, which you can download from your Teleport account workspace. Otherwise, visit Installation for instructions on downloading tctl and tsh for Teleport Community Edition.

  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example: 
    $ tsh login --proxy=teleport.example.com --user=email@example.com
    $ tctl status
    # Cluster  teleport.example.com
    # Version  15.4.22
    # CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Database CA rotation​

Teleport uses two certificate authorities for self-hosted database access:

  • The db certificate authority is used to issue server certificates to self-hosted databases.
  • The db_client certificate authority is used to issue an ephemeral client certificate to the Teleport Database Service when it connects to a self-hosted database.

Certificate authority rotation consists of several rotation phases. The rotation phases have different effects depending on which CA is rotated. This section will explain the differences and show you how to rotate your database CAs.

db CA Rotation phases​

  • standby: All operations have completed or haven't started yet.
  • init: All Teleport cluster services are notified of the rotation. A new db certificate authority is created in your cluster. Remote trusted clusters must fetch the new CA. tctl auth sign will issue database server certs with the new CA.
  • update_clients: Ephemeral client certificates issued to the Teleport Proxy Service will be issued by the new CA.
  • update_servers: No effect during db CA rotation.
  • rollback: The rotation was aborted and is rolling back to the old certificate authority.

db_client CA Rotation phases​

  • standby: All operations have completed or haven't started yet.
  • init: All Teleport cluster services are notified of the rotation. A new db_client certificate authority is created in your cluster. tctl auth sign will output both the old and new certificate authorities in its trusted CA output.
  • update_clients: Ephemeral client certificates issued to the Teleport Database Service will be issued by the new CA.
  • update_servers: No effect during db_client CA rotation.
  • rollback: The rotation was aborted and is rolling back to the old certificate authority.

Rotation types​

There are two types of certificate authority rotation processes:

  • Manual: it is the cluster administrator's responsibility to transition between each phase of the rotation while monitoring database connectivity. Manual rotations are performed by providing the --manual flag and the desired phase using the --phase flag to the tctl auth rotate command.
  • Semi-automatic: Teleport automatically transitions between phases of the rotation after some amount of time (known as a grace period) elapses.

For both types of rotation, the cluster goes through the phases in the following order:

  • standby -> init -> update_clients -> update_servers -> standby

It does not make sense to use semi-automatic rotation when rotating the db or db_client CA, because self-hosted databases are not automatically reconfigured by the semi-automatic process. Therefore this guide will only explain manual rotation.

In manual mode we transition between phases while monitoring database connectivity, and reconfigure databases with new CA certificates as needed.

You will need to reconfigure any given self-hosted database only once during the rotation: it does not need to be done at each step. You can reconfigure your databases during any phase, although it's best to do so during the init phase to avoid losing database access later in the rotation.

If you lose connectivity to your databases during any phase prior to standby, you can roll back to the old certificate authority to restore connectivity.

Step 1/4. Start the rotation​

Initiate manual rotation of the db or db_client certificate authority:

Want to rotate both CAs?

You should not rotate the db or db_client CA in parallel with other CA rotations. Rotating in parallel with other CA types, especially the host CA, is more complicated, more likely to cause problems, and does not save you any effort.

However, if you want to rotate both the db and db_client CA, then you can (and should) save yourself some effort by rotating both of them at the same time.

You can rotate both db and db_client certificate authorities in parallel by transitioning both CAs through each phase step. Just use the tctl auth rotate command for each step with --type=db and then --type=db-client.

$ tctl auth rotate --manual --type=type --phase=init 
Updated rotation phase to "init". To check status use 'tctl status'

Use tctl to confirm that there is an active rotation in progress:

$ tctl status
Avoid losing access to databases

If you are rotating the db_client CA and you do not want to lose access to your self-hosted databases in the next phase, you should reconfigure your databases now. You do not need to reconfigure databases at this point if you are rotating only the db CA, although there is no harm in doing so.

Consult the appropriate Teleport Database Access Guide for your databases before proceeding to the update_clients rotation phase.

Step 2/4. Update clients​

Execute the transition from init to update_clients:

$ tctl auth rotate --manual --type=type --phase=update_clients 
# Updated rotation phase to "update_clients". To check status use 'tctl status'

If you are rotating the db_client CA, this is the phase where the Teleport Database Service will start using client certificates issued by the new CA to connect to databases. You should verify that you can still access your databases after transitioning to this phase.

If you lose connectivity to your databases, you can reconfigure your databases to restore access. If that does not restore access or you are unable to reconfigure a database, then roll back to the old certificate authority.

Step 3/4. Update servers​

Execute the transition from update_clients to update_servers:

$ tctl auth rotate --manual --type=type --phase=update_servers 
# Updated rotation phase to "update_servers". To check status use 'tctl status'

This phase does not actually do anything for db or db_client CA rotation, but it's required as part of the generalized Teleport CA rotation process before we can finish the rotation.

Step 4/4. Finish the rotation​

If any of your self-hosted databases have not yet been reconfigured, you will lose access to those databases after transitioning to the standby phase in this final step.

To avoid down time, consult the appropriate Teleport Database Access Guide and reconfigure your databases before proceeding. Otherwise, access may still be restored by reconfiguring your self-hosted databases after this step.

Execute the transition from update_servers to standby:

$ tctl auth rotate --manual --type=type --phase=standby
# Updated rotation phase to "standby". To check status use 'tctl status'

Rollback​

Rollback can only be performed before the rotation is completed, meaning before the rotation transitions to standby from update_servers.

The most common reason you would want to rollback if is if you cannot reconfigure your databases. If you have connectivity issues after reconfiguring a database, it's likely that you simply misconfigured the database. If you roll back the rotation, then you will have to manually revert any database reconfigurations and start from step 1 to reattempt rotating your certificate authorities, so consider this a method of last resort.

First, enter the rollback phase with a manual phase transition:

$ tctl auth rotate --manual --type=type --phase=rollback
# Updated rotation phase to "rollback". To check status use 'tctl status'

If you reconfigured any of your databases during the rotation, you will need to reconfigure them again before transitioning to standby from the rollback phase.

Enter the standby phase manually:

$ tctl auth rotate --manual --type=type --phase=standby
# Updated rotation phase to "standby". To check status use 'tctl status'

It is very unlikely that you will lose database connectivity during a db CA rotation and need to rollback. However, when rotating your db_client CA you will definitely lose access to any database that was not reconfigured during the init rotation phase. If you are unable to reconfigure those databases, then rolling back should restore access to them immediately.

Further reading​