API Series Part 4 - Secrets Management with Docker Secrets

Working with secrets correctly in todays security environment is crucial. Storing database passwords and API keys in plain text files or in the source control system is just plain reckless. There are a few good options out there and today we are going to look at Docker Secrets.

What I really like about Docker Secrets are that it is so simple to use. Complexity is the enemy of security and with Docker Secrets being so simple it is hard to mess it up.

Docker Secrets only work in Swarm Mode so take that into account before choosing Docker Secrets as your secrets management mechanism. In our case, in this series the plan is to use Swarm Mode as our container orchestrator so Docker Secrets is a great fit.

Secrets Management Requirements

Ideally we want any secrets management technology to comply with the following requirements:

  • Encryption in transit
  • Encryption at rest
  • Access Control (choose who we give access to each secret)
  • Secret Rotation
  • Revocation

Docker Secrets fills all of these needs, though arguably revocation could be disputed. Let's look at each requirement.

Encryption in Transit

Docker Secrets are encrypted in transit via a TLS connection with the Swarm Manager node.

Encryption at Rest

Secrets are stored in the Raft log which is encrypted at all times. While on the file system, the secret remains encrypted. When we grant a service access to a secret, Docker mounts the decrypted secret into the service containers in an in-memory file system. The service can load the secret as if it were a file from the location /run/secrets/<secret-name>. So the only time secrets are decrypted are when they are loaded into the memory of a container.

Access Control

Access to secrets must be granted to the services that require it - a whitelist approach. If they are not granted access then the secret is not mounted into the containers of that service and so there is no way for them to gain access to the secret. Likewise access can be removed and the secret gets unmounted.

We can choose which services get access to which secrets in our docker-compose.yml file.

Secret Rotation

Due to security policy or regulatory compliance, credentials and other secrets may need to be rotated regularly. Docker Secrets does not provide an out-of-the-box rotation funcionality and in fact secrets are immutable - they can be created and removed only. You cannot remove a secret that is being used by any services. So in order to remove a secret you must revoke access to the secret by all services and then remove it.

Secret rotation can be achieved with the use of environment variables to indicate the path to a secret. That way you can create a new version of a secret with a new name and update the environment variable to point to the path of the new secret. The new secret now gets used and so access to the original secret can be revoked and then the secret itself can be removed from the raft log. Note that environment variables at no point store the secret itself, they simply store the location.

Revocation

We need to be clear what this means. We can revoke access to a secret or we could revoke a secret itself. Docker Secrets can revoke access to a secret which means that the secret is expunged from the memory of the container. But if the secret is the credentials to a database then Docker has no way of revoking those credentials themselves. Hashicorp Vault does have that capability so we will be looking at Vault very soon. 

We'll be adding Docker secrets in two steps:

  • Create a Docker Swarm in Swarm Mode
  • Add Docker Secrets

If you are already well versed in creating swarms then jump straight to Adding Secrets to Our Swarm.

Create a Docker Swarm in Swarm Mode

In the last post we added VS 2017 Docker support to Govrnanza, our ASP.NET Core 2.0 API we are building in this series. Visual Studio does not use Swarm Mode, it simply uses the docker compose up command to stand up a set of services. But in order to use Docker Secrets we need Swarm Mode and we'll be using Swarm Mode when we deploy Govrnanza to an environment.

So VS can't help us with this one, we're going to need to run some Docker commands ourselves. First of all, during development I would like to use my own local Docker image repository.

Step 1 - Create our own local image repository

Docker provides a containerized Docker image repository with the name:tag of registry:2

docker run -d -p 5000:5000 --name registry registry:2

We can now build an image and push it to our local repository then reference that image from a new docker-compose.yml file that we'll create soon.

Step 2 - Make the MobyLinuxVM a Swarm Manager

You could use docker-machine to create a separate VM and create the Swarm there. That way it won't interfer with running the application via Visual Studio, which does not support Swarm Mode. However, if you regularly use a VPN like me, this option is out as docker-machine does not play nice with VPNs and vice versa. So I will be working with the main Docker for Windows VM instead. But if you want to see how to create separate VMs to host your Swarm then the Docker Swarms tutorial Part 4 really nicely explains how to do that on Docker for Windows.

Let's make the swarm manager. All commands to work with secrets are handled by manager nodes.

docker swarm init
Swarm initialized: current node (ano7kbd6pvh653i4ywd8zvtd2) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-408garbbuqsk9y66601qb3h7b90tf506s2j5ywll1jpxibd4zf-3oid84463w70wb5322mor6teb 192.168.65.2:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

We can stop the main Docker for Windows VM from being the manager with the docker swarm leave command later. We are not going to create any worker nodes as we are just creating a swarm for development purposes and one VM is enough. So we now have a manager node that we can create a swarm on.

Step 3 - Create a container image and push it to our local repository

Navigate to the Govrnanza.Registry.WebApi directory and run dotnet publish.

cd C:\Users\Jack\Github\Govrnanza\src\Registry\Govrnanza.Registry.WebApi
dotnet publish -c Release -o ./obj/Docker/publish

Create a Docker image

docker build -t govrnanza.registry.webapi .

Tag the image. Note that localhost:5000 is our local image repository.

docker tag govrnanza.registry.webapi localhost:5000/govrnanza.registry.webapi:dev

Push the tagged image to the repository

docker push localhost:5000/govrnanza.registry.webapi:dev

Step 4 - Create a new docker-compose.yml

Our new docker-compose.yml has some similarities to that of our Visual Studio one but you'll see that the image points to the one we just pushed to our local repo. We also have a deploy section where we state that we want a single copy of the service and that it should automatically restart on failure.

version: '3.1'

services:
  registry:
    image: localhost:5000/govrnanza.registry.webapi:dev
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - "80:80"
    networks:
      - web

networks:
  web:

This docker-compose.yml can be saved anywhere, as long as we have access to the image repo we can use it to stand up a swarm.

Step 5 - Stand up the swarm

We use the docker stack deploy command to stand up a swarm (aka a stack) in swarm mode. From the same directory where the docker-compose.yml file is saved, we run the following command:

docker stack deploy -c docker-compose.yml govrnanza
Creating network govrnanza_web
Creating service govrnanza_registry

That's it, the swarm is up and running and is called govrnanza. We can see that by running some commands:

List the swarms aka stacks

docker stack ls
NAME                SERVICES
govrnanza           1

List the services inside the govrnanza stack

docker stack ps govrnanza
ID                  NAME                   IMAGE                                          NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
pnuk45s2h085        govrnanza_registry.1   localhost:5000/govrnanza.registry.webapi:dev   moby                Running             Running 13 seconds ago

Now we can access the application at http://localhost/api-docs

Adding Secrets to Our Swarm

Secrets can be added and removed by either docker secret commands or by adding them to the docker-compose.yml. Likewise, services can be granted access to secrets by either commands or the docker-compose.yml.

It's a no brainer that using the docker-compose.yml for granting access to secrets is the way to go. We clearly see what services have access to what secrets in an easy to understand declarative document.

Adding secrets via the docker-compose.yml is more problematic for me as it assumes you have the decrypted secrets in local files, which sounds pretty scary. An alternative is to create secrets via commands, which gives us much more flexibility regarding how we securely source our secrets for docker. So that is the approach we'll take.

Step 1 - Create the secrets

In our case we only need to store the database user password. We use the docker secret create command with the - parameter to signify user input, then we enter the password. Obviously, there are better more automated ways of doing this.

docker secret create registry_db_password -
dkf72jhdfv40g4nf7b

List the current secrets

docker secret ls
ID                          NAME                   CREATED             UPDATED
riwhew55srt173zoc7hiyxszb   registry_db_password   4 seconds ago       4 seconds ago

Grant Access via docker-compose.yml

Notice the following new elements in our docker-compose.yml

  • a root element called secrets with a secret called registry_db_password which is declared as external. It is exteral because it was created outside of this docker-compose.yml, but exists in the swarm.
  • a new secrets element under the registry service, which lists our new registry_db_password secret. This grants this service access to this secret.
  • a new environment variable which stores the path to our new secret. By storing the path to the secret in an environment variable we can rotate the secret later and update the path and the application will need to be updated.
version: '3.1'

services:
  registry:
    image: localhost:5000/govrnanza.registry.webapi:dev
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - REGISTRY_DB_PASSWORD_SECRET_FILE=/run/secrets/registry_db_password
    ports:
      - "80:80"
    networks:
      - web
    secrets:
       - registry_db_password

secrets:
   registry_db_password:
     external: true

networks:
  web:

Change Govrnanza to Get the Password from the Secret

How you want to integrate the secret into your app is up to you. I quite like embedding secrets in other configuration values using the format {secret:<secret name>}.

So I changed the connection string in the appsettings.json as follows:

{
  "ConnectionStrings": {
    "Registry": "Server=192.168.76.4,1433;Database=Govrnanza_Registry;User Id=RegistryApp;Password={secret:REGISTRY_DB_PASSWORD};MultipleActiveResultSets=True"
  }
}

Now when we load the connection string I pass it through a class that loads the secret and embeds it in the connection string.

public class SecretsResolver
{
    public static string ResolveEmbeddedSecret(string inputText)
    {
        var secretsResults = Regex.Matches(inputText, @"(?<secret>\{secret:[\s\w-\.]+\})");

        foreach (Match secretTagResult in secretsResults)
        {
            var groupText = secretTagResult.Groups[0].Value;
            var secretId = groupText.Substring(8, groupText.Length - 9).Trim();

            var pathToSecret = Environment.GetEnvironmentVariable(secretId + "_SECRET_FILE");

            if (!File.Exists(pathToSecret))
            {
                throw new Exception($"SecretId: {secretId} does not exist at " + pathToSecret + " or this service does not have access to it");
            }

            var storedSecret = File.ReadAllText(pathToSecret);
            inputText = inputText.Replace(groupText, storedSecret);
        }

        return inputText;
    }
}
public void ConfigureServices(IServiceCollection services)
{
    services.AddDbContext<RegistryDbContext>(opt =>
            opt.UseSqlServer(SecretsResolver.ResolveEmbeddedSecret(Configuration.GetConnectionString("Registry"))));
    ...
}

So, in summary we do the following:

  1. load the connection string from the appsettings.json
  2. take the secret name from the connection string which is the name of our environment variable
  3. get the path to the secret from the environment variable
  4. load the secret using the path, as if it were a file in the file system
  5. embed the secret in the connection string

There are libraries that add docker secrets integration to the ASP.NET Core configuration abstraction also, so that you can work with secrets like there were any other configuration value.

Now we need to go through the same publish, build, tag and push commands like we did before, followed by the same docker stack deploy. Then load http://localhost/api-docs and execute one of the operations to see that it accesses the database correctly.

Rotating the secret

Let's change the password for our RegistryApp user in the database. Now when you execute an operation it fails. We can update the secret without needing to change the application code as we always load the path to the secret via the same environment variable.

Create a new secret for the new password:

docker secret create registry_db_passwordV2 -
kag36g9ndv38fgkd

List the secrets

docker secret ls
ID                          NAME                     CREATED             UPDATED
riwhew55srt173zoc7hiyxszb   registry_db_password     About an hour ago   About an hour ago
u1po4rw5xtfm6dm2av4f2pu6x   registry_db_passwordV2   6 seconds ago       6 seconds ago

Now we replace the registry_db_password the new registry_db_passwordV2 version in the docker-compose.yml and change the REGISTRY_DB_PASSWORD environment variable to point to the path of the v2 version.

version: '3.1'

services:
  registry:
    image: localhost:5000/govrnanza.registry.webapi:dev
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - REGISTRY_DB_PASSWORD_SECRET_FILE=/run/secrets/registry_db_passwordV2
    ports:
      - "80:80"
    networks:
      - web
    secrets:
       - registry_db_passwordV2

secrets:
   registry_db_passwordV2:
     external: true

networks:
  web:

Then we redeploy with the docker stack deploy command.

docker stack deploy -c docker-compose-with-secret.yml govrnanza
Updating service govrnanza_registry (id: fh3tupqbe30avkq8bt8i8gs8o)

Finally we can now remove the original secret:

docker secret rm registry_db_password

docker secret ls
ID                          NAME                     CREATED             UPDATED
u1po4rw5xtfm6dm2av4f2pu6x   registry_db_passwordV2   6 minutes ago       6 minutes ago

Conclusions

Docker secrets are a great option for securely managing secrets because:

  • It is such a simple system so it is hard to make mistakes
  • Secrets exist in each container, so we don't have to worry about high availability (HA). They are local and so always available.

The drawbacks are that you only get them in Swarm Mode which means that we need an alternative when running our application from Visual Studio. Remember that before running the application from Visual Studio again we need to execute docker swarm leave. If you can use docker-machine to create a local VM dedicated to your Swarm Mode swarms then all the better.

Finally Docker Secrets are oriented towards being an operational store for secrets rather than a master secrets store. For that we can look to technology like Hashicorp Vault.