API Series Part 5 - Configuration with Docker Configs

All code described in this post can be found in Github

In the last post we stored the database user password securely as a Docker Secret. In this post we'll look at Docker Configs which uses the same model as the secrets except that files are mounted into the container file system unencrypted.

Why use Docker Configs instead of the other alternatives? Let's quickly compare Docker Configs with other options.

Docker Configs vs Local Config Files

Both of the options have the benefit that the files are accessible locally on the file system, which means that we don't have to worry about flaky networks and highly available configuration services.

Where Docker Configs are superior to a local appsettings.json is the ease of deployment and ease of sharing common configuration files. With an appsettings.json, whenever it changes you must build a new docker image, push it to your repo and deploy the new version of your service to your swarm. With Docker Config you just need to update your docker-compose.yml and run docker stack deploy again. We decouple the local configuration files from the images.

If you have common configurations, such as logging configurations, with Docker Configs you can register the config once in the swarm and grant every service access to that single configuration file. That file then gets mounted into the file system of each container of the services which have access granted.

Docker Configs vs Consul

I use Consul as an example of a technology that can be used as a configuration provider over HTTP. Consul has its KV store and provides High Availability (HA).

With Consul you can change your configuration in a central location and it get propagated to all your ASP.NET Core applications automatically. You need to build your application to check for changes periodically and then load any changes dynamically to enable that. There is a configuration provider for Consul that can reload configuration, called Winton.Extensions.Configuration.Consul.

You can update your services with Docker Config but configs are immutable so we have to perform the same rotation method as with the secrets. Using environment variables and file rotation. BUT, like with secrets, docker will shutdown the existing services and start up new ones. To avoid service disruption you can stagger the update to ensure you always have services running and accepting requests.

So while Consul can make configuration changes propagate more easily for services that can reload their configuration dynamically, it still has the same weakness being that you must access your configuration over HTTP, while Docker Configs store the configuration locally.

Finally Docker Configs are simpler to use and operate than Consul. If you are using Swarm Mode then you probably don't need Consul or another service discovery mechanism given that Swarm Mode has service discovery built-in. So adding it just for configuration just adds unnecessaary complexity.

Integrating Docker Configs into an ASP.NET Core 2.0 application

Just like with Docker Secrets, the Configs are only available in Swarm Mode. Which means when we are debugging our application using the Docker support of Visual Studio 2017, we can't use it. So we need a solution that uses local files when we run our Web API from Visual Studio and uses Docker Configs when we stand up the Swarm.

Accessing configuration and secrets files from VS and Swarms

In the last post we embedded secrets tags in the connection string in the appsettings.json and then loaded that secret from Docker, for example:

"ConnectionStrings": {
    "Registry": "Server=192.168.1.35,1433;Database=Govrnanza_Registry;User Id=RegistryApp;Password={secret:REGISTRY_DB_PASSWORD};MultipleActiveResultSets=True"
  },

We are going to create a configuration provider for ASP.NET Core that can do the following:

  • Load configuration from Docker Configs when in Swarm Mode
  • Load configuration from a local file when run from VS
  • Detect embedded secrets in our configuration and load them from Docker Secrets when in Swarm Mode
  • Detect embedded secrets in our configuration and load them from local files when in run from VS

To achieve different behaviour in VS or a swarm we use environment variables to control the behaviour. 

We add REGISTRY_CONFIG_FILE to point the application to the right configuration file. Whether it is a config file in our VS project or a Docker Config file, they are just paths to a file on the file system. So from the docker-compose.yml we can control where the application sources its configuration.

Likewise for secrets, we use environment variables to point the application to where the secrets are stored. When running in VS we store the unencrypted secrets in a local config file - one secret per file just like Docker Secrets. You could put these files in VS or outside. If we don't want credentials to our dev environment in source control, we simply put those files outside of the application, but accessible locally. Additionally we add a check that when in production, we can only load secrets from Docker.

Our docker-compose.override.yml in our Docker support files, points us to an appsetting.json file in our project and a txt file that contains our dev environment db password.

version: '3'

services:
  govrnanza.registry.webapi:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - REGISTRY_CONFIG_FILE=devAppSettings.json
      - REGISTRY_DB_PASSWORD_SECRET_FILE=InsecureSecretFiles/RegistryDbPassword.txt
    ports:
      - "80"

The docker-compose.yml we use to stand up our swarm in Swarm Mode points the application to the Docker Config and Docker Secret.

Note that in the last post I loaded the secret via a command and referenced the secret from the docker-compose.yml as an external secret. For the purposes of simplicity during development I have switched to loading the secret from a locally unencrpyted file. This is obviously not a good idea when dealing with real credentials.

version: '3.3'

services:
  registry:
    image: localhost:5000/govrnanza.registry.webapi:dev
    deploy:
      mode: replicated
      replicas: 2
      restart_policy:
        condition: on-failure
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - REGISTRY_CONFIG_FILE=/registry-config
      - REGISTRY_DB_PASSWORD_SECRET_FILE=/run/secrets/registry_db_password
    ports:
      - "31000:80"
    networks:
      - web
    secrets:
      - registry_db_password
    configs:
      - registry-config

secrets:
  registry_db_password:
    file: ./secrets/registryDbPassword.txt

configs:
  registry-config:
    file: ./configs/registryAppSettings.json

networks:
  web:

When we run docker stack deploy, Docker takes the registryAppSettings.json and mounts it into each container of the registry service. The path to this file from inside the container is /registry-config (no file extension).

In the configs section you see the path ./configs/registryAppSettings.json, that is the path relative to the docker-compose.yml.

Creating Our Own Configuration Provider

We are now ready to write code to read from the environment variables and load configuration and secrets from the locations specified by the environment variables.

To create a configuration provider we'll create a new project called Govrnanza.Extensions.Configuration and implement a Microsoft.Extensions.Configuration provider.

We need the following files:

  • Implementation of IConfigurationSource. This has a single method Build that returns an IConfigurationProvider.
  • Sub class of ConfigurationProvider which implements IConfigurationProvider. This is the class that loads the configuration.
  • JSON file parser. Converts a JSON file into an IDictionary<string,string>
  • Secrets resolver like from the last post. Detects secret tags in config values and replaces them with secrets.
  • Extensions class that adds an extension method for IConfigurationBuilder for adding our provider

Extensions Class

We can pass one or more environment variables which point to configuration files that should be loaded.

namespace Govrnanza.Extensions.Configuration
{
    public static class ConfigurationBuilderExtensions
    {
        public static IConfigurationBuilder AddGovrnanzaConfig(this IConfigurationBuilder builder, SecretsMode secretsMode, params string[] pathEnvironmentVariables)
        {
            var settingsConfigSource = new GovrnanzaConfigurationSource(secretsMode, pathEnvironmentVariables.ToList());
            return builder.Add(settingsConfigSource);
        }
    }
}

IConfigurationSource implementation.

namespace Govrnanza.Extensions.Configuration
{
    public class GovrnanzaConfigurationSource : IConfigurationSource
    {
        private SecretsMode _secretsMode;
        private List<string> _pathEnvironmentVariables;

        public GovrnanzaConfigurationSource(SecretsMode secretsMode,
            List<string> pathEnvironmentVariables)
        {
            _secretsMode = secretsMode;
            _pathEnvironmentVariables = pathEnvironmentVariables;
        }

        public IConfigurationProvider Build(IConfigurationBuilder builder)
        {
            return new GovrnanzaConfigurationProvider(
                _secretsMode,
                _pathEnvironmentVariables);
        }
    }
}

Sub class of ConfigurationProvider

Our provider loads a JSON file from the file path indicated by each environment variable passed to it. It then parses the JSON, using the SecretsResolver to replace any secret tags with the secret values. I took the JSON parser directly from the Microsoft.Extensions.Configuration project and modified it to use the SecretsResolver.

namespace Govrnanza.Extensions.Configuration
{
    public class GovrnanzaConfigurationProvider : ConfigurationProvider
    {
        private SecretsMode _secretsMode;
        private List<string> _pathEnvironmentVariables;

        public GovrnanzaConfigurationProvider(SecretsMode secretsMode,
            List<string> pathEnvironmentVariables)
        {
            _secretsMode = secretsMode;
            _pathEnvironmentVariables = pathEnvironmentVariables;
        }

        public override void Load()
        {
            try
            {
                foreach (var pathEnvVariable in _pathEnvironmentVariables)
                {
                    var configPath = Environment.GetEnvironmentVariable(pathEnvVariable);
                    if (configPath != null)
                    {
                        var jsonBytes = LoadJsonFile(configPath);

                        var secretsResolver = new SecretsResolver(_secretsMode);
                        var parser = new JsonConfigurationFileParser(secretsResolver);
                        var configDict = parser.Parse(new MemoryStream(jsonBytes));

                        foreach (var key in configDict.Keys)
                            Data.Add(key, configDict[key]);
                    }
                }
            }
            catch(ConfigurationException)
            {
                throw;
            }
            catch (Exception ex)
            {
                throw new ConfigurationException("Failed on loading json file", ex);
            }
        }

        private byte[] LoadJsonFile(string path)
        {
            if (!File.Exists(path))
                throw new ConfigurationException($"The file at path: {path} does not exist");

            return File.ReadAllBytes(path);
        }
    }
}

SecretsResolver

We embed secrets using the format {secret:SECRET_NAME} and by convention we look for an environment variable with the name SECRET_NAME_SECRET_FILE which will indicate the path to load the secret from.

We also ensure that when SecretsMode.DockerSecrets is specified it ensures that the path starts with /run/secrets/ which is where Docker Secrets are mounted. This is just an extra security to avoid oversights.

namespace Govrnanza.Extensions.Configuration
{
    internal class SecretsResolver
    {
        private SecretsMode _secretsMode;

        public SecretsResolver(SecretsMode secretsMode)
        {
            _secretsMode = secretsMode;
        }

        public string ResolveEmbeddedSecret(string inputText)
        {
            var secretsResults = Regex.Matches(inputText, @"(?<secret>\{secret:[\s\w-\.]+\})");

            foreach (Match secretTagResult in secretsResults)
            {
                var groupText = secretTagResult.Groups[0].Value;
                var secretId = groupText.Substring(8, groupText.Length - 9).Trim();

                string storedSecret = LoadSecret(secretId);
                inputText = inputText.Replace(groupText, storedSecret);
            }

            return inputText;
        }

        private string LoadSecret(string secretId)
        {
            var pathToSecret = Environment.GetEnvironmentVariable(secretId + "_SECRET_FILE");
            if(_secretsMode == SecretsMode.DockerSecrets)
            {
                if (!pathToSecret.StartsWith("/run/secrets/", StringComparison.OrdinalIgnoreCase))
                    throw new ConfigurationException("Cannot load secrets from local files when run in Enforce mode");
            }

            if (!File.Exists(pathToSecret))
            {
                throw new ConfigurationException($"SecretId: {secretId} does not exist at " + pathToSecret + " or this service does not have access to it");
            }

            var storedSecret = File.ReadAllText(pathToSecret);

            return storedSecret;
        }
    }
}

JSON Parser

The Microsoft.Extensions.Configuration abstraction converts JSON files into a flat path format that gets loaded into a IDictionary<string, string>. For example the following JSON file:

{
    "apiVersionDefaults" : {
        "majorVersion" : 0,
        "minorVersion" : 1,
    },
    "versionApprovals" : [
        {
            "majorVersion" : 0,
            "approvalRequired" : false
        },
        {
            "majorVersion" : 1,
            "approvalRequired" : true,
            "approver" : "Johnny"
        }
    ]
}

Gets converted to a flat set of key/value:

apiVersionDefaults:majorVersion, 0
apiVersionDefaults:minorVersion, 1
versionApprovals:0:majorVersion, 0
versionApprovals:0:approvalRequired, false
versionApprovals:1:majorVersion, 1
versionApprovals:1:approvalRequired, true
versionApprovals:1:approver, Johnny

I won't put all the code of the JSON parser here. Just the method that I customised to resolve secrets.

private void VisitPrimitive(JValue data)
{
    var key = _currentPath;

    if (_data.ContainsKey(key))
    {
        throw new FormatException("Duplicate key: " + key);
    }

    var value = data.ToString(CultureInfo.InvariantCulture);
    var resolvedValue = _secretsResolver.ResolveEmbeddedSecret(value);
    _data[key] = resolvedValue;
}

Program.cs

Now we just need to add a reference to our new Govrnanza.Extensions.Configuration project and call the AddGovrnanzaConfig method. We make sure that we use SecretsMode.DockerSecrets when using the Production environment. This just ensure we really do use Docker Secrets instead of a local file for our secrets.

 

namespace Govrnanza.Registry.WebApi
{
    /// <summary>
    /// Entry point
    /// </summary>
    public class Program
    {
        /// <summary>
        /// Main method
        /// </summary>
        /// <param name="args"></param>
        public static void Main(string[] args)
        {
            BuildWebHost(args).Run();
        }

        /// <summary>
        /// Creates the IWebHost
        /// </summary>
        /// <param name="args"></param>
        /// <returns></returns>
        public static IWebHost BuildWebHost(string[] args)
        {
            var webHostBuilder = new WebHostBuilder()
                .UseKestrel()
                .UseContentRoot(Directory.GetCurrentDirectory())
                .ConfigureAppConfiguration((hostingContext, config) =>
                {
                    var secretsMode = GetSecretsMode(hostingContext.HostingEnvironment);
                    config.AddGovrnanzaConfig(secretsMode, "REGISTRY_CONFIG_FILE");
                })
                .UseStartup<Startup>()
                .Build();

            return webHostBuilder;
        }

        private static SecretsMode GetSecretsMode(IHostingEnvironment env)
        {
            if (env.IsProduction())
                return SecretsMode.DockerSecrets;

            var useDockerSecrets = Environment.GetEnvironmentVariable("REGISTRY_USE_DOCKER_SECRETS");
            if (useDockerSecrets != null && useDockerSecrets.Equals("false", StringComparison.OrdinalIgnoreCase))
                return SecretsMode.LocalFile;

            return SecretsMode.DockerSecrets;
        }
    }
}

I added an extra environment variable, REGISTRY_USE_DOCKER_SECRETS, to my Docker support docker-compose.override.yml to indicate whether Docker Secrets should be used in development or not. It allows me to run in Swarm Mode using Docker Secrets in the Development environment if I want.

services:
  govrnanza.registry.webapi:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - REGISTRY_CONFIG_FILE=devAppSettings.json
      - REGISTRY_DB_PASSWORD_SECRET_FILE=InsecureSecretFiles/RegistryDbPassword.txt
      - REGISTRY_USE_DOCKER_SECRETS=false
    ports:
      - "80"

Deployment

We need to update our dev Docker image.

cd C:\Users\Jack\Github\Govrnanza\src\Registry\Govrnanza.Registry.WebApi
dotnet publish -c Release -o ./obj/Docker/publish
docker build -t govrnanza.registry.webapi .
docker tag govrnanza.registry.webapi localhost:5000/govrnanza.registry.webapi:dev
docker push localhost:5000/govrnanza.registry.webapi:dev

Assuming we are already in Swarm Mode, now we run the docker stack deploy command

cd C:\Users\Jack\Github\Govrnanza\environments\dev
docker stack deploy -c docker-compose.yml govrnanza

Now our Registry service is running with our Docker Config and Secret.

Conclusion

So we have seen that we can use local files while in development and switch easily to Docker Secrets and Configs when deploying to a staging, test or production environment.

We didn't use multiple config and secret files but once we add more functionality and APIs to our Govrnanza system we'll need to. We'll see how we can use one config file among many services for things like logging configuration and other cross-cutting concerns.

We'll also be looking at how we can rotate secrets and configs without service disruption. We'll use Gatling to launch a load test against our Registry API and then start rotating secrets and configs and confirm that we don't cause any service disruption.

Go to the series introduction to see all links in the series.