Adam Cameron's Dev Blog: Getting my brain around Docker Secrets

G'day:

This is following on from yesterday's On the other hand… Docker Swarm.

Until now, when working with dockerised apps, I have had a fairly loosey-goosey way of dealing with things like DB passwords. Basically I stick 'em in a non-source-controlled envVars.private file, and when spinning up the container via either docker compose or docker run, include that file with the relevant mechanism, eg:

# docker-compose.yaml
services:
  # ...
  mariadb:
    container_name: db
    build:
      context: mariadb
      dockerfile: Dockerfile

    env_file:
      - envVars.public
      - envVars.private
      - mariadb/envVars.private

(--env-file docker/envVars.private in a docker run statement).

This works (and it's even how they recommend doing it in the MariaDB Docker image guidelines), but it's less than ideal:

docker inspect db | jq '.[0].Config.Env'
[
  "MARIADB_PASSWORD=123",
  "MARIADB_ROOT_PASSWORD=1234",
  "MARIADB_HOST=host.docker.internal",
  "MARIADB_PORT=3380",
  "MARIADB_DATABASE=db1",
  "MARIADB_USER=user1",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "GOSU_VERSION=1.17",
  "LANG=C.UTF-8",
  "MARIADB_VERSION=1:10.11.13+maria~ubu2204"
]

It's not cool having the passwords flapping about like that. And it's easy enough to deal with.

Dev environment (using `docker compose`)

docker-compose.yaml

There's two bits to contend with here. First there's a top-level secrets section:

secrets:
  app_secrets:
    file: php/appEnvVars.private
  mariadb_password:
    file: mariadb/mariadb_password_file.private
  mariadb_root_password:
    file: mariadb/mariadb_root_password_file.private

In this bit we give a secret a name (the app_secrets bit), and - in my case - a file with the value(s) in it. php/appEnvVars.private is thus:

APP_SECRET=does_not_matter

Formerly this was in envVars.private and was used by docker compose via an env_file directive as per further up.

The MariaDB ones work slightly differently; they contain only the password value (not a name=value pair), as that's the kinda accepted approach when using the file-based way of telling the MariaDB docker container where to find its passwords. One can either set an env var, eg: MARIADB_PASSWORD=123, or one can use MARIADB_PASSWORD_FILE (or MARIADB_ROOT_PASSWORD_FILE), and put just the password in the file instead. This means only the path to the file is exposed as an env var, not the password itself.

These secrets are applied to the containers via a secrets subsection of the service definition, eg:

services:
  # [...]
  php:
    # [...]

    secrets:
      - app_secrets
      - mariadb_password

This tells it which of the overall secrets are needed in this container. The PHP app doesn't need to know the MariaDB root password, so we're not setting that.

Within the container, these files are written to /run/secrets, with the filename being the name of the secret, so - eg - /run/secrets/app_secrets:

$ docker exec php cat /run/secrets/app_secrets
APP_SECRET=does_not_matter

For the MariaDB service we have this:

services:
  # [...]
  mariadb:
    # [...]

    secrets:
      - mariadb_password
      - mariadb_root_password

    environment:
      MARIADB_PASSWORD_FILE: /run/secrets/mariadb_password
      MARIADB_ROOT_PASSWORD_FILE: /run/secrets/mariadb_root_password

And that's all we need to do. The MariaDB install in the container is configured to pick these up automatically.

For the PHP app, we need to tell it how to load the values it needs.

PHP app

There are two things we need to achieve here. Firstly we need the value in /run/secrets/mariadb_password to be used for the value of a MARIADB_PASSWORD Symfony "environment" variable; and secondly we need to load the contents of /run/secrets/app_secrets similarly into Symfony's environment.

To do this I've created an EnvironmentService:

<?php

namespace App\Service;

use RuntimeException;
use Symfony\Component\Dotenv\Dotenv;

class EnvironmentService
{

    private const string MARIADB_SECRET_FILE = '/run/secrets/mariadb_password';
    private const string APP_SECRET_FILE = '/run/secrets/app_secrets';

    public static function load(): void
    {
        self::loadAppSecrets();
        self::loadMariaDbPassword();
    }

    private static function loadAppSecrets(): void
    {
        if (!file_exists(self::APP_SECRET_FILE)) {
            throw new RuntimeException(
                'App secrets file not found: ' . self::APP_SECRET_FILE
            );
        }

        $dotEnv = new Dotenv();
        $dotEnv->loadEnv(self::APP_SECRET_FILE);
    }

    private static function loadMariaDbPassword(): void
    {
        if (!file_exists(self::MARIADB_SECRET_FILE)) {
            throw new RuntimeException(
                'MariaDB password file not found: ' . self::MARIADB_SECRET_FILE
            );
        }

        $raw = file_get_contents(self::MARIADB_SECRET_FILE);
        if ($raw === false) {
            throw new RuntimeException(
                'Failed to read MariaDB password from: ' . self::MARIADB_SECRET_FILE
            );
        }
        $password = trim($raw);

        $_ENV['MARIADB_PASSWORD'] = $password;
        $_SERVER['MARIADB_PASSWORD'] = $password;
    }
}

Notes:

I'm using Symfony's DotEnv component to load the app_secrets, as that file is already in NAME=VALUE format.
I have to dick around slightly with the MARIADB_PASSWORD, but it's still pretty straight forward.

I call this from two places: public/index.php:

use App\Kernel;
use App\Service\EnvironmentService;

require_once dirname(__DIR__) . '/vendor/autoload_runtime.php';

EnvironmentService::load();

return function (array $context) {
    return new Kernel($context['APP_ENV'], (bool) $context['APP_DEBUG']);
};

And in the same way in tests/bootstrap.php so the tests have access to these settings too.

I have to admit that I did not write any tests for this stuff. As it's directly accessing external services (file system, environment variables), it's difficult to test the unhappy path, and write tests with no side effects. I do have other tests that rely on the happy path behaviour, so that's at least something (testing by inference, if not directly). I reasoned that this is only code for a blog, and it's not the main point of the article so: so be it. If this was for a production environment, I'd refactor this so it uses dependencies / service adapters so external-service interaction could be mocked out. This is possibly a topic for another blog article.

Test environment

I had to tweak my DB connection fixture slightly (tests/Integration/Fixtures/Database.php):

public static function getConnectionParameters(): stdClass
{
    return (object)[
        'host' => getenv('MARIADB_HOST'),
        'port' => getenv('MARIADB_PORT'),
        'database' => getenv('MARIADB_DATABASE'),
        'username' => getenv('MARIADB_USER'),
        'password' => getenv('MARIADB_PASSWORD')
        'password' => $_ENV['MARIADB_PASSWORD']
    ];
}

getEnv only gets "real" env vars; I need the one that I loaded directly into $_ENV via the EnvironmentService (above).

I could have exposed MARIADB_PASSWORD via geetenv as well, simply by using putenv as well as accessing $ENV and $_SERVER directly. So why didn't I? Two reasons:

I used the DotEnv component for the application secrets, and that does not use putenv, and I wanted to stay analogous with the MARIADB_PASSWORD handling.
I wanted to be very sure that the secret data did not somehow find its way back into actual environment variables (see tests below), so if I used putenv I would not be able to distinguish between "actual env vars" (security leak vector), and "faked" ones (safe).

This time I diligently (obsessively?) updated my testing of this stuff too (tests/Integration/System/EnvironmentTest.php):

#[TestDox('Tests of environment variables')]
class EnvironmentTest extends TestCase
{
    #[TestDox('The expected environment variables exist')]
    public function testEnvironmentVariables(): void
    {
        $varNames = [
            'APP_CACHE_DIR',
            'APP_LOG_DIR',
            'MARIADB_DATABASE',
            'MARIADB_HOST',
            'MARIADB_PORT',
            'MARIADB_USER',
        ];

        foreach ($varNames as $varName) {
            $this->assertNotFalse(
                getenv($varName),
                "Expected environment variable $varName to exist"
            );
        }
    }

    #[TestDox('Prohibited environment variables are not set')]
    public function testProhibitedEnvironmentVariables()
    {
        $varNames = [
            'APP_SECRET',
            'MARIADB_PASSWORD',
        ];

        foreach ($varNames as $varName) {
            $this->assertFalse(
                getenv($varName),
                "Prohibited environment variable $varName should not be set"
            );
        }
    }

    #[TestDox('Secret environment variables are set')]
    public function testSecretEnvironmentVariables(): void
    {
        $varNames = [
            'APP_SECRET',
            'MARIADB_PASSWORD',
        ];

        foreach ($varNames as $varName) {
            $this->assertNotEmpty(
                $_ENV[$varName],
                "Expected secret environment variable $varName to be set and to have a value"
            );
        }
    }
}

Hopefully that's all pretty self-explanatory: now I make sure the "secret" env vars are not set; but they are still availed via Symfony.

When I rebuild the app's containers, everything is working fine: the DB builds; the integration tests can find the DB; and so can the app. So that's the dev environment sorted.

Prod environment (using Docker Swarm)

Having done the prep by getting docker compose using secrets, and rejigging the app to access them, most of the work is done.

I've created a shell script to do the work (docker/php/bin/createService.sh), in which there are two relevant parts:

#!/bin/bash

DOCKER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"

docker secret create mariadb_password "$DOCKER_DIR/mariadb/mariadb_password_file.private"
docker secret create app_secrets "$DOCKER_DIR/php/appEnvVars.private"

docker service create \
    --name php \
    --replicas 3 \
    --publish published=9000,target=9000 \
    --env-file docker/envVars.public \
    --env-file docker/php/envVars.public \
    --env-file docker/php/envVars.prod.public \
    --host host.docker.internal:host-gateway \
    --secret app_secrets \
    --secret mariadb_password \
    adamcameron/php-swarm:latest

One creates secrets via docker secret create, and it has the same bits to it that the docker compose approach had.
And then one tells the service creation about said secrets.
Note I'm not doing anything about mariadb_root_password here, because only the DB container needs that, and this swarm service is only for the PHP application.

To run this I need a adamcameron/php-swarm:latest image:

$ docker build \
    -f docker/php/Dockerfile.prod \
    -t adamcameron/php-swarm:latest \
    -t adamcameron/php-swarm:0.1 \
    .
    
$ docker push adamcameron/php-swarm:latest
$ docker push adamcameron/php-swarm:0.1

Then I run those steps from the shell script (I could just run the shell script, I know):

# I already had it running. This is how to get rid
$ docker service rm php    
php

$  docker secret rm mariadb_password
mariadb_password

$ docker secret rm app_secrets
app_secrets

# recreate it all

$ docker secret create mariadb_password docker/mariadb/mariadb_password_file.private
hxra78q1xsoyf271deh92akrw
$ docker secret create app_secrets docker/php/appEnvVars.private
kinp56h4991j3gtr0w283oni3

$ docker secret ls
ID                          NAME               DRIVER    CREATED              UPDATED
kinp56h4991j3gtr0w283oni3   app_secrets                  32 seconds ago       32 seconds ago
hxra78q1xsoyf271deh92akrw   mariadb_password             About a minute ago   About a minute ago

$ docker service create \
    --name php \
    --replicas 3 \
    --publish published=9000,target=9000 \
    --env-file docker/envVars.public \
    --env-file docker/php/envVars.public \
    --env-file docker/php/envVars.prod.public \
    --host host.docker.internal:host-gateway \
    --secret app_secrets \
    --secret mariadb_password \
    adamcameron/php-swarm:latest
51h8y1axmdypz2qeptghius0g
overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service 51h8y1axmdypz2qeptghius0g converged

$ docker service ls
ID             NAME      MODE         REPLICAS   IMAGE                          PORTS
51h8y1axmdyp   php       replicated   3/3        adamcameron/php-swarm:latest   *:9000->9000/tcp

$ docker container ls --all --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep "php."
php.2.oairq1ddjpbh3jpzbgo6nppe6    Up About a minute (healthy)   9000/tcp
php.3.r4tkr6erup0r7ywpov62k7wpl    Up About a minute (healthy)   9000/tcp
php.1.o7avp09tz70o4dgcdxcif9daw    Up About a minute (healthy)   9000/tcp

# Is it working?
$ curl -s localhost:8080 | grep "Instance ID"
    Instance ID: 8a99b5b44744

$ curl -s localhost:8080 | grep "Instance ID"
    Instance ID: a95ed71029d9

$ curl -s localhost:8080 | grep "Instance ID"
    Instance ID: 888b7926ecda

$ curl -s localhost:8080 | grep "Instance ID"
    Instance ID: 8a99b5b44744

Note that each time I curl it, it's hitting a different PHP container of the three I have created.

So that's a docker swarm set-up using docker secrets. Box ticked. Obviously there's a bunch more to managing and using the swarm service, but that's not in scope here. I might look at it later, but only when I need to.

Righto.

--
Adam

Wednesday, 16 July 2025

Getting my brain around Docker Secrets

Dev environment (using docker compose)

docker-compose.yaml

PHP app

Test environment

Prod environment (using Docker Swarm)

Dev environment (using `docker compose`)