Secure Secrets in Docker Builds


Note: in this blog post, we echo out secrets in Dockerfiles for demonstration purposes only — you should never echo a sensitive value in a Dockerfile anywhere else.

TLDR

If you need to use a secret during a Docker build but don’t want it to persist in the final Docker image, you can mount a file containing sensitive values with the new --secret flag:

DOCKER_BUILDKIT=1 docker build \
    --secret id=mycoolsecret,src=file_containing_secrets.txt \
    --progress=plain --no-cache -f Dockerfile-buildkit \
    -t i-used-buildkit .

You can then reference the file containing secrets at Docker build time by using the --mount flag in your Dockerfile. When you use the --mount flag in a Dockerfile step, the secret will be available at the specified destination:

# syntax = docker/dockerfile:1.0-experimental

FROM alpine:edge
RUN --mount=type=secret,id=mycoolsecret,dst=/tmp/file_containing_secrets.txt cat /tmp/file_containing_secrets.txt
RUN cat /tmp/file_containing_secrets.txt

The contents of the secret will not persist in the image or any intermediate layers, although a stub file at /tmp/file_containing_secrets.txt will remain in the image (see GitHub issue here).

Introduction

There are a variety of reasons you might want to use a secret during a Docker build: maybe you need to clone a private repository in GitHub (and hence would need a token or an SSH key), or maybe you need to make a call to another external service that requires authentication. The general problem is that your Docker build process needs sensitive values to create a Docker image, even though you do not want those sensitive values to persist in the final Docker image or any intermediate layers.

Don’t ADD then rm files containing secrets

You might think you could ADD a secret to your Dockerfile, do what you need with it, and then rm the file. No problem, right? Nope — the secret that you added will persist in the intermediate Docker layers, which are cached. Here’s an example Dockerfile that adds a secret file to a Docker image and then deletes it (catting out secrets is done for demonstration purposes and is not a secure practice):

FROM alpine:edge
ADD file_containing_secrets.txt /tmp/file_containing_secrets.txt
RUN cat /tmp/file_containing_secrets.txt
RUN echo "pre-deletion /tmp listing:" && ls /tmp
RUN rm /tmp/file_containing_secrets.txt
RUN echo "post-deletion /tmp listing:" && ls /tmp

Since you deleted your secret from your Docker image, you could be forgiven for thinking that your secret would never see the light of day. Here, we run a Docker build with this Dockerfile.

echo "mysecretcontents" > file_containing_secrets.txt
docker build --no-cache -f Dockerfile -t foobar .
Sending build context to Docker daemon  67.58kB
Step 1/6 : FROM alpine:edge
 ---> 3e8d7a5561f0
Step 2/6 : ADD file_containing_secrets.txt /tmp/file_containing_secrets.txt
 ---> c4ffeb46c264
Step 3/6 : RUN cat /tmp/file_containing_secrets.txt
 ---> Running in 25ed810b9957
mysecretcontents
Removing intermediate container 25ed810b9957
 ---> 263ca69cebfb
Step 4/6 : RUN echo "pre-deletion /tmp listing:" && ls /tmp
 ---> Running in 5925c1de68ce
pre-deletion /tmp listing:
file_containing_secrets.txt
Removing intermediate container 5925c1de68ce
 ---> b8b441d8fe65
Step 5/6 : RUN rm /tmp/file_containing_secrets.txt
 ---> Running in 62faa4c8f798
Removing intermediate container 62faa4c8f798
 ---> 87cc8dfc9f26
Step 6/6 : RUN echo "post-deletion /tmp listing:" && ls /tmp
 ---> Running in 10953cc51075
post-deletion /tmp listing:
Removing intermediate container 10953cc51075
 ---> 34ca6fc8fd13
Successfully built 34ca6fc8fd13
Successfully tagged foobar:latest

As you can see above, the secret seems to be gone from the container ("post-deletion /tmp listing:").

But because of the way that Docker caches layers, your secret is still hanging around! Each line of the Dockerfile produces a working image that the subsequent line builds on. This means that we can read out the value of our file_containing_secrets.txt by running one of these intermediate layers. Check it out:

$ docker run b8b441d8fe65 cat /tmp/file_containing_secrets.txt
mysecretcontents

That hash — b8b441d8fe65 — is a fully functional Docker image containing our secret! Bad!

There are a variety of workarounds for this, but none of them are pretty. Fortunately, there is some Docker functionality that makes using secrets at build time easy!

DOCKER_BUILDKIT and –secret is your friend!

As part of the 18.09 release, Docker added a --secret flag for docker build which lets you pass a file containing secrets to a Docker build without the contents persisting to any intermediate layer (docs here). The contents of the file containing sensitive values will be available during the build, but will not persist in any Docker layer.

Note: the name of the file that you mount the secret to in the Docker image will persist in the final image, but its contents will not (see this open issue for details).

The syntax is broken across two different places: the Dockerfile and the docker build command. The Dockerfile will look something like this:

# syntax = docker/dockerfile:1.0-experimental

FROM alpine:edge
# Secret echoed out for demonstration; don't do this with actually sensitive values
RUN --mount=type=secret,id=mycoolsecret,dst=/tmp/file_containing_secrets.txt cat /tmp/file_containing_secrets.txt
RUN cat /tmp/file_containing_secrets.txt

The first line tells Docker that you want to use the new syntax (which is used in the Dockerfile’s second command). The action is in the --mount flag, where we say that we want a secret with the id “mycoolsecret” to be mounted in the build with the destination /tmp/file_containing_secrets.txt.

The other piece of the puzzle comes when we run docker build against this Dockerfile. At that time we need to tell Docker two things:

  1. we want to use the new buildkit features
  2. where to find the secret that we want to mount into the container, identified by “mycoolsecret” in our Dockerfile

You do that by specifying DOCKER_BUILDKIT=1 and passing a --secret flag to the command. It’ll look like this:

DOCKER_BUILDKIT=1 docker build \
    --secret id=mycoolsecret,src=file_containing_secrets.txt \
    --progress=plain -f Dockerfile-buildkit -t foobar-buildkit .

The --secret flag declares a secret with a specific id and a path to the file that we want to mount (secretly!) into the container (this is the src=file_containing_secrets.txt part). When you run the docker build command, the truncated output is as follows:

echo "mysecretcontents" > file_containing_secrets.txt
DOCKER_BUILDKIT=1 docker build --no-cache --secret id=mycoolsecret,src=file_containing_secrets.txt --progress=plain -f Dockerfile-buildkit -t foobar-buildkit .

...

#7 [2/3] RUN --mount=type=secret,id=mycoolsecret,dst=/tmp/file_containing_s...
#7       digest: sha256:5709b47194baa2bc49ff7087f7e4b0405a93b253d94d2c8e431a3b7e17f717a1
#7         name: "[2/3] RUN --mount=type=secret,id=mycoolsecret,dst=/tmp/file_containing_secrets.txt cat /tmp/file_containing_secrets.txt"
#7      started: 2019-04-19 19:58:03.8456457 +0000 UTC
#7 0.524 mysecretcontents
#7    completed: 2019-04-19 19:58:04.6194124 +0000 UTC
#7     duration: 773.7667ms


#8 [3/3] RUN cat /tmp/file_containing_secrets.txt
#8       digest: sha256:90ab2e02b5f3804cc30ee5ab9b394ddef1efc892b3f41e0adb3ed421c187fe4f
#8         name: "[3/3] RUN cat /tmp/file_containing_secrets.txt"
#8      started: 2019-04-19 19:58:04.6280145 +0000 UTC
#8    completed: 2019-04-19 19:58:05.6051815 +0000 UTC
#8     duration: 977.167ms

...

As you can see, the output of step 7 — where we cat /tmp/file_containing_secrets.txt — is “mysecretcontents”. This is because of the --mount flag present in the Dockerfile. But when we cat the contents of /tmp/file_containing_secrets.txt in step 8, the contents of the file are gone. Hooray!

If you run docker history, you can see that the secret is also not visible there, although you can see that we catted it out.

What else?

It’s also worth noting that you can forward SSH agent connections when running Docker build with the following directive: RUN --mount=type=ssh. For more information on that, check the docs.

Try it yourself

The syntax is a bit tricky here, so I put up a GitHub repo with various types of Dockerfile here.