mounting permission denied when "userns-remap" is enabled in Docker v20.10.3+ #2200
Description
This issue is likely a duplicate of #2087 (secret mount) and #2059 (ssh mount)
Since docker v20.10.3, a breaking change was introduced that causes "permission denied" error when mounting secret or ssh while "userns-remap" is enabled. We believe it was caused by this change which affect how userns-remap behave with docker build
- CVE-2021-21284 Lock down file permissions to prevent remapped root from accessing docker state
(from https://docs.docker.com/engine/release-notes/)
Steps to replicate on v20.10.3+
daemon.json config
{
"debug": true,
"userns-remap" : "default"
}
local files
$ cat mysecret.txt
WARMACHINEROX
$ cat secret-test.Dockerfile
# syntax=docker/dockerfile:1.2
FROM alpine
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
$ cat ssh-test.Dockerfile
# syntax=docker/dockerfile:1.2
FROM alpine
RUN apk add --no-cache openssh-client git
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh ssh -T git@github.com
test mounting secret
$ docker build --no-cache --secret id=mysecret,src=mysecret.txt -f secret-test.Dockerfile .
[+] Building 1.3s (7/7) FINISHED
=> [internal] load build definition from secret-test.Dockerfile 0.0s
=> => transferring dockerfile: 49B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.2 1.0s
=> CACHED docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> CACHED [1/2] FROM docker.io/library/alpine 0.0s
=> ERROR [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret 0.1s
------
> [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret:
#6 0.077 container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/var/lib/docker/231072.231072/tmp/buildkit-secrets242796215/fqqyn8lizraz0w248sd729sqm" to rootfs at "/run/secrets/mysecret" caused: stat /var/lib/docker/231072.231072/tmp/buildkit-secrets242796215/fqqyn8lizraz0w248sd729sqm: permission denied
------
executor failed running [/bin/sh -c cat /run/secrets/mysecret]: exit code: 1
test mounting ssh
$ docker build --no-cache --ssh default -f ssh-test.Dockerfile .
[+] Building 7.4s (9/9) FINISHED
=> [internal] load build definition from ssh-test.Dockerfile 0.0s
=> => transferring dockerfile: 47B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.2 1.0s
=> CACHED docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> CACHED [1/4] FROM docker.io/library/alpine 0.0s
=> [2/4] RUN apk add --no-cache openssh-client git 3.4s
=> [3/4] RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts 2.4s
=> ERROR [4/4] RUN --mount=type=ssh ssh -T git@github.com 0.3s
------
> [4/4] RUN --mount=type=ssh ssh -T git@github.com:
#9 0.248 container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/var/lib/docker/231072.231072/tmp/.buildkit-ssh-sock704823831/ssh_auth_sock" to rootfs at "/run/buildkit/ssh_agent.0" caused: stat /var/lib/docker/231072.231072/tmp/.buildkit-ssh-sock704823831/ssh_auth_sock: permission denied
------
executor failed running [/bin/sh -c ssh -T git@github.com]: exit code: 1
For comparision, this is the expected output on v20.10.2
test mounting secret
$ docker build --secret id=mysecret,src=mysecret.txt -f secret-test.Dockerfile .
[+] Building 0.5s (8/8) FINISHED
=> [internal] load build definition from secret-test.Dockerfile 0.0s
=> => transferring dockerfile: 49B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.2 0.3s
=> CACHED docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [1/2] FROM docker.io/library/alpine 0.0s
=> CACHED [2/2] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:efaf08af45db93bbf501bfc5771aa656514eb231eff280a80571c2d00c149a20 0.0s
test mounting ssh
$ docker build --ssh default -f ssh-test.Dockerfile .
[+] Building 2.9s (9/9) FINISHED
=> [internal] load build definition from ssh-test.Dockerfile 0.0s
=> => transferring dockerfile: 47B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.2 1.2s
=> CACHED docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [1/4] FROM docker.io/library/alpine 0.0s
=> CACHED [2/4] RUN apk add --no-cache openssh-client git 0.0s
=> CACHED [3/4] RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts 0.0s
=> ERROR [4/4] RUN --mount=type=ssh ssh -T git@github.com 1.5s
------
> [4/4] RUN --mount=type=ssh ssh -T git@github.com:
#9 1.428 git@github.com: Permission denied (publickey).
------
executor failed running [/bin/sh -c ssh -T git@github.com]: exit code: 255