"filesystem: mkdir /var/lib/registry/docker: permission denied" during zarf init

[unknown0816]

Hi,
I have a rocky linux with a rke2 installed and running. During zarf installation I configure to use my hostpath pv, but during installation following error occurs:

2026-03-03 10:27:17 INF starting deploy package=init
2026-03-03 10:27:17 INF deploying component name=zarf-injector
2026-03-03 10:27:17 INF copying files count=1
2026-03-03 10:27:17 INF loading file name=###ZARF_TEMP###/zarf-injector
2026-03-03 10:27:17 INF waiting for cluster connection
2026-03-03 10:27:18 INF creating Zarf injector resources
2026-03-03 10:27:19 INF adding archived binary configmaps of registry image to the cluster
2026-03-03 10:27:29 INF deploying component name=zarf-seed-registry
2026-03-03 10:27:29 INF processing Helm chart name=docker-registry version=1.0.1 source=Zarf-generated
2026-03-03 10:27:29 INF performing Helm upgrade chart=docker-registry
2026-03-03 10:27:32 INF running health checks chart=docker-registry
2026-03-03 10:27:41 INF deploying component name=zarf-registry
2026-03-03 10:27:41 INF pushing image name=docker.io/alpine/socat:1.8.0.3
2026-03-03 10:27:46 INF image push in progress name=docker.io/alpine/socat:1.8.0.3 complete=0.0% remaining=4.41 MBs
2026-03-03 10:27:54 INF image push in progress name=docker.io/alpine/socat:1.8.0.3 complete=0.0% remaining=4.41 MBs
2026-03-03 10:27:57 INF pushing image name=docker.io/library/registry:3.0.0
2026-03-03 10:28:02 INF image push in progress name=docker.io/library/registry:3.0.0 complete=0.0% remaining=18.56 MBs
2026-03-03 10:28:10 INF image push in progress name=docker.io/library/registry:3.0.0 complete=0.0% remaining=18.56 MBs
2026-03-03 10:28:13 INF pushing image name=docker.io/library/registry:3.0.0
2026-03-03 10:28:18 INF image push in progress name=docker.io/library/registry:3.0.0 complete=0.0% remaining=18.56 MBs
2026-03-03 10:28:26 INF image push in progress name=docker.io/library/registry:3.0.0 complete=0.0% remaining=18.56 MBs
2026-03-03 10:28:28 ERR failed to deploy package: unable to deploy component "zarf-registry": unable to push images to the registry: All attempts fail:
#1: All attempts fail:
#1: failed to push image docker.io/alpine/socat:1.8.0.3: failed to perform "Push" on destination: POST "http://127.0.0.1:45763/v2/alpine/socat/blobs/uploads/": response status code 500: unknown: unknown error: filesystem: mkdir /var/lib/registry/docker: permission denied
#2: failed to push image docker.io/alpine/socat:1.8.0.3: failed to perform "Push" on destination: POST "http://127.0.0.1:45763/v2/alpine/socat/blobs/uploads/": response status code 500: unknown: unknown error: filesystem: mkdir /var/lib/registry/docker: permission denied
#2: All attempts fail:
#1: failed to push image docker.io/library/registry:3.0.0: failed to perform "Push" on destination: POST "http://127.0.0.1:43609/v2/library/registry/blobs/uploads/": response status code 500: unknown: unknown error: filesystem: mkdir /var/lib/registry/docker: permission denied
#2: failed to push image docker.io/library/registry:3.0.0: failed to perform "Push" on destination: POST "http://127.0.0.1:43609/v2/library/registry/blobs/uploads/": response status code 500: unknown: unknown error: filesystem: mkdir /var/lib/registry/docker: permission denied
#3: All attempts fail:
#1: failed to push image docker.io/library/registry:3.0.0: failed to perform "Push" on destination: POST "http://127.0.0.1:44685/v2/library/registry/blobs/uploads/": response status code 500: unknown: unknown error: filesystem: mkdir /var/lib/registry/docker: permission denied
#2: failed to push image docker.io/library/registry:3.0.0: failed to perform "Push" on destination: POST "http://127.0.0.1:44685/v2/library/registry/blobs/uploads/": response status code 500: unknown: unknown error: filesystem: mkdir /var/lib/registry/docker: permission denied

I tried to create the directory manual and provide the permissions but nothing worked so far.

Can someone help please?

[brandtkeller]

Hey @unknown0816 , It sounds like your are on the right track for debugging.

This does sound like a configuration issue between distribution and the host. Could be a number of things but top of mind logs would help us review further - preferably as close to when the error occurs.

zarf tools kubectl -n zarf logs -l app=docker-registry --tail=200

Could be selinux or maybe even some flavor of pod context.

getenforce

  zarf tools kubectl -n zarf get pod -l app=docker-registry -o yaml | \
    zarf tools yq '.items[0].spec.securityContext,
        .items[0].spec.containers[0].securityContext,
        .items[0].spec.containers[0].volumeMounts,
        .items[0].spec.volumes'

That may help us moving forward with debugging this.

[unknown0816]

@brandtkeller Thanks for your response.

The logs show the same error as the command line.

getenforce responses with "Enforcing" and

the last command provides this:

fsGroup: 2000
runAsGroup: 2000
runAsUser: 1000
seccompProfile:
  type: RuntimeDefault
allowPrivilegeEscalation: false
capabilities:
  drop:
    - ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
- mountPath: /var/lib/registry/
  name: data
- mountPath: /etc/docker/registry
  name: config
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: kube-api-access-7p7rm
  readOnly: true
- name: config
  secret:
    defaultMode: 420
    items:
      - key: configData
        path: config.yml
      - key: htpasswd
        path: htpasswd
    secretName: zarf-docker-registry-secret
- name: data
  persistentVolumeClaim:
    claimName: zarf-docker-registry
- name: kube-api-access-7p7rm
  projected:
    defaultMode: 420
    sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
            - key: ca.crt
              path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace

Further, I tried to go into the container and see what is going on at /var/lib/registry, but I cannot even execute "ls" without a permission denied response.

[unknown0816]

I removed all the security config inside the deployment and I get this for the last command:

"- mountPath: /var/lib/registry/
name: data

• mountPath: /etc/docker/registry
  name: config
• mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: kube-api-access-rq4xg
  readOnly: true
• name: config
  secret:
  defaultMode: 420
  items:
  - key: configData
  path: config.yml
  - key: htpasswd
  path: htpasswd
  secretName: zarf-docker-registry-secret
• name: data
  persistentVolumeClaim:
  claimName: zarf-docker-registry
• name: kube-api-access-rq4xg
  projected:
  defaultMode: 420
  sources:
  - serviceAccountToken:
  expirationSeconds: 3607
  path: token
  - configMap:
  items:
  - key: ca.crt
  path: ca.crt
  name: kube-root-ca.crt
  - downwardAPI:
  items:
  - fieldRef:
  apiVersion: v1
  fieldPath: metadata.namespace
  path: namespace"

Still there is the permission denied trying to acces /var/lib/registry with ls

[brandtkeller]

Could we try disabling selinux and trying again? I'd love to rule out that being a blocker as the debugging paths deviate.

I believe

sudo setenforce 0

[unknown0816]

@brandtkeller Sorry for the late response.

Actually, that was it. The init command went through successfully now. Thank you!

However, I have access to a different machine (OpenSUSE Leap micro), where RKE2 and zarf is running without problems. And I checked there, too. Here the selinux is active.. That somehow confuses me.. Could it be, that on the other maschine longhorn is the PV provider and not hostpath like on my maschine?

[brandtkeller]

I would say it is fair that there are multiple permutations involved that could produce this problem. I would look for any clear differences between them to see if you might be able to replicate behaviors.

I'm going to close this issue for now - if there is a zarf problem we can resolve related to this please feel free to re-open.