`docker save` sometimes returns images missing blobs

[cowwoc]

Version 0.7

If I enable the following option in k3d's config file:

options:
  k3s:
    extraArgs:
      - arg: "--disable-default-registry-endpoint"
        nodeFilters:
          - "all:*"

Then coredns fails to load properly:

Normal Pulling 26s kubelet Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1"
Warning Failed 5s kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": rpc error: code = NotFound desc = failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to copy: httpReadSeeker: failed open: could not fetch content descriptor sha256:8fd3d3275362698251cd271235e68f78674a216a940db7e4857cda5302c2ce5a (application/vnd.oci.image.config.v1+json) from remote: not found

All other images seem to work fine.

Removing the above option allows k3d to bypass your proxy if it runs into an error using it, which is probably why you're not seeing this error without it.

[ligfx]

Works for me with the config file:

apiVersion: k3d.io/v1alpha5
kind: Simple
options:
  k3s:
    extraArgs:
    - arg: "--disable-default-registry-endpoint"
      nodeFilters:
      - "all:*"
registries:
  create:
    image: ligfx/k3d-registry-dockerd
    proxy:
      remoteURL: "*"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

And running kubectl run mypod --image=registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1 manually, since kube-webhook-certgen doesn't seem to be used in my install of k3d.

Can you provide more information?

[cowwoc]

Hmm, that's a real mystery... Okay, here are more detailed repro steps:

1. (Ideally) Remove all docker containers, images, volumes to ensure that you are starting from a blank slate.
2. Make sure that k3d cluster ls and k3d registry ls do not contain any conflicting entries (in my case, they are empty).
3. Make sure that docker network ls does not contain any existing network for the cluster we are about to create.
4. Run k3d cluster create foobar --config k3d-config.yaml where k3d-config.yaml contains:

apiVersion: k3d.io/v1alpha5
kind: Simple
registries:
  create:
    image: ligfx/k3d-registry-dockerd:v0.7
    proxy:
      remoteURL: "*"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
options:
  k3s:
    extraArgs:
      - arg: "--disable-default-registry-endpoint"
        nodeFilters:
          - "all:*"

I get this output:

INFO[0000] Using config file k3d-config.yaml (k3d.io/v1alpha5#simple)
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-foobar'
INFO[0000] Created image volume k3d-foobar-images
INFO[0000] Creating node 'k3d-foobar-registry'
INFO[0000] Pulling image 'ligfx/k3d-registry-dockerd:v0.7'
INFO[0000] Successfully created registry 'k3d-foobar-registry'
INFO[0000] Starting new tools node...
INFO[0001] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.8.1'
INFO[0001] Creating node 'k3d-foobar-server-0'
INFO[0002] Pulling image 'docker.io/rancher/k3s:v1.31.4-k3s1'
INFO[0007] Starting node 'k3d-foobar-tools'
INFO[0017] Creating LoadBalancer 'k3d-foobar-serverlb'
INFO[0017] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.8.1'
INFO[0022] Using the k3d-tools node to gather environment information
INFO[0023] Starting new tools node...
INFO[0023] Starting node 'k3d-foobar-tools'
INFO[0024] Starting cluster 'foobar'
INFO[0024] Starting servers...
INFO[0025] Starting node 'k3d-foobar-server-0'
INFO[0028] All agents already running.
INFO[0028] Starting helpers...
INFO[0028] Starting node 'k3d-foobar-registry'
INFO[0029] Starting node 'k3d-foobar-serverlb'
INFO[0035] Injecting records for hostAliases (incl. host.k3d.internal) and for 4 network members into CoreDNS configmap...
INFO[0038] Cluster 'foobar' created successfully!
INFO[0039] You can now use it like this:
kubectl cluster-info

I am using the following tools/versions:

docker version returns:

Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:41:44 2025
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.38.0 (181591)
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:17 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

k3d --version returns:

k3d version v5.8.1
k3s version v1.31.4-k3s1 (default)

on Microsoft Windows [Version 10.0.19045.5487]

docker images returns:

REPOSITORY                         TAG                    IMAGE ID       CREATED        SIZE
ligfx/k3d-registry-dockerd         v0.7                   cd6d1487acb9   11 days ago    25.5MB
ghcr.io/k3d-io/k3d-proxy           5.8.1                  f2ac9af57761   3 weeks ago    93.1MB
ghcr.io/k3d-io/k3d-tools           5.8.1                  bb3b76a2eefa   3 weeks ago    31.9MB
rancher/k3s                        v1.31.4-k3s1           627fc392e5e7   7 weeks ago    298MB
rancher/klipper-helm               v0.9.3-build20241008   73ff7ef39971   4 months ago   262MB
rancher/local-path-provisioner     v0.0.30                9b9148811700   4 months ago   71.1MB
rancher/mirrored-library-traefik   2.11.10                b19934b3f8e0   4 months ago   218MB
rancher/mirrored-metrics-server    v0.7.2                 dccf8474fb91   5 months ago   89.8MB
rancher/klipper-lb                 v0.4.9                 dd380f5d89a5   6 months ago   18.4MB
rancher/mirrored-pause             3.6                    74c4244427b7   3 years ago    986kB
rancher/mirrored-coredns-coredns   1.12.0                 82979ddf442c   55 years ago   20.9MB

So although we can see that rancher/mirrored-coredns-coredns:1.12.0 exists on the host, the cluster still complains.

kubectl get pods --namespace kube-system returns:

NAME READY STATUS RESTARTS AGE
coredns-ccb96694c-4m7bt 0/1 ImagePullBackOff 0 3m9s
helm-install-traefik-6rw9x 0/1 Completed 1 3m9s
helm-install-traefik-crd-4brh8 0/1 Completed 0 3m9s
local-path-provisioner-5cf85fd84d-9rb6g 1/1 Running 0 3m9s
metrics-server-5985cbc9d7-kmf2d 1/1 Running 0 3m9s
svclb-traefik-11a7acdb-hzfsf 2/2 Running 0 2m35s
traefik-57b79cf995-hc72d 1/1 Running 0 2m35s

kubectl events coredns-ccb96694c-4m7bt --namespace kube-system returns:

LAST SEEN               TYPE      REASON                 OBJECT                                         MESSAGE
4m18s                   Normal    ApplyingManifest       Addon/runtimes                                 Applying manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml"
4m18s                   Normal    ApplyingManifest       Addon/metrics-apiservice                       Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
4m18s                   Normal    ApplyingManifest       Addon/resource-reader                          Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
4m18s                   Normal    AppliedManifest        Addon/auth-delegator                           Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
4m18s                   Normal    ApplyingManifest       Addon/auth-reader                              Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
4m18s                   Normal    AppliedManifest        Addon/auth-reader                              Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
4m18s                   Normal    ApplyingManifest       Addon/ccm                                      Applying manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml"
4m18s                   Normal    AppliedManifest        Addon/ccm                                      Applied manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml"
4m18s                   Normal    AppliedManifest        Addon/metrics-server-deployment                Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
4m18s                   Normal    AppliedManifest        Addon/traefik                                  Applied manifest at "/var/lib/rancher/k3s/server/manifests/traefik.yaml"
4m18s                   Normal    ApplyingManifest       Addon/traefik                                  Applying manifest at "/var/lib/rancher/k3s/server/manifests/traefik.yaml"
4m18s                   Normal    AppliedManifest        Addon/runtimes                                 Applied manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml"
4m18s                   Normal    ApplyingManifest       Addon/auth-delegator                           Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
4m18s                   Normal    AppliedManifest        Addon/rolebindings                             Applied manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
4m18s                   Normal    ApplyingManifest       Addon/local-storage                            Applying manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml"
4m18s                   Normal    ApplyingManifest       Addon/metrics-server-deployment                Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
4m18s                   Normal    AppliedManifest        Addon/metrics-server-service                   Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
4m18s                   Normal    AppliedManifest        Addon/resource-reader                          Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
4m18s                   Normal    AppliedManifest        Addon/metrics-apiservice                       Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
4m18s                   Normal    AppliedManifest        Addon/aggregated-metrics-reader                Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
4m18s                   Normal    AppliedManifest        Addon/local-storage                            Applied manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml"
4m18s                   Normal    ApplyingManifest       Addon/metrics-server-service                   Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
4m18s                   Normal    ApplyingManifest       Addon/aggregated-metrics-reader                Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
4m18s                   Normal    ApplyingManifest       Addon/rolebindings                             Applying manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
4m16s                   Normal    ScalingReplicaSet      Deployment/coredns                             Scaled up replica set coredns-ccb96694c to 1
4m16s                   Normal    ScalingReplicaSet      Deployment/local-path-provisioner              Scaled up replica set local-path-provisioner-5cf85fd84d to 1
4m16s                   Normal    ScalingReplicaSet      Deployment/metrics-server                      Scaled up replica set metrics-server-5985cbc9d7 to 1
4m15s                   Normal    SuccessfulCreate       ReplicaSet/coredns-ccb96694c                   Created pod: coredns-ccb96694c-4m7bt
4m15s                   Normal    SuccessfulCreate       ReplicaSet/local-path-provisioner-5cf85fd84d   Created pod: local-path-provisioner-5cf85fd84d-9rb6g
4m15s                   Normal    SuccessfulCreate       Job/helm-install-traefik-crd                   Created pod: helm-install-traefik-crd-4brh8
4m15s                   Normal    SuccessfulCreate       Job/helm-install-traefik                       Created pod: helm-install-traefik-6rw9x
4m15s                   Warning   FailedMount            Pod/helm-install-traefik-crd-4brh8             MountVolume.SetUp failed for volume "kube-api-access-fspdc" : configmap "kube-root-ca.crt" not found
4m15s                   Warning   FailedMount            Pod/helm-install-traefik-6rw9x                 MountVolume.SetUp failed for volume "kube-api-access-xf9rs" : configmap "kube-root-ca.crt" not found
4m15s                   Normal    SuccessfulCreate       ReplicaSet/metrics-server-5985cbc9d7           Created pod: metrics-server-5985cbc9d7-kmf2d
4m14s                   Normal    Scheduled              Pod/helm-install-traefik-crd-4brh8             Successfully assigned kube-system/helm-install-traefik-crd-4brh8 to k3d-foobar-server-0
4m14s                   Normal    Scheduled              Pod/helm-install-traefik-6rw9x                 Successfully assigned kube-system/helm-install-traefik-6rw9x to k3d-foobar-server-0
4m14s                   Normal    Scheduled              Pod/coredns-ccb96694c-4m7bt                    Successfully assigned kube-system/coredns-ccb96694c-4m7bt to k3d-foobar-server-0
4m14s                   Normal    Scheduled              Pod/local-path-provisioner-5cf85fd84d-9rb6g    Successfully assigned kube-system/local-path-provisioner-5cf85fd84d-9rb6g to k3d-foobar-server-0
4m14s                   Normal    Scheduled              Pod/metrics-server-5985cbc9d7-kmf2d            Successfully assigned kube-system/metrics-server-5985cbc9d7-kmf2d to k3d-foobar-server-0
4m13s                   Normal    Pulling                Pod/helm-install-traefik-6rw9x                 Pulling image "rancher/klipper-helm:v0.9.3-build20241008"
4m13s                   Normal    Pulling                Pod/local-path-provisioner-5cf85fd84d-9rb6g    Pulling image "rancher/local-path-provisioner:v0.0.30"
4m13s                   Normal    Pulling                Pod/helm-install-traefik-crd-4brh8             Pulling image "rancher/klipper-helm:v0.9.3-build20241008"
4m13s                   Normal    Pulling                Pod/metrics-server-5985cbc9d7-kmf2d            Pulling image "rancher/mirrored-metrics-server:v0.7.2"
4m3s (x2 over 4m18s)    Normal    AppliedManifest        Addon/coredns                                  Applied manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml"
4m3s (x2 over 4m18s)    Normal    ApplyingManifest       Addon/coredns                                  Applying manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml"
4m                      Normal    Started                Pod/local-path-provisioner-5cf85fd84d-9rb6g    Started container local-path-provisioner
4m                      Normal    Created                Pod/local-path-provisioner-5cf85fd84d-9rb6g    Created container local-path-provisioner
4m                      Normal    Pulled                 Pod/local-path-provisioner-5cf85fd84d-9rb6g    Successfully pulled image "rancher/local-path-provisioner:v0.0.30" in 13.601s (13.601s including waiting). Image size: 18584855 bytes.
3m52s                   Normal    Created                Pod/metrics-server-5985cbc9d7-kmf2d            Created container metrics-server
3m52s                   Normal    Started                Pod/metrics-server-5985cbc9d7-kmf2d            Started container metrics-server
3m52s                   Warning   Unhealthy              Pod/metrics-server-5985cbc9d7-kmf2d            Readiness probe failed: Get "https://10.42.0.3:10250/readyz": dial tcp 10.42.0.3:10250: connect: connection refused
3m52s                   Normal    Pulled                 Pod/metrics-server-5985cbc9d7-kmf2d            Successfully pulled image "rancher/mirrored-metrics-server:v0.7.2" in 21.276s (21.276s including waiting). Image size: 19494186 bytes.
3m43s                   Normal    Pulled                 Pod/helm-install-traefik-crd-4brh8             Successfully pulled image "rancher/klipper-helm:v0.9.3-build20241008" in 30.311s (30.311s including waiting). Image size: 70496341 bytes.
3m43s                   Normal    Pulled                 Pod/helm-install-traefik-6rw9x                 Successfully pulled image "rancher/klipper-helm:v0.9.3-build20241008" in 30.274s (30.274s including waiting). Image size: 70496341 bytes.
3m43s                   Normal    Started                Pod/helm-install-traefik-crd-4brh8             Started container helm
3m43s                   Normal    Created                Pod/helm-install-traefik-crd-4brh8             Created container helm
3m42s (x2 over 3m43s)   Normal    Created                Pod/helm-install-traefik-6rw9x                 Created container helm
3m42s                   Normal    Pulled                 Pod/helm-install-traefik-6rw9x                 Container image "rancher/klipper-helm:v0.9.3-build20241008" already present on machine
3m42s (x2 over 3m43s)   Normal    Started                Pod/helm-install-traefik-6rw9x                 Started container helm
3m41s                   Normal    SuccessfulCreate       DaemonSet/svclb-traefik-11a7acdb               Created pod: svclb-traefik-11a7acdb-hzfsf
3m41s                   Normal    AppliedDaemonSet       Service/traefik                                Applied LoadBalancer DaemonSet kube-system/svclb-traefik-11a7acdb
3m41s                   Normal    ScalingReplicaSet      Deployment/traefik                             Scaled up replica set traefik-57b79cf995 to 1
3m41s                   Normal    EnsuringLoadBalancer   Service/traefik                                Ensuring load balancer
3m41s                   Normal    SuccessfulCreate       ReplicaSet/traefik-57b79cf995                  Created pod: traefik-57b79cf995-hc72d
3m41s                   Normal    Pulling                Pod/traefik-57b79cf995-hc72d                   Pulling image "rancher/mirrored-library-traefik:2.11.10"
3m41s                   Normal    Pulling                Pod/svclb-traefik-11a7acdb-hzfsf               Pulling image "rancher/klipper-lb:v0.4.9"
3m40s                   Normal    Scheduled              Pod/traefik-57b79cf995-hc72d                   Successfully assigned kube-system/traefik-57b79cf995-hc72d to k3d-foobar-server-0
3m40s                   Normal    Scheduled              Pod/svclb-traefik-11a7acdb-hzfsf               Successfully assigned kube-system/svclb-traefik-11a7acdb-hzfsf to k3d-foobar-server-0
3m40s                   Normal    Completed              Job/helm-install-traefik-crd                   Job completed
3m40s (x9 over 4m18s)   Normal    ApplyJob               HelmChart/traefik-crd                          Applying HelmChart using Job kube-system/helm-install-traefik-crd
3m37s (x9 over 3m51s)   Warning   Unhealthy              Pod/metrics-server-5985cbc9d7-kmf2d            Readiness probe failed: HTTP probe failed with statuscode: 500
3m37s                   Normal    Completed              Job/helm-install-traefik                       Job completed
3m37s (x9 over 4m18s)   Normal    ApplyJob               HelmChart/traefik                              Applying HelmChart using Job kube-system/helm-install-traefik
3m36s                   Normal    Pulled                 Pod/svclb-traefik-11a7acdb-hzfsf               Successfully pulled image "rancher/klipper-lb:v0.4.9" in 5.232s (5.232s including waiting). Image size: 4990278 bytes.
3m36s                   Normal    Created                Pod/svclb-traefik-11a7acdb-hzfsf               Created container lb-tcp-443
3m36s                   Normal    Started                Pod/svclb-traefik-11a7acdb-hzfsf               Started container lb-tcp-80
3m36s                   Normal    Pulled                 Pod/svclb-traefik-11a7acdb-hzfsf               Container image "rancher/klipper-lb:v0.4.9" already present on machine
3m36s                   Normal    Created                Pod/svclb-traefik-11a7acdb-hzfsf               Created container lb-tcp-80
3m36s                   Normal    Started                Pod/svclb-traefik-11a7acdb-hzfsf               Started container lb-tcp-443
3m35s                   Normal    UpdatedLoadBalancer    Service/traefik                                Updated LoadBalancer with new IPs: [] -> [172.19.0.3]
3m25s                   Normal    Started                Pod/traefik-57b79cf995-hc72d                   Started container traefik
3m25s                   Normal    Created                Pod/traefik-57b79cf995-hc72d                   Created container traefik
3m25s                   Normal    Pulled                 Pod/traefik-57b79cf995-hc72d                   Successfully pulled image "rancher/mirrored-library-traefik:2.11.10" in 16.327s (16.327s including waiting). Image size: 47012064 bytes.
3m23s (x2 over 3m24s)   Normal    UpdatedLoadBalancer    Service/traefik                                Updated LoadBalancer with new IPs: [172.19.0.3] -> [172.19.0.3]
2m31s (x4 over 3m54s)   Warning   Failed                 Pod/coredns-ccb96694c-4m7bt                    Failed to pull image "rancher/mirrored-coredns-coredns:1.12.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/rancher/mirrored-coredns-coredns:1.12.0": failed to copy: httpReadSeeker: failed open: could not fetch content descriptor sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b (application/vnd.docker.container.image.v1+json) from remote: not found
2m31s (x4 over 4m13s)   Normal    Pulling                Pod/coredns-ccb96694c-4m7bt                    Pulling image "rancher/mirrored-coredns-coredns:1.12.0"
2m31s (x4 over 3m54s)   Warning   Failed                 Pod/coredns-ccb96694c-4m7bt                    Error: ErrImagePull
2m1s (x6 over 3m54s)    Warning   Failed                 Pod/coredns-ccb96694c-4m7bt                    Error: ImagePullBackOff
107s (x7 over 3m54s)    Normal    BackOff                Pod/coredns-ccb96694c-4m7bt                    Back-off pulling image "rancher/mirrored-coredns-coredns:1.12.0"

I hope this helps.

[ligfx]

Hmm, still works fine for me. What are the outputs of docker logs k3d-foobar-registry and docker logs k3d-foobar-server-0?

[cowwoc]

See the attached logs:

k3d-foobar-registry.log
k3d-foobar-server-0.log

[cowwoc]

It looks like that last line might be the problem...

2025/02/12 00:36:42 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/blobs/sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b?ns=docker.io - (26.721µs)

[cowwoc]

BTW, once I get into this bad state, recreating the cluster without "--disable-default-registry-endpoint" does not help. I keep on getting the same error.

I need to manually delete the "bad image" from the docker engine and only then can I recreate the cluster without "--disable-default-registry-endpoint" and it'll work.

[ligfx]

This sequence is interesting:

2025/02/12 00:36:03 Exporting Docker image rancher/mirrored-coredns-coredns:1.12.0 sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5
2025/02/12 00:36:03 {"status":"Downloading","progressDetail":{"current":24117248,"total":47557697},"progress":"[=========================\u003e                         ]  24.12MB/47.56MB","id":"3fef7b4658fe"}
2025/02/12 00:36:03 Wrote cache/blobs/sha256/2324f485c8db937628a18c293d946327f3a7229b9f77213e8f2256f0b616a4ee (3024 bytes)
2025/02/12 00:36:03 Wrote cache/blobs/sha256/82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5 (1272 bytes)
2025/02/12 00:36:03 Wrote cache/indexes/sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5/index.json (353 bytes)
2025/02/12 00:36:03 Wrote cache/blobs/sha256/aa732b10ee2c6132fbe5c455ffd4bc353a92a424299201b24f097a4f938bcbad (353 bytes)
2025/02/12 00:36:03 Wrote cache/indexes/sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5/manifest.json (1159 bytes)
2025/02/12 00:36:03 Wrote cache/blobs/sha256/be5ecfdad146d04bb0a1a9890424b63f6dc3aec87cf59e30beface839003b5ae (1159 bytes)
2025/02/12 00:36:03 Wrote cache/indexes/sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5/oci-layout (30 bytes)
2025/02/12 00:36:03 Wrote cache/blobs/sha256/18f0797eab35a4597c1e9624aa4f15fd91f6254e5538c1e0d193b2a95dd4acc6 (30 bytes)
2025/02/12 00:36:03 302 Found - HEAD /v2/rancher/mirrored-coredns-coredns/manifests/1.12.0?ns=docker.io - (17.713000346s)
2025/02/12 00:36:03 200 OK - HEAD /v2/rancher/mirrored-coredns-coredns/manifests/sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5?ns=docker.io - (121.265µs)
2025/02/12 00:36:03 302 Found - GET /v2/rancher/mirrored-coredns-coredns/manifests/1.12.0?ns=docker.io - (20.247004ms)
2025/02/12 00:36:03 200 OK - GET /v2/rancher/mirrored-coredns-coredns/manifests/sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5?ns=docker.io - (123.29µs)
2025/02/12 00:36:03 200 OK - GET /v2/rancher/mirrored-coredns-coredns/manifests/sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5?ns=docker.io - (74.273µs)
2025/02/12 00:36:03 200 OK - GET /v2/rancher/mirrored-coredns-coredns/manifests/sha256:2324f485c8db937628a18c293d946327f3a7229b9f77213e8f2256f0b616a4ee?ns=docker.io - (180.118µs)
2025/02/12 00:36:03 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/blobs/sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b?ns=docker.io - (42.152µs)

It downloads 1.12.0 which resolves to sha256:82979ddf442c593027a57239ad90616deb874e90c365d1a96ad508c2104bdea5. Then that manifest is retrieved twice (normal, k3d seems to always download the manifest, then also do a second download from its sha256 location, even if the first one was already a sha256 url).

But then it gets the manifest sha256:2324f485c8db937628a18c293d946327f3a7229b9f77213e8f2256f0b616a4ee, which we seem to have, but it was downloaded as an anonymous blob and not a manifest file 🤔

Could you upload the contents of both of those URLs (the last two that succeeded)? I wonder if the server needs to do a better job distinguishing blobs and manifests.

[pvannierop]

@ligfx I have the same problem. This does not pull for me:

Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366"

[cowwoc]

Actually, I'm increasingly sure that the problem is caused by tailscale/tailscale#1227

I'm going to switch to an alternative and let you know if the problem repeats itself.

@pvannierop Any chance you are also using tailscale or a similar technology that might mess with your networking?

[cowwoc]

@ligfx Unfortunately, I was wrong. I am getting this problem even without Tailscale.

I don't know what I changed, but now docker logs k3d-foobar-registry always returns 2025/02/13 00:01:08 Listening on :5000.

curl "http://localhost:53947/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io" returns:
curl: (52) Empty reply from server

Something is majorly wrong here but I can't figure out what... I tried removing the cluster, the registry, the network, all docker containers, volumes, images and restart docker. Same result. I'm going to try rebooting next...

[cowwoc]

Yeah... I'm stuck. Rebooting didn't help. I even went as far as reinstalling Docker Desktop (deleting any configuration files I could find, scattered across 5 different directories). In spite of that, the proxy doesn't work at all any more.

[cowwoc]

@ligfx Is there a way for me to enable verbose logging in the proxy? Curl says I'm getting a curl: (52) Empty reply from server but docker logs don't show the registry logging anything...

[cowwoc]

@ligfx Further investigation shows that the proxy is responding to requests from inside the container but not from outside it.

The container is bound to two different networks: bridge and k3d-foobar. It's possible that the server is binding to the wrong one.

I remember updating k3d recently. The only item in the the changelog is k3d-io/k3d#1550 and it relates to the registry so I suspect this broke the proxy for some reason....

[pvannierop]

@cowwoc

Any chance you are also using tailscale or a similar technology that might mess with your networking?

No, I am running this on a simple self-hosted Linux box with k3d. This self hosted system runs k3d from the nolar/setup-k3d-k3s helper like so:

  test-k8s-versions:
    needs: matrix-prep
    runs-on: self-hosted
    strategy:
      matrix: ${{ fromJson(needs.matrix-prep.outputs.matrix) }}
    steps:
      - uses: nolar/setup-k3d-k3s@v1
        with:
          version: ${{ matrix.K3S_VERSION }}
          k3d-args: --k3s-arg --disable=traefik@server:0 --k3s-arg --disable-helm-controller@server:0 --config .github/ci_config/k3d-config.yaml
          # Note: k3d adds a 'k3d-' prefix to this name
          k3d-name: ${{ matrix.KUBECTL_VERSION }}

with k3d version matrix:

[
  {
    "KUBECTL_VERSION": "v1.31.0",
    "K3S_VERSION": "v1.31.2+k3s1",
    "runOnBranch": "always"
  },
  {
    "KUBECTL_VERSION": "v1.30.0",
    "K3S_VERSION": "v1.30.6+k3s1",
    "runOnBranch": "main"
  }
]