Add k3s 1.34 support with nix-snapshotter v0.4.0#175
Open
nuketownada wants to merge 21 commits intopdtpartners:mainfrom
Open
Add k3s 1.34 support with nix-snapshotter v0.4.0#175nuketownada wants to merge 21 commits intopdtpartners:mainfrom
nuketownada wants to merge 21 commits intopdtpartners:mainfrom
Conversation
The wrapper swallows PATH
Override k3s_1_34 to apply the nix-snapshotter integration patch, which adds the nix snapshotter plugin to k3s's embedded containerd. The patch targets nix-snapshotter v0.4.0 (which includes the gRPC forward-compatibility fix) and uses a clean one-line import of pkg/plugin — no vendored code patching needed. This overlay is temporary until the equivalent PR is merged upstream in k3s-io/k3s (reviving #9319). Co-Authored-By: Joshua Perry <josh@6bit.com>
Fail early with a helpful error message if nix-store is not available, rather than failing at image pull time with a cryptic exec error. Co-Authored-By: Joshua Perry <josh@6bit.com> Signed-off-by: Ada <ada@6bit.com>
Regenerate the k3s-nix-snapshotter patch from the rebased k3s commit:
- Rebased onto k3s main (v1.35.2)
- Fixed gofmt import ordering (nix import alphabetical position)
- Added exec.LookPath("nix-store") check to NixSupported()
Co-Authored-By: Joshua Perry <josh@6bit.com>
Signed-off-by: Ada <ada@6bit.com>
Regenerate patch against k3s_1_34 source (v1.34.3+k3s3) which has slightly different go.mod dependency versions (selinux v1.13.0, otiai10/copy v1.7.0) than the v1.35 branch used for the upstream PR. Signed-off-by: Ada <ada@6bit.com> Co-Authored-By: Joshua Perry <josh@6bit.com> Signed-off-by: Ada <ada@6bit.com>
The nix-snapshotter plugin needs the containerd gRPC address to initialize its image service. When running as an embedded containerd plugin in k3s, plugins.PropertyGRPCAddress may not be populated. Add containerd_address to the containerd config template so the plugin can find the backend CRI endpoint. Signed-off-by: Ada <ada@6bit.com> Co-Authored-By: Joshua Perry <josh@6bit.com> Signed-off-by: Ada <ada@6bit.com>
The containerd v2 Transfer API requires explicit unpack_config entries to know which snapshotter to use when unpacking images. Without this config, image pulls fail with "unable to initialize unpacker: no unpack platforms defined: invalid argument" (containerd/containerd#11606). Add unpack_config entries for both linux/amd64 and linux/arm64 to the containerd config templates when the nix snapshotter is selected. Co-Authored-By: Joshua Perry <josh@6bit.com> Signed-off-by: Ada <ada@6bit.com>
Two fixes for the k3s nix-snapshotter patch: 1. Use filepath.Join(nodeConfig.Containerd.State, "nix-snapshotter.sock") instead of hardcoded /run/nix-snapshotter/nix-snapshotter.sock. In rootless mode, /run is not writable for new directories, but the containerd state directory (/run/k3s/containerd/) is writable via rootlesskit's copy-up mount. This also works in privileged mode. 2. Add differ = "walking" to both unpack_config entries in the containerd config templates. Without explicit differ, containerd iterates diff plugins by platform matching, but the walking differ only registers for the host platform. The arm64 entry fails with "no matching diff plugins: not found". With explicit differ, containerd uses ID lookup which always succeeds. Co-Authored-By: Joshua Perry <josh@6bit.com>
Add vendorHash mapping for k3s 1.34.3+k3s3 alongside the existing 1.34.2+k3s1 mapping. The patch applies cleanly to both versions. Co-Authored-By: Joshua Perry <josh@6bit.com>
NRI (Node Resource Interface) tries to bind /var/run/nri/nri.sock which fails in rootless mode with permission denied. Add NRI disable to the V3 containerd config template when IsRunningInUserNS is true. Tested: k3s rootless with nix-snapshotter starts successfully, nix hello image pulls and runs with "Hello, world!" output. Co-Authored-By: Joshua Perry <josh@6bit.com> Signed-off-by: Ada <ada@6bit.com>
The k3s wrapper binary adds runc to PATH, but the rootless module uses .k3s-wrapped directly (to control PATH ordering for suid newuidmap). Without runc in PATH, containerd-shim-runc-v2 fails with "exec: runc: executable file not found in $PATH". Co-Authored-By: Joshua Perry <josh@6bit.com> Signed-off-by: Ada <ada@6bit.com>
Contributor
|
This is running rootless on my laptop successfully now. All the changes made to the k3s patch have been updated in the k3s-io/k3s#13676 PR and I added an integration test there that runs a nix-snapshotter image on the k3s CI by installing nix on ubuntu. |
nuketownada
force-pushed
the
k3s-1.34-support
branch
3 times, most recently
from
2270fb8 to
9569070
Compare
containerd v2.1+ runs checkIfCheckpointOCIImage on every CreateContainer
call to detect CRIU checkpoint images. When the image isn't yet in the
CRI image store, LocalResolve returns ErrNotFound and the function
propagates it as a hard error — failing CreateContainer entirely.
This breaks image service proxies (like nix-snapshotter) that bypass
containerd's CRI PullImage and load images directly into the content
store. The CRI image store is only populated asynchronously via event
handlers, creating a race between image loading and CreateContainer.
An image that is not found cannot be a checkpoint image. The patch
returns ("", nil) on ErrNotFound instead of propagating the error,
matching the existing behavior for empty input and local file paths.
Applied to the vendored containerd in the k3s build overlay via
preBuild, since k3s embeds containerd as a Go module dependency.
Co-Authored-By: Joshua Perry <josh@6bit.com>
Signed-off-by: Ada <ada@6bit.com>
nuketownada
force-pushed
the
k3s-1.34-support
branch
from
9569070 to
aba4731
Compare
Co-Authored-By: Joshua Perry <josh@6bit.com>
nuketownada
force-pushed
the
k3s-1.34-support
branch
from
8381e0c to
87d099d
Compare
Co-Authored-By: Joshua Perry <josh@6bit.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reduce go.mod context lines so the patch applies across k3s minor versions where dependency versions shift (selinux, otiai10/copy). Co-Authored-By: Joshua Perry <josh@6bit.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Joshua Perry <josh@6bit.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix go.mod patch hunk to use minimal context (survives dep version bumps) - Add vendorHash for k3s 1.34.5+k3s1 Co-Authored-By: Joshua Perry <josh@6bit.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lib.mdDocdeprecation fixes across all modules)k3s Overlay
The overlay intercepts
buildGoModulefork3s_1_34to apply a patch that adds nix-snapshotter as an embedded containerd plugin. This uses the cleanpkg/pluginimport from nix-snapshotter v0.4.0 — no vendored code patching needed.The same patch has been submitted upstream as k3s-io/k3s#13676. This overlay is temporary until that PR is merged and a k3s release includes it.
Test plan
nix build .#nix-snapshotterpassesnix build .#k3sproduces k3s v1.34.2+k3s1 with nix-snapshotter pluginnix flake check --no-buildevaluates all modules/packages (pre-existingdocker-distributionrename issue intest-push-n-pullunrelated to this PR)🤖 Generated with Claude Code