fix: migrations for Windows

[lidel]

What?

• Aiming to fix 0.38.0: migration fails on Windows #11009:
  • add dedicated CI workflow for migration tests on Windows/macOS
  • workflow triggers on migration-related file changes only
  • confirm CI fails on Windows
  • fix Windows
  • add regression tests for lock-related behaviors

Why?

• We did not test migrations on Windows, tests only run on linux
  • Fix: Need to test migrations on all platforms
• LockedByOtherProcess works differently on Windows - it leaves behind a lock file that persists due to FILE_FLAG_DELETE_ON_CLOSE async behavior
  • Fix: Migrations used cctx.Context() instead of req.Context – the cctx one has side-effect of lazy-loading node, and as a side-effect, also creating a repo.lock. This, together with the FILE_FLAG_DELETE_ON_CLOSE behavior of Windows broke Windows.
  • See explainer inline.

[lidel]

Writing it down here as "Interesting Findings For Posterity^WLLM" :-)

The Two Contexts Explained

(1) req.Context - The Request Context (Standard Go Context)

What it is:

• A standard context.Context from Go's stdlib
• Created when the CLI command is invoked
• Carries request-scoped values like:
  • Cancellation signals (Ctrl+C)
  • Deadlines/timeouts
  • Request-specific metadata
• Lives for the duration of the command execution

Where it comes from:

  // From go-ipfs-cmds library
  type Request struct {
      Context context.Context  // ← This is req.Context
      Command *Command
      Path    []string
      Arguments []string
      Options OptMap
      // ...
  }

Purpose: Standard Go context for cancellation, timeouts, and request lifetime management.

(2) cctx.Context() - The Node Context (Lazy-Loading)

What it is:

• A method on Kubo's custom commands.Context struct
• Lazily constructs an IPFS node if it doesn't exist
• Returns the node's internal context

The implementation (from commands/context.go:98-107):

  func (c *Context) Context() context.Context {
      n, err := c.GetNode()  // ← CONSTRUCTS NODE if not cached!
      if err != nil {
          log.Debug("error getting node: ", err)
          return context.Background()
      }
      return n.Context()  // Returns the node's context
  }

What GetNode() does (lines 45-54):

  func (c *Context) GetNode() (*core.IpfsNode, error) {
      var err error
      if c.node == nil {
          if c.ConstructNode == nil {
              return nil, errors.New("nil ConstructNode function")
          }
          c.node, err = c.ConstructNode()  // ← CONSTRUCTS THE NODE
      }
      return c.node, err
  }

What ConstructNode does (from start.go:122-124):

ConstructNode: func() (n *core.IpfsNode, err error) {
    r, err := fsrepo.Open(repoPath)  // ← OPENS REPO AND ACQUIRES LOCK!
    // ...
    n, err = core.NewNode(ctx, &core.BuildCfg{Repo: r})
    return n, nil
}

Purpose: Provides access to the fully initialized IPFS node with all its subsystems (networking, DHT, bitswap, datastore, etc.).

Why This Caused the Bug

The Call Chain (BEFORE the fix):

  ipfs repo migrate
    ↓
  Run() handler (repo.go:404)
    ↓
  migrations.RunHybridMigrations(cctx.Context(), ...)  ← OLD CODE
    ↓
  cctx.Context() (context.go:99)
    ↓
  GetNode() (context.go:100)
    ↓
  ConstructNode() (start.go:117)
    ↓
  fsrepo.Open() (start.go:122)  ← ACQUIRES LOCK #1
    ↓
  lockfile.Lock(repoPath, "repo.lock")
    ✓ Lock acquired, added to go4.org/lock's global map
    ↓
  migrations.RunEmbeddedMigrations()
    ↓
  lockfile.Lock(repoPath, "repo.lock")  ← TRIES TO ACQUIRE LOCK #2
    ✗ ERROR: "lock is already held by us" (same process, global map check fails)

The Problem:

1. cctx.Context() has a side effect: It opens the repository
2. Opening the repo acquires the lock (fsrepo.go:165)
3. Migrations need the lock for themselves
4. Same process, same lock path → go4.org/lock's global locked map reports "already locked"

Why The Fix Works

The Call Chain (AFTER the fix):

  ipfs repo migrate
    ↓
  Run() handler (repo.go:404)
    ↓
  migrations.RunHybridMigrations(req.Context, ...)  ← NEW CODE
    ↓
  req.Context is just a plain context.Context
    ✓ No side effects, no node construction, no repo opening
    ↓
  migrations.RunEmbeddedMigrations()
    ↓
  lockfile.Lock(repoPath, "repo.lock")  ← FIRST AND ONLY LOCK ATTEMPT
    ✓ Lock acquired successfully
    ✓ Migrations run
    ✓ Lock released

[lidel]

ℹ️ we need to add this here, since we no longer use implicit locks provided by cctx.Context() (it was protecting via lock "by sheer luck", this is explicitly creating lock here, which is better/safer)

[guillaumemichel]

Nice!

[lidel]

We have end-to-end tests on all green on CI on all three platforms, but to be extra sure I've run manual tests:

Manual Windows smoke-test

Just to be extra safe, I've manually tested binary with fix here:

GOOS=windows GOARCH=amd64 go build -o cmd/ipfs/ipfs.exe ./cmd/ipfs

And then swapped ipfs.exe in failing repo from ipfs/ipfs-desktop#2998 and the error from #11009 is gone.

Migration finished correctly on Windows and v0.38 build started fine.

Manual Linux smoke-test

Created fresh repo (ipfs init) with kubo_v0.24.0_linux-amd64.tar.gz (produces repo verison 15), then started ipfs daemon with binary from this PR. It worked fine and executed both legacy external and then modern embedded migrations:

$ ipfs daemon                                                                                                                                                                        130 ...indows-migration-panic
Initializing daemon...
Kubo version: 0.38.0
Repo version: 18
System version: amd64/linux
Golang version: go1.25.1 X:nodwarf5
Kubo repository at /tmp/tmp.OTlsNtYqTO has version 15 and needs to be migrated to version 18.
Run migrations now? [y/N] y
Starting hybrid migration from version 15 to 18
Using hybrid migration strategy: external to v16, then embedded
Phase 1: External migration from v15 to v16
Looking for suitable migration binaries.
Need 1 migrations, downloading.
Downloading migration: fs-repo-15-to-16...
Fetching with HTTP: "https://trustless-gateway.link/ipfs/QmRzRGJEjYDfbHHaALnHBuhzzrkXGdwcPMrgd5fgM7hqbe/fs-repo-15-to-16/versions"
Fetching with HTTP: "https://trustless-gateway.link/ipfs/QmRzRGJEjYDfbHHaALnHBuhzzrkXGdwcPMrgd5fgM7hqbe/fs-repo-15-to-16/v1.0.1/fs-repo-15-to-16_v1.0.1_linux-amd64.tar.gz"
Downloaded and unpacked migration: /tmp/migrations223617285/fs-repo-15-to-16 (v1.0.1)
Running migration fs-repo-15-to-16 ...
  => Running: /tmp/migrations223617285/fs-repo-15-to-16 -path=/tmp/tmp.OTlsNtYqTO -verbose=true
applying 15-to-16 repo migration
locking repo at "/tmp/tmp.OTlsNtYqTO"
  - verifying version is '15'
> Upgrading config to new format
updated version file
Migration 15 to 16 succeeded
Success: fs-repo migrated to version 16.
Phase 2: Embedded migration from v16 to v18
Looking for embedded migrations.
Running embedded migration fs-repo-16-to-17...
applying 16-to-17 repo migration
> Upgrading config to use AutoConf system
updated version file
Migration 16-to-17 succeeded
Embedded migration fs-repo-16-to-17 completed successfully
Running embedded migration fs-repo-17-to-18...
applying 17-to-18 repo migration
> Migrating Provider and Reprovider configuration to unified Provide configuration
updated version file
Migration 17-to-18 succeeded
Embedded migration fs-repo-17-to-18 completed successfully
Success: fs-repo migrated to version 18 using embedded migrations.
Hybrid migration completed successfully: v15 → v18
PeerID: 12D3KooWLTLvxDNsA8KdbATivkp9qoJzycxkwDLQKXdvRLLvfZX1
Swarm listening on 127.0.0.1:4001 (TCP+UDP)

...

RPC API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
^C
Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)

Merging and starting release dance for v0.38.1 with this fix.