Skip to content

fix(db): serialize database init and migrations across processes - #44558

Closed
gitRasheed wants to merge 6 commits into
anomalyco:devfrom
gitRasheed:fix/db-migration-race
Closed

gitRasheed wants to merge 6 commits into
anomalyco:devfrom
gitRasheed:fix/db-migration-race

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Closes #33320

Type of change

  • Bug fix

What does this PR do?

I reproduced the race by starting six opencode run processes against a fresh global SQLite database. Five failed with "database is locked" within ~15ms.

Both drivers enable WAL before setting busy_timeout, and changing journal mode bypasses the busy handler. DatabaseMigration.apply uses an in-process semaphore around its check and write, so separate processes can run the same migration. Because one shipped migration resets the database, running it twice can silently delete session data. The migration transactions also used DEFERRED mode, which returns SQLITE_BUSY during a read-to-write upgrade without consulting busy_timeout.

I set busy_timeout before the first write in both drivers and retry WAL conversion within that budget. DatabaseMigration.apply now takes the existing cross-process Flock used by models-dev.ts, keyed by the path from pragma_database_list so aliases share a lock. Memory databases skip the lock. Migrations use BEGIN IMMEDIATE and recheck the journal inside the transaction, allowing raced callers to skip completed migrations.

#38849 stays open: this covers its startup crashes but not the mid-session ones, which come from a different write path. Its busy_timeout=0 reading is a per-connection artifact; the pragma is set, just bypassed here.

How did you verify your code works?

Two cross-process tests in test/database-migration.test.ts spawn real children on a file barrier: one races four processes at a fresh database, the other rewinds the last migration and asserts the journal ends with exactly one row. They fail unpatched; the manual six-process race went from 1/6 to 6/6 clean. Full core suite and typecheck pass.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Concurrent CLI processes racing one database file crash at startup:
the driver enables WAL at connection-open before any busy_timeout is
set, so the losers fail instantly with 'database is locked', and the
migration path guards its check-then-write with an in-process
semaphore only, letting two fresh processes both run schema.up — or
both run a pending migration, which for destructive migrations means
data loss.

Set busy_timeout in both drivers before the first write, and take the
cross-process Flock (already used for the models.dev cache race)
inside DatabaseMigration.apply, keyed by SQLite's own resolved path so
every caller is covered and path aliases collapse. Migration
transactions use immediate behavior so a read-to-write upgrade waits
on busy_timeout instead of failing at once.
@gitRasheed
gitRasheed force-pushed the fix/db-migration-race branch from aeffe0b to 7024062 Compare August 24, 2026 00:10
@Enough1122

Copy link
Copy Markdown

AI code review — automated review for reference, author can ignore or act on any point.

Overall: this addresses a real multi-process footgun thoroughly — cross-process flock keyed on the resolved DB path, immediate transactions so deferred read→write upgrades don't bypass busy_timeout, and a completion re-check inside each migration transaction. The barrier-based subprocess race harness with fail-fast on pre-barrier child exits is well built. Suggestions:

  1. packages/core/src/database/migration.ts:25 — the lock key is case-normalized only on win32. Default APFS on macOS is case-insensitive too, so /Data/x.db vs /data/x.db resolve to one file but take two different flocks, reintroducing the race there. Consider process.platform !== "linux" for the lowercase branch (or normalizing unconditionally).

  2. packages/core/src/database/sqlite.bun.ts:169-182 and packages/core/src/database/sqlite.node.ts:159-172 — the WAL retry loops are near-duplicates differing only in busy-error detection (.code === "SQLITE_BUSY" vs errcode & 0xff === 5) and the exec call. A tiny shared helper taking an isBusy(error) predicate would keep the deadline/backoff semantics from drifting between runtimes; right now a future tweak (e.g. jittered backoff) must be made twice.

  3. packages/core/src/database/migration.ts:24 — when pragma_database_list yields no file (:memory:), you skip flock and migrate directly; correct, but worth a word in the existing comment since readers may assume the lock is always held.

Minor: db-race-child.ts polls the barrier forever — bounded by the test's own 30s timeout, so fine, just noting. The completed.has() staleness concern is properly neutralized by the in-transaction SELECT 1 re-check (migration.ts, applyOnly) — nice detail. The node side also gains a real default busy timeout (previously undefined), which is a quiet but meaningful improvement on its own.

Default filesystems on Windows and macOS are case-insensitive, so two
spellings of one database path must map to one lock. Folding on
case-sensitive filesystems too only over-serializes two distinct files
for the duration of a migration.
@gitRasheed

Copy link
Copy Markdown
Contributor Author
  1. packages/core/src/database/migration.ts:25 — the lock key is case-normalized only on win32. Default APFS on macOS is case-insensitive too, so /Data/x.db vs /data/x.db resolve to one file but take two different flocks, reintroducing the race there. Consider process.platform !== "linux" for the lowercase branch (or normalizing unconditionally).

ty mr bot! took point 1 and lowercasing unconditionally.

@woodsonl

woodsonl commented Sep 7, 2026

Copy link
Copy Markdown

Related: #47567 retries SQLITE_BUSY statements at runtime (this PR covers init/migrations; that one covers the mid-session write path — e.g. Session.updateMessage, createUserMessage failures in #47566).

@github-actions

Copy link
Copy Markdown
Contributor

Automated PR Cleanup

Thank you for contributing to opencode.

Due to the high volume of PRs from users and AI agents, we periodically close older PRs using automated criteria so maintainers can focus review time on the most active and community-supported contributions.

This PR was closed because it matched the following cleanup criteria:

  • The PR was created more than 1 month ago
  • The PR had fewer than 2 positive reactions
  • Positive reactions are counted as thumbs-up, heart, celebration, or rocket reactions on the PR

PRs created within the last month are not affected by this cleanup.

If you believe this PR was closed incorrectly, or if you are still actively working on it, please leave a comment explaining why it should be reopened. A maintainer can review and reopen it if appropriate.

Thanks again for taking the time to contribute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent opencode run workers can fail at startup with database is locked

3 participants