Skip to content

Add index caching for use in development#4371

Open
backspace wants to merge 18 commits intomainfrom
cache-index-development-cs-10664
Open

Add index caching for use in development#4371
backspace wants to merge 18 commits intomainfrom
cache-index-development-cs-10664

Conversation

@backspace
Copy link
Copy Markdown
Contributor

@backspace backspace commented Apr 9, 2026

This adds a CI job on main that indexes built-in realms and stores an SQL dump of them for use in development. Adding INDEX_CACHE with mise run dev or …dev-all opts into this feature. This facilitates reviewing PRs locally that need the catalog or other large realms, which otherwise take 20+ minutes to index for me.

Here you see me running off an earlier commit (5ab4d6a20, if you want to try it yourself) where the script to download the cache would fetch the artifact from this branch instead of main, where it doesn’t yet exist:

screencast.2026-04-10.11-30-18.mp4

I had the host application running in another tab to remove the build time from the video.

@backspace backspace self-assigned this Apr 9, 2026
@backspace backspace added the enhancement New feature or request label Apr 9, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Realm Server Test Results

  1 files  ±0    1 suites  ±0   14m 12s ⏱️ +5s
844 tests +1  844 ✅ +1  0 💤 ±0  0 ❌ ±0 
915 runs  +1  915 ✅ +1  0 💤 ±0  0 ❌ ±0 

Results for commit 9497d9e. ± Comparison against base commit e549179.

This pull request removes 1 and adds 2 tests. Note that renamed tests count towards both.
default ‑ should handle streaming requests
default ‑ should fall back to generation cost API when inline cost is missing
default ‑ should handle streaming requests and deduct credits from inline cost

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Host Test Results

2 194 tests  +2 194   2 179 ✅ +2 179   2h 4m 18s ⏱️ + 2h 4m 18s
    1 suites +    1      15 💤 +   15 
    1 files   +    1       0 ❌ ±    0 

Results for commit 9497d9e. ± Comparison against base commit e549179.

♻️ This comment has been updated with latest results.

@backspace backspace marked this pull request as ready for review April 10, 2026 18:56
@habdelra habdelra requested a review from Copilot April 10, 2026 18:57
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 668297e252

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a CI-produced realm index cache artifact and an opt-in dev workflow (INDEX_CACHE=true) to import that cached index into an environment-mode database to avoid slow first-time indexing.

Changes:

  • Introduces a new CI job that boots realms, waits for readiness, and uploads a gzipped SQL dump of key index tables as an artifact.
  • Adds a dev-mode import script to download the latest cache artifact from main, optionally remap URLs for BOXEL_ENVIRONMENT, and import into the local DB.
  • Documents the new INDEX_CACHE workflow in the README.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
scripts/import-cached-index.sh New script to download CI artifact and import cached index into local Postgres (with URL remapping in env mode).
mise-tasks/lib/dev-common.sh Hooks cache import into environment-mode dev bootstrap when INDEX_CACHE=true.
mise-tasks/ci/cache-index New CI task to start services, wait for readiness, and dump index-related tables.
.github/workflows/ci.yaml Adds cache-index job on main to run the cache-index task and upload artifacts/logs.
packages/realm-server/package.json Adds wait-on dependency used by the new CI task.
pnpm-lock.yaml Lockfile updates for wait-on and transitive dependencies.
README.md Documents INDEX_CACHE usage and behavior.
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@backspace backspace requested a review from a team April 10, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants