Add the ability to use local optimization_config by mu-hun · Pull Request #276 · AdguardTeam/FiltersCompiler

mu-hun · 2026-04-24T06:35:34Z

continue from #275

Related: AdguardTeam/FiltersRegistry#1186

Summary

Rename: filterOptimization* → optimization* for consistency with existing naming in optimization.js
Local optimization config: New localOptimizationConfig namespace allows loading percent.json and per-filter stats.json from a local directory instead of fetching from remote URLs
- setPath(configPath), sets the local config dir; once set, getOptimizationPercent / getOptimizationConfig read locally
- downloadPercentJson(configPath) and downloadStatsFromPercentJson(configPath), downloads the percent.json and stats.json for each filter to the local dir for future use; per-filter writes run in parallel
- reset() — clears path and in-memory cache
Commit dist/: Built artifacts included so the package can be installed via GitHub shorthand URL (e.g. npm i AdguardTeam/FiltersCompiler#branch)

Test plan

Unit tests for localOptimizationConfig (setPath, downloadPercentJson, downloadStatsFromPercentJson reset) pass
Existing optimization, generator, and builder tests pass
dist/ artifacts match npm run build output

…ization' To ensure consistency with other names. Most variable and function names in that file start with 'optimization' without the 'filter' prefix.

mu-hun · 2026-04-24T06:37:03Z

commit singing missed 😅

Add unit tests covering `optimizationConfigLocal` and its members. Introduce `optimizationConfigLocal` namespace with three members: - `setPath(configPath)`: Sets the local config directory path. When set, `getOptimizationPercent` and `getOptimizationConfig` read from local files instead of downloading from remote URLs. - `generate(configPath)`: Downloads `percent.json` and per-filter `stats.json` from remote and saves them to `configPath`, creating `configPath/percent.json` and `configPath/filters/{filterId}/stats.json`. Per-filter file writes run in parallel via `Promise.all`. - `reset()`: Clears the local config path and nulls the in-memory `optimizationPercent` cache. Intended for restoring clean state between tests.

GitHub shorthand installation does not trigger a build step, so dist/ must be present in the branch directly. This is a temporary measure for the PR review period. dist/ should be removed from the branch before merging, and the version should be published to npm instead.

…onStats "Config" was ambiguous — the function returns per-filter stats (hit counts, groups), not configuration. "Stats" matches the source data (stats.json) and the new generateStats() method name.

…izationConfig BREAKING CHANGE: exported name changed. Update all import sites: - before: import { optimizationConfigLocal } from '@adguard/filters-compiler' - after: import { localOptimizationConfig } from '@adguard/filters-compiler' Noun-first naming matches the other module-level vars (localOptimizationConfigPath, optimizationPercent).

download percent.json locally first, enabling users to edit it Before stats are fetched. downloadStatsFromPercentJson() fills in missing stats via two-steps lookup: local file → network download. Local stats.json files are preserved (never overwritten), so user edits to stats directly affect the compiled output. getOptimizationConfig() reads from in-memory cache (populated by downloadStatsFromPercentJson()) when localOptimizationConfigPath is set, instead of re-reading the file on every call.

reset() now takes an explicit configPath instead of reading stored state. getOptimizationStats() simplified: cache hit → return, else download. BREAKING CHANGE: setPath(), getOptimizationPercent() no longer exported; reset() signature changed from reset() to reset(configPath).

mu-hun · 2026-05-20T09:52:00Z

Including dist/ temporarily for testing/review is understandable.

I will perform a force-push to exclude the changes in the dist/ directory. Separating them into a new commit makes the review process easier.

maximtop

I also tested the compiled package with optimization enabled. Platform generation fails on a non-optimized filter because the new code requests filters/10/stats.json and the server returns 404. Existing builder tests do not catch this because they call disableOptimization().

maximtop · 2026-05-20T11:06:39Z

+describe('local optimization config', () => {
+    let tmpDir;
+
+    afterEach(() => {


reset is async but this afterEach does not await or return it. Also, the nested test block declares another const tmpDir, so cleanup still uses the outer tmpDir and leaves the stats test directory behind.

Please make cleanup awaited and avoid shadowing tmpDir.

I applied your comment. Please see the changes.

By the way, should I keep using fs.promises in 'reset()` or switch to synchronous methods?

I mean that the others, like readFile, writeFile, and mkdtemp, are called either synchronously or asynchronously in the existing codebase.

In short: When should I call the node fs module synchronously or asynchronously in this project?

In this PR, I switched to using fs.promises today for consistency (d393a90) because I had been using both.

I applied your comment. Please see the changes.

By the way, should I keep using fs.promises in 'reset()` or switch to synchronous methods?

I mean that the others, like readFile, writeFile, and mkdtemp, are called either synchronously or asynchronously in the existing codebase.

In short: When should I call the node fs module synchronously or asynchronously in this project?

For this PR, fs.promises is the right direction. These helpers are already async and can run during build workflows, so async filesystem calls avoid blocking the event loop and keep the API consistent.

Sync fs is fine for tiny startup/config reads or test setup when it makes tests simpler, but new runtime/build-path code should generally prefer fs.promises.

Without the gate, every uncached filterId called filters/{id}/stats.json, which 404s for non-optimized filters. - Re-introduce optimizableFilterIds Set (lazily loaded from percent.json on first getOptimizationStats call) - Return null for filterIds not listed in percent.json - Restore groups array validation on downloaded stats - Populate optimizableFilterIds in downloadStatsFromPercentJson - Reset optimizableFilterIds to null in reset() - Extract PERCENT_JSON / STATS_JSON / FILTERS_DIR constants - Add JSDoc to localOptimizationConfig and getOptimizationStats

- Move async setup from describe callbacks into beforeAll/afterAll - Remove inner const tmpDir shadowing the outer let tmpDir - Await localOptimizationConfig.reset() in cleanup hooks - Add: getOptimizationStats returns null for unlisted filterId and does not call the stats endpoint

It currently works synchronously, but the code becomes misleading and unsafe for future refactoring.

mu-hun · 2026-05-21T10:35:07Z

I wrapped the download functions with async for consistency, because some parts were already called with await, and fixed the unsafe for future refactoring.

0b51ccb#diff-4a18d6d902a0c695e320cad11d7803a4d99d655e107051d5f5794a2c46509704R94

apply the review: AdguardTeam/FiltersRegistry#1186 (comment)

maximtop

One blocking repository-level issue remains: this PR is currently not mergeable with the current master. GitHub reports mergeable_state=dirty; the PR is still based on 6e0fa5a (v3.2.6), while current master is b75b738. Please rebase or merge master before final review, because package / Rollup / changelog areas have changed on master since this PR branched off.

maximtop · 2026-05-22T09:46:52Z

-    // config: [{filterId: 1, percent: 45}, ...]
-    const filterOptimizationPercent = getFiltersOptimizationPercent().config
-        .find((config) => config.filterId === filterId);
+    if (optimizationStatsCache[filterId] !== undefined) {


Cached local stats bypass the validation below. downloadStatsFromPercentJson() stores parsed local stats.json directly in optimizationStatsCache, and this early return sends it to callers without checking that groups exists and is non-empty.

Please apply the same validation to cached and downloaded stats, preferably via a small shared helper.

maximtop · 2026-05-22T09:46:52Z

+
+- `localOptimizationConfig` export for managing a local optimization config
+  cache: download `percent.json`, pre-fetch per-filter `stats.json`, and reset
+  cached state. Intended for offline / reproducible local filter builds.


Project rules require an [Unreleased] link reference when adding the ## [Unreleased] section. Please add it after the section entries, e.g.

[Unreleased]: https://github.com/AdguardTeam/FiltersCompiler/compare/v3.2.6...HEAD

maximtop · 2026-05-22T09:46:52Z

+    logPath: string,
+    reportPath: string,
+    platformsPath: string,
+    whitelist: number[],


These typings are stricter than the actual JS API. compile() callers can omit whitelist / blacklist and this code also accepts null, but the declaration requires number[].

Please make these arguments optional and nullable to match runtime usage, and do the same for customPlatformsConfig if null is expected.

Clean-up: Rename to begin with 'optimization' instead of 'filterOptim…

48bf377

…ization' To ensure consistency with other names. Most variable and function names in that file start with 'optimization' without the 'filter' prefix.

mu-hun requested a review from Alex-302 April 24, 2026 06:35

mu-hun self-assigned this Apr 24, 2026

mu-hun force-pushed the local_optimization_config branch from a69c0f0 to e54e49d Compare April 24, 2026 06:44

mu-hun added 2 commits April 27, 2026 16:50

mu-hun force-pushed the local_optimization_config branch from e54e49d to 615d6e0 Compare April 27, 2026 07:51

mu-hun added 2 commits May 7, 2026 14:19

fix: load stats file from local if exist

9283da7

upd: /dist

2231ebc

mu-hun mentioned this pull request May 7, 2026

Add the ability to use local optimization_config AdguardTeam/FiltersRegistry#1186

Open

adguard closed this May 12, 2026

adguard deleted the local_optimization_config branch May 12, 2026 10:34

Alex-302 restored the local_optimization_config branch May 13, 2026 07:51

Alex-302 reopened this May 13, 2026

adguard closed this May 14, 2026

adguard deleted the local_optimization_config branch May 14, 2026 11:31

Alex-302 restored the local_optimization_config branch May 15, 2026 17:58

Alex-302 reopened this May 15, 2026

mu-hun added 4 commits May 20, 2026 00:31

refactor(optimization): rename getOptimizationConfig to getOptimizati…

98e6f6b

…onStats "Config" was ambiguous — the function returns per-filter stats (hit counts, groups), not configuration. "Stats" matches the source data (stats.json) and the new generateStats() method name.

upd: dist/

3e24d93

mu-hun force-pushed the local_optimization_config branch from 238440f to 3e24d93 Compare May 20, 2026 09:52

maximtop reviewed May 20, 2026

View reviewed changes

mu-hun added 3 commits May 21, 2026 18:11

docs(changelog): add [Unreleased] entry for localOptimizationConfig

2df6b52

mu-hun added 3 commits May 21, 2026 18:26

refactor: wraps the download functions with async

0b51ccb

It currently works synchronously, but the code becomes misleading and unsafe for future refactoring.

refactor: calling fs module with async

d393a90

upd: dist/

c2b113c

mu-hun requested a review from maximtop May 21, 2026 10:28

mu-hun commented May 21, 2026

View reviewed changes

mu-hun added 2 commits May 21, 2026 21:44

types: creates type definition with all four public exports

ad1170d

apply the review: AdguardTeam/FiltersRegistry#1186 (comment)

upd: dist/

ce43c72

Alex-302 requested a review from piquark6046 May 22, 2026 06:37

maximtop reviewed May 22, 2026

View reviewed changes

adguard closed this May 22, 2026

adguard deleted the local_optimization_config branch May 22, 2026 10:14

Alex-302 restored the local_optimization_config branch May 24, 2026 21:49

Alex-302 reopened this May 24, 2026

Conversation

mu-hun commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

mu-hun commented Apr 24, 2026

Uh oh!

mu-hun commented May 20, 2026

Uh oh!

maximtop left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maximtop May 20, 2026

Choose a reason for hiding this comment

Uh oh!

mu-hun May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mu-hun May 21, 2026

Choose a reason for hiding this comment

Uh oh!

maximtop May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mu-hun May 21, 2026

Choose a reason for hiding this comment

Uh oh!

maximtop left a comment

Choose a reason for hiding this comment

Uh oh!

maximtop May 22, 2026

Choose a reason for hiding this comment

Uh oh!

maximtop May 22, 2026

Choose a reason for hiding this comment

Uh oh!

maximtop May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mu-hun commented Apr 24, 2026 •

edited

Loading

mu-hun May 21, 2026 •

edited

Loading