feat(parser): index HTML pages, their scripts and anchored sections by tirth8205 · Pull Request #1159 · tirth8205/code-review-graph

tirth8205 · 2026-11-06T18:59:49Z

What was wrong

A .html or .htm file gave nothing: no file node, no functions, no way to find a section of a long page. Functions in inline scripts could not be found, and a script file could not say which pages load it.

What changed

This follows the review on #521.

Inline scripts go to the JavaScript or TypeScript parser, like Vue and Svelte blocks, with the page's own line numbers.
Classic scripts on one page share one scope, so a block can call what another defines. A type="module" script keeps its names to itself.
Scripts that hold data (application/json, text/x-template and so on) are not read as code.
DocSection is a new node kind. A div, section or article with an id becomes a node named after its anchor (#cover), with a CONTAINS edge from the file.
Search finds a section by its anchor and first heading only. The body text is not indexed, so nothing here claims prose search. The heading is capped at 200 characters and a nested section keeps its own.
<script src> becomes an IMPORTS_FROM edge to the local file. URLs and template expressions add no edge. A local path that does not resolve keeps its raw text, as JS imports do.
An inline handler such as onclick="openSearch(1)" becomes a call, but only for functions the page defines. Otherwise those functions would look dead.
html joins the JavaScript language family, so callers_of can cross from a page to a script file.

Style blocks and heading anchors stay out, as the review asked.

Limits

Pages over 2 MB keep only their file node (html_truncated). They are most likely generated.
A page keeps at most 500 sections, also marked html_truncated.
A repeated id keeps the first element and counts the rest in html_id_occurrences. Browsers do the same, and two nodes with one name would collide.
An id made of a template expression ({{ item.id }}) is not an anchor.
An element never closed is still found, but its line range is only its opening tag.

An existing graph needs build to pick up pages, because update only re-reads changed files.

Left out

<a href> links between pages or to anchors.
Script src paths resolved through a web root other than the repo root. These keep their raw text.

Real numbers

Fresh builds from staging (115dccf) and from this branch, on copies of each repo.

Repo	Nodes	Edges	Database	Build
`pallets/flask`	1,669 → 1,700	8,697 → 8,764	21.16 → 21.32 MB (+0.8%)	2.4 s → 1.2 s
`h5bp/html5-boilerplate`	319 → 321	445 → 446	2.19 → 2.18 MB (no real change)	0.4 s → 0.4 s
`mdn/learning-area`	325 → 553	1,781 → 2,161	5.17 → 6.71 MB (+29.8%)	0.9 s → 0.7 s

Build times on runs this short are mostly noise. learning-area grows most because 154 of its 212 indexed files are HTML. It gains 42 DocSection nodes and 51 import edges from pages to scripts.

Code symbols in the top 10 search hits, before → after:

Repo	Terms
flask	`session` 10 → 10, `blueprint` 10 → 10, `render_template` 6 → 6, `config` 8 → 8, `login` 7 → 7
learning-area	`fetch` 5 → 5, `canvas` 0 → 0, `button` 0 → 3, `draw` 4 → 4, `game` 0 → 6
html5-boilerplate	`analytics`, `index`, `webpack`, `manifest`, `main`: all 0 → 0

No term got worse. The html5-boilerplate zeros mean nothing either way, since that repo gains only two file nodes (src/index.html and src/404.html) and one script import, and has almost no code.

Spot checks

I read the claimed source line for a random sample on learning-area: 5 DocSection, 5 Function, 5 CALLS, 5 IMPORTS_FROM and 5 CONTAINS, plus 3 each of Function, CALLS and CONTAINS on flask. All matched. Examples:

css/web-fonts/fonts/cicle_fina-demo.html:286 #glyphs, heading "Language Support", on <div id="glyphs">
accessibility/css/mouse-and-keyboard-events.html:30 showImg on function showImg() {
javascript/apis/third-party-apis/nytimes/finished/index.html → .../finished/script.js on <script src="script.js" defer></script>

Tests

New tests/test_html_parser.py (57 tests) and tests/fixtures/sample_html.html. They cover what the review listed: multiple and nested scripts, external and empty scripts, malformed HTML, duplicate ids, unquoted and case-varied attributes, .htm, line offsets, import and call edges, and full-build graph queries.
ruff and mypy pass.
Full suite: 6,445 passed, 2 failed. Both failures come from my short temp folder sitting under the home folder, which holds a .code-review-graph marker. Both pass from the normal temp folder. I then tidied type hints only, and re-ran the closest 718 tests, which pass. CI runs them all.

Fixes #521

🤖 Generated with Claude Code

A .html file gave nothing at all. Inline scripts now go to the JS/TS parsers with the page's own line numbers, and classic scripts on a page share one scope, so one block can call what another defines. A script src becomes an import of the local file, and an inline handler that calls a function on the page becomes a call. A div, section or article with an id becomes a DocSection node, connected to its file by CONTAINS, as #521's review asked. It holds the anchor and first heading, so that is what search can find; the body text is not indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-11-06T19:00:39Z

code-review-graph review

Overall risk: 0.65 (MEDIUM) — 72 changed function(s)/class(es), 53 affected flow(s), 15 test gap(s) (13 with no tested caller found, 2 reached only through a caller)

Risk-scored changes

Risk	Level	Symbol	Location	Tested
0.65	medium	code_review_graph/graph.py::GraphStore.search_edges_by_target_name	code_review_graph/graph.py:1325	yes
0.55	medium	code_review_graph/cli.py::build_parser	code_review_graph/cli.py:1068	yes
0.55	medium	code_review_graph/graph.py::GraphStore	code_review_graph/graph.py:710	yes
0.40	medium	code_review_graph/parser.py::_html_start_tag	code_review_graph/parser.py:3161	no
0.40	medium	code_review_graph/parser.py::_html_tag_name	code_review_graph/parser.py:3169	no
0.40	medium	code_review_graph/parser.py::_html_attributes	code_review_graph/parser.py:3176	no
0.35	low	code_review_graph/parser.py::_html_script_language	code_review_graph/parser.py:3198	no
0.35	low	code_review_graph/parser.py::_html_is_local_src	code_review_graph/parser.py:3208	no
0.35	low	code_review_graph/parser.py::_html_first_heading	code_review_graph/parser.py:3215	no
0.35	low	tests/test_html_parser.py::_git_repo	tests/test_html_parser.py:471	(test)

Affected execution flows

process — criticality 0.74, 665 node(s) across 18 file(s)
_run — criticality 0.74, 698 node(s) across 20 file(s)
build_or_update_graph_tool — criticality 0.73, 762 node(s) across 27 file(s)
refactor_tool — criticality 0.72, 165 node(s) across 12 file(s)
main — criticality 0.72, 1107 node(s) across 45 file(s)
...and 48 more affected flow(s)

Test gaps

code_review_graph/main.py::semantic_search_nodes_tool (code_review_graph/main.py:538)
code_review_graph/main.py::find_large_functions_tool (code_review_graph/main.py:681)
code_review_graph/main.py::cross_repo_search_tool (code_review_graph/main.py:1309)
code_review_graph/parser.py::_html_start_tag (code_review_graph/parser.py:3161)
code_review_graph/parser.py::_html_tag_name (code_review_graph/parser.py:3169)
...and 8 more with no direct test

Reached only through a caller

No test names these directly, but a tested caller reaches them along the call path shown. That is a path in the call graph, not a record of execution -- the caller's tests may never take this branch. Treat it as where to look, not as coverage: these count as gaps and are scored as untested.

code_review_graph/parser.py::CodeParser._extract_bytes (code_review_graph/parser.py:3587) — via code_review_graph/parser.py::CodeParser.parse_bytes, 1 hop(s)
code_review_graph/parser.py::CodeParser._parse_html (code_review_graph/parser.py:4251) — via code_review_graph/parser.py::CodeParser.parse_bytes, 2 hop(s)

Token savings: this graph-backed report used ~115,715 fewer tokens (~27%) than reading every changed file in full (estimated, chars/4 approximation).

Powered by code-review-graph — local-first analysis; no code leaves the CI runner.

Copilot

🟡 Changes recommended

Critical scope and call-resolution errors merge distinct module definitions, expose module-private functions, misresolve imported bindings, and create false or missing call edges.

7 open issues

What changed in this PR

Adds HTML pages to the code-review graph, making inline code and anchored document sections discoverable through existing search and graph queries.

Changes:

Extracts inline JavaScript/TypeScript, local script imports, handler calls, and anchored DocSection nodes.
Integrates HTML with cross-language caller queries, CLI filtering, and graph visualization.
Adds parser and integration tests, a representative fixture, and documentation.

File	Description
`tests/test_visualization.py`	Checks DocSection visualization support.
`tests/test_html_parser.py`	Covers HTML extraction, limits, search, and graph queries.
`tests/fixtures/sample_html.html`	Supplies scripts, handlers, and anchored sections for tests.
`README.md`	Adds HTML to supported formats.
`docs/USAGE.md`	Documents HTML support and DocSection nodes.
`docs/schema.md`	Defines DocSection metadata and relationships.
`docs/COMMANDS.md`	Documents DocSection query filters.
`docs/architecture.md`	Lists HTML among targeted parser formats.
`code_review_graph/visualization.py`	Adds DocSection styling and filtering.
`code_review_graph/parser.py`	Implements HTML detection and extraction.
`code_review_graph/main.py`	Updates tool documentation for DocSection filters.
`code_review_graph/graph.py`	Includes HTML in the JavaScript language family.
`code_review_graph/docs/LLM-OPTIMIZED-REFERENCE.md`	Adds HTML to the served language reference.
`code_review_graph/cli.py`	Accepts DocSection as a search kind.
`CHANGELOG.md`	Records HTML support, limits, and rebuild guidance.

_{Review effort: Balanced}

tirth8205 · 2026-11-09T15:57:33Z

+        }
+        for event, value, line in handlers:
+            seen: set[str] = set()
+            for called in _HTML_HANDLER_CALL_RE.findall(value):


Fixed in 2610e84: handlers are now read with the JavaScript grammar and only a call of a plain name counts, so strings, comments and obj . known() no longer link.

tirth8205 · 2026-11-09T15:57:35Z

+            self._extract_from_tree(
+                root, text, script_lang, file_path, nodes, edges,
+                import_map={**shared_imports, **import_map},
+                defined_names=shared_names | defined_names,


Fixed in 2610e84: a name a module imports is taken out of the page globals for that module, so its import wins.

tirth8205 · 2026-11-09T15:57:38Z

+            self._extract_from_tree(
+                root, text, script_lang, file_path, nodes, edges,
+                import_map={**shared_imports, **import_map},
+                defined_names=shared_names | defined_names,
+            )


Partly fixed in 2610e84: inline handlers now reach only classic-script functions, as they run in global scope. Giving each module block its own node identity changes how page nodes are named, so I have left that out of this PR.

tirth8205 · 2026-11-09T15:57:39Z

+            all_nodes.extend(nodes)
+            all_edges.extend(edges)


Fixed in 2610e84: script calls now go through _resolve_call_targets over what each block can see, so app.handle() reaches page.html::app.handle. Test added.

tirth8205 · 2026-11-09T15:57:42Z

+                if part.type == "text":
+                    words.append(part.text.decode("utf-8", errors="replace"))
+                else:
+                    parts.extend(reversed(part.children))


Fixed in 2610e84: entity leaves are kept and joined with no gap where the page has none, so Café reads Café and & alone gives &.

tirth8205 · 2026-11-09T15:57:44Z

+                        extra={"html_event": event},
+                    ))
+
+        if test_file:


Fixed in 2610e84: TESTED_BY edges are now made whenever the page has test nodes, as in the normal path, not only on test pages.

tirth8205 · 2026-11-09T15:57:47Z

+                candidate = self._repo_root / relative.lstrip("/")
+                if candidate.is_file():
+                    resolved = normalize_file_path(candidate)


Fixed in 2610e84: a root-relative src is now made canonical (.. stops at the root, as in a browser) and a forgotten file stays bare.

… a browser does - Inline handlers are read with the JavaScript grammar, so a name in a string, a comment or after a dot is no longer a call, and they reach only the page's classic functions, not a module's. - A name a module imports shadows a page global of that name. - Script calls go through the same-file resolver, so app.handle() reaches the app.handle the page assigns. - Headings keep their entities, so Café reads Café. - A test_ function on any page links what it calls, as in a .js file. - A root-relative src is made canonical, and a forgotten file stays bare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tirth8205 requested a balanced review from Copilot November 9, 2026 14:59

Copilot started reviewing on behalf of tirth8205 November 9, 2026 15:00

View session

Copilot AI reviewed Nov 9, 2026

View reviewed changes

tirth8205 mentioned this pull request Nov 9, 2026

Merge train: 43 reviewed PRs (2026-11-09) #1186

Merged

tirth8205 closed this pull request by merging all changes into staging in d01c1fd Nov 9, 2026

tirth8205 mentioned this pull request Nov 9, 2026

test: re-record the pinned corpus baselines for 2.4.0 #1197

Merged

Navigation Menu

Repository navigation

feat(parser): index HTML pages, their scripts and anchored sections - #1159

feat(parser): index HTML pages, their scripts and anchored sections#1159
2 commits merged into
stagingfrom
feat/521-html

tirth8205 commented Nov 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

tirth8205 Nov 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tirth8205 commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

What changed

Limits

Left out

Real numbers

Spot checks

Tests

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

code-review-graph review

Risk-scored changes

Affected execution flows

Test gaps

Reached only through a caller

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tirth8205 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tirth8205 commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading