Skip to content

Feat/generic yaml indexing - #1092

Merged
3 commits merged into
tirth8205:stagingfrom
EmilVorre:feat/generic-yaml-indexing
Nov 6, 2026
Merged

3 commits merged into
tirth8205:stagingfrom
EmilVorre:feat/generic-yaml-indexing

Conversation

@EmilVorre

@EmilVorre EmilVorre commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request

Linked issue

Closes #

What & why

How it was tested

uv run pytest tests/ --tb=short -q
uv run ruff check code_review_graph/
uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional

Checklist

  • Tests added for new functionality
  • All tests pass: uv run pytest tests/ --tb=short -q
  • Linting passes: uv run ruff check code_review_graph/
  • Type checking passes: uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional
  • Lines are at most 100 characters
  • Docs updated where behavior changed (README, docs/, docstrings)

@EmilVorre

Copy link
Copy Markdown
Contributor Author

I have taken the code from pr: #816

and merged main and fixed conflicts, this is done as the other pr seems to be going stale and i really need this feature.

Verification:

  • 4287 passed, 851 skipped, 2 xfailed, 2 xpassed
  • ruff passed
  • mypy passed
  • git diff --check passed

@tirth8205

Copy link
Copy Markdown
Owner

Thank you for this PR.

I built, updated and ran detect-changes on test repos with many YAML files.

Before I can merge it, please fix these:

  1. A block's line range runs onto the next key's line. _yaml_end_line returns end_mark.line + 1 (code_review_graph/parser.py:2895). Example: build:\n steps: [compile, test]\ndeploy:\n region: eu\n. $.build gets lines 1-3, but line 3 is deploy:. So adding deploy also marks $.build as changed. Failing test: test_appending_a_sibling_key_does_not_flag_the_previous_block.

  2. pnpm-lock.yaml is ignored only at the repo root (code_review_graph/incremental.py:259). packages/web/pnpm-lock.yaml is still indexed, with package names as nodes. Adding **/pnpm-lock.yaml fixes it. Failing test: test_pnpm_lockfiles_are_ignored_at_any_depth.

Both tests are in tests/test_generic_yaml_edge_cases.py on the branch review/pr-1092 in this repo. If a test there is marked xfail, please remove that mark with your fix.

After you push the fix, I will test it again.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Repeated expansion of shared YAML subtrees can stall indexing despite existing limits, while source-range, key-handling, metadata, and response-bounding defects remain unresolved.

8 open issues
What changed in this PR

Adds value-free generic YAML indexing to the knowledge graph, extending structural queries and change analysis while preserving specialized Spring and Ansible parsing.

Changes:

  • Introduces aggregated YamlPath nodes, duplicate-key diagnostics, and alias references.
  • Adds YAML metadata, change reporting, search filtering, and visualization support.
  • Adds regression coverage, updates documentation, and excludes pnpm-lock.yaml by default.
File Description
tests/​test_yaml_parser.py Adds parser, metadata, limits, and graph integration coverage.
tests/​test_visualization.py Checks YAML node and reference-edge visuals.
tests/​test_spring_config.py Updates expectations for generic YAML fallback.
tests/​test_incremental.py Checks the pnpm lockfile exclusion.
tests/​test_embedding_initialization.py Reorders Windows test setup.
tests/​test_changes.py Tests YAML occurrence-range change reporting.
tests/​test_ansible_parser.py Checks fallback for ordinary YAML.
tests/​conftest.py Clears inherited Git configuration environment variables.
README.md Advertises YAML indexing and change analysis.
docs/​USAGE.md Documents YAML support and node types.
docs/​schema.md Describes YAML metadata and identities.
docs/​COMMANDS.md Updates filters and change-analysis documentation.
code_review_graph/​visualization.py Adds YAML styling, filters, and reference-edge visuals.
code_review_graph/​tools/​review.py Exposes YAML change results and minimal-mode counts.
code_review_graph/​parser.py Implements generic YAML parsing and fallback routing.
code_review_graph/​main.py Updates tool docstrings for YAML support.
code_review_graph/​incremental.py Excludes pnpm-lock.yaml by default.
code_review_graph/​graph.py Serializes public YAML metadata.
code_review_graph/​docs/​LLM-OPTIMIZED-REFERENCE.md Updates the embedded language reference.
code_review_graph/​cli.py Adds the YamlPath search filter.
code_review_graph/​changes.py Maps YAML ranges and reports changed paths separately.

Review effort: Balanced

Comment on lines +7487 to +7489
elif isinstance(skipped_node, _YamlSequence):
for child in skipped_node.value:
consume_skipped_aliases(child, skipped_ancestors)
Comment on lines +783 to +786
_max_yaml_paths = env_int("CRG_MAX_CHANGED_YAML_PATHS", 200)
yaml_paths_truncated = len(changed_yaml_paths) > _max_yaml_paths
if yaml_paths_truncated:
changed_yaml_paths = changed_yaml_paths[:_max_yaml_paths]
original_target = shared_targets[id(value_node)]
reference_line = next_alias_line(value_node, line_start)
node_line_start = min(line_start, reference_line) if line_is_key else reference_line
node_line_end = max(node_line_start, line_end, reference_line)
yaml_merge: bool = False,
line_is_key: bool = False,
) -> None:
if not can_emit(schema_path, depth, exact_path, identity_path):
Comment on lines +7509 to +7510
signature = (str(getattr(key_node, "tag", "")), raw_key)
key_counts[signature] = key_counts.get(signature, 0) + 1
Comment on lines +7524 to +7525
key_tag = str(getattr(key_node, "tag", ""))
signature = (key_tag, raw_key)
"path": bounded_exact_path,
**duplicate_group,
})
is_merge = raw_key == "<<"
Comment thread docs/schema.md
### YamlPath
A path-addressable mapping key or sequence shape parsed from generic `.yml` / `.yaml` files. Repeated sequence entries are aggregated under `[*]` paths with bounded occurrence/range samples, allowing similarly shaped files to be compared without one graph node per list item. Scalar values are deliberately discarded.

The persisted `extra` JSON is the internal source of truth. Query, search, and change-analysis responses expose its safe public subset under one nested `yaml` object:
@tirth8205 tirth8205 closed this pull request by merging all changes into tirth8205:staging in 0160eb0 Nov 6, 2026
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
Adds tests/test_generic_yaml_edge_cases.py for generic YAML indexing:
- empty, comment-only, explicit-null, scalar and sequence roots, mixed types
- multi-document scoping and reused anchor names across documents
- key spellings that look like path syntax, unicode keys, BOM, CRLF, bad UTF-8
- Windows-style paths keep forward-slash identities with no dangling edges
- alias bombs (including unemittable anchors), depth and node limits
- duplicate-group caps, determinism, missing PyYAML, size limit boundary
- Ansible vars/meta precedence, Ansible-named non-Ansible fallback,
  Spring profile files, dbt project and schema YAML next to dbt models
- values never reach the SQLite file; spaces and unicode in file paths
- incremental update equals a fresh full build, rebuild is idempotent,
  deleted YAML files lose their paths
- detect_changes standard/minimal output, CRG_MAX_CHANGED_YAML_PATHS cap,
  truncated line samples, sibling-boundary precision
- node_to_dict with malformed metadata, CLI --kind YamlPath, nested
  pnpm-lock.yaml ignore

Two cases fail on the PR head: appending a sibling key reports the
previous block mapping as changed, and pnpm-lock.yaml is ignored only
at the repository root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
Mark the two confirmed defects as strict xfail so the branch stays green:
block collections ending on the next sibling key's line, and
pnpm-lock.yaml being ignored only at the repository root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
…eview of tirth8205#1092)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
…tection (review of tirth8205#1092)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
…1092)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
…205#1092)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
# Conflicts:
#	README.md
#	code_review_graph/docs/LLM-OPTIMIZED-REFERENCE.md
#	docs/USAGE.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants