Skip to content

[hermes] Add some tests for Charon invocation#3015

Open
joshlf wants to merge 1 commit intoGe1bd84b5d857e18e3668e809844c6d73b808a020from
G60bb1cd41a5fade3144a05f2bdafb2a670059e03
Open

[hermes] Add some tests for Charon invocation#3015
joshlf wants to merge 1 commit intoGe1bd84b5d857e18e3668e809844c6d73b808a020from
G60bb1cd41a5fade3144a05f2bdafb2a670059e03

Conversation

@joshlf
Copy link
Member

@joshlf joshlf commented Feb 8, 2026


Latest Update: v11 — Compare vs v10

📚 Full Patch History

Links show the diff between the row version and the column version.

Version v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v11 v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v10 v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v9 v8 v7 v6 v5 v4 v3 v2 v1 Base
v8 v7 v6 v5 v4 v3 v2 v1 Base
v7 v6 v5 v4 v3 v2 v1 Base
v6 v5 v4 v3 v2 v1 Base
v5 v4 v3 v2 v1 Base
v4 v3 v2 v1 Base
v3 v2 v1 Base
v2 v1 Base
v1 Base
⬇️ Download this PR

Branch

git fetch origin refs/heads/G60bb1cd41a5fade3144a05f2bdafb2a670059e03 && git checkout -b pr-G60bb1cd41a5fade3144a05f2bdafb2a670059e03 FETCH_HEAD

Checkout

git fetch origin refs/heads/G60bb1cd41a5fade3144a05f2bdafb2a670059e03 && git checkout FETCH_HEAD

Cherry Pick

git fetch origin refs/heads/G60bb1cd41a5fade3144a05f2bdafb2a670059e03 && git cherry-pick FETCH_HEAD

Pull

git pull origin refs/heads/G60bb1cd41a5fade3144a05f2bdafb2a670059e03

Stacked PRs enabled by GHerrit.

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @joshlf, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the testing infrastructure for the hermes tool by introducing several new test fixtures for Charon invocation. These additions cover critical scenarios such as handling artifact name collisions in workspaces, gracefully managing projects without Lean specifications, and enabling precise selection of binary or library targets for verification. The overall impact is improved robustness and reliability of Charon's analysis capabilities across diverse Rust project configurations.

Highlights

  • New Test Fixtures for Charon Invocation: This pull request introduces a comprehensive set of new test fixtures designed to validate various invocation scenarios for the Charon tool within hermes.
  • Collision Test Case: A new test fixture (collision) has been added to specifically test Charon's behavior when multiple crates within a workspace attempt to define libraries with the same artifact name, ensuring proper handling of such conflicts.
  • No Specs Test Case: A no_specs test fixture verifies Charon's response when no Hermes annotations (Lean blocks) are present in the source code, expecting a failure status and a specific error message.
  • Target Selection Tests: Two new test fixtures, select_bin and select_lib, have been added to test Charon's ability to verify specific binary (--bin) or library (--lib) targets within a Rust project, ensuring granular control over the verification process.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • tools/hermes/tests/fixtures/collision/args.txt
    • Added verify --workspace argument to test Charon's behavior in a multi-crate workspace.
  • tools/hermes/tests/fixtures/collision/expected/crates/a/Cargo.toml
    • Added Cargo.toml for pkg_a to define a library named common_utils.
  • tools/hermes/tests/fixtures/collision/expected/crates/a/src/lib.rs
    • Added a dummy lib.rs for pkg_a.
  • tools/hermes/tests/fixtures/collision/expected/crates/b/Cargo.toml
    • Added Cargo.toml for pkg_b to define a library also named common_utils, creating a collision scenario.
  • tools/hermes/tests/fixtures/collision/expected/crates/b/src/lib.rs
    • Added a dummy lib.rs for pkg_b.
  • tools/hermes/tests/fixtures/collision/source/Cargo.toml
    • Added workspace Cargo.toml to include pkg_a and pkg_b.
  • tools/hermes/tests/fixtures/collision/source/crates/a/Cargo.toml
    • Added Cargo.toml for pkg_a in the source directory.
  • tools/hermes/tests/fixtures/collision/source/crates/a/src/lib.rs
    • Added lib.rs for pkg_a in the source directory.
  • tools/hermes/tests/fixtures/collision/source/crates/b/Cargo.toml
    • Added Cargo.toml for pkg_b in the source directory.
  • tools/hermes/tests/fixtures/collision/source/crates/b/src/lib.rs
    • Added lib.rs for pkg_b in the source directory.
  • tools/hermes/tests/fixtures/no_specs/expected_status.txt
    • Added failure as the expected status for projects without Lean specifications.
  • tools/hermes/tests/fixtures/no_specs/expected_stderr.txt
    • Added No Hermes annotations as the expected standard error output for projects without Lean specifications.
  • tools/hermes/tests/fixtures/no_specs/source/Cargo.toml
    • Added Cargo.toml for the no_specs test case.
  • tools/hermes/tests/fixtures/no_specs/source/src/lib.rs
    • Added lib.rs for the no_specs test case, intentionally without Lean blocks.
  • tools/hermes/tests/fixtures/select_bin/args.txt
    • Added verify --bin app argument to test selecting a specific binary target.
  • tools/hermes/tests/fixtures/select_bin/expected/src/main.rs
    • Added expected main.rs content for the selected binary target.
  • tools/hermes/tests/fixtures/select_bin/source/Cargo.lock
    • Added Cargo.lock for the select_bin test case.
  • tools/hermes/tests/fixtures/select_bin/source/Cargo.toml
    • Added Cargo.toml for the select_bin test case, explicitly defining an app binary and disabling autolib.
  • tools/hermes/tests/fixtures/select_bin/source/src/lib.rs
    • Added lib.rs for the select_bin test case.
  • tools/hermes/tests/fixtures/select_bin/source/src/main.rs
    • Added main.rs for the select_bin test case, defining the app binary.
  • tools/hermes/tests/fixtures/select_lib/args.txt
    • Added verify --lib argument to test selecting a specific library target.
  • tools/hermes/tests/fixtures/select_lib/expected/src/lib.rs
    • Added expected lib.rs content for the selected library target.
  • tools/hermes/tests/fixtures/select_lib/source/Cargo.toml
    • Added Cargo.toml for the select_lib test case.
  • tools/hermes/tests/fixtures/select_lib/source/src/lib.rs
    • Added lib.rs for the select_lib test case, defining the library.
  • tools/hermes/tests/fixtures/select_lib/source/src/main.rs
    • Added main.rs for the select_lib test case.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a valuable set of test fixtures for various hermes invocation scenarios, covering crate/target selection, name collisions, and cases with no annotations. These tests significantly improve the coverage of the command-line interface and project resolution logic. The fixtures are well-organized and clear. I have one suggestion to enhance the robustness of a test by using a more specific expected error message.

@@ -0,0 +1 @@
No Hermes annotations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The expected stderr message is very generic. The actual error message produced is "No Hermes annotations (/// ```lean ...) found in the selected targets. Nothing to verify.". While the test likely passes using a substring match, making the expected error more specific would make the test more robust. This prevents the test from passing incorrectly if other error messages are added in the future that also contain this short substring.

No Hermes annotations (/// ```lean ...) found in the selected targets. Nothing to verify.

@codecov-commenter
Copy link

codecov-commenter commented Feb 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.87%. Comparing base (3434d56) to head (8f20400).

Additional details and impacted files
@@                            Coverage Diff                             @@
##           Ge1bd84b5d857e18e3668e809844c6d73b808a020    #3015   +/-   ##
==========================================================================
  Coverage                                      91.87%   91.87%           
==========================================================================
  Files                                             20       20           
  Lines                                           6057     6057           
==========================================================================
  Hits                                            5565     5565           
  Misses                                           492      492           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@joshlf joshlf force-pushed the G60bb1cd41a5fade3144a05f2bdafb2a670059e03 branch 3 times, most recently from 41abe7f to 8391931 Compare February 8, 2026 04:38
gherrit-pr-id: G60bb1cd41a5fade3144a05f2bdafb2a670059e03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants