[generator] add recursive component discovery for GitHub repositories

[ShigrafS]

Closes #726

Description

This PR implements recursive component discovery for Meshery’s generator logic, addressing the limitation where only files directly under a specified directory were processed. With this enhancement:

• Users can point to any root directory in a repository, and Meshery will automatically discover all component definition files in subdirectories.

• Recursive search is enabled by default, with configurable options for:

  • Maximum recursion depth (MaxDepth)
  • File pattern filtering (*.yaml, *.yml, *.json)
  • Enabling/disabling recursion

• Backward compatibility is maintained for existing workflows.

• Relevant CLI commands and API endpoints have been updated to support these options.

Changes include updates to:

• generators/generator.go & generator_test.go – Added GeneratorOptions for recursion and depth.
• generators/github/git_repo.go & package_manager.go – GitRepo and GitHubPackageManager updated to handle recursive directory traversal.
• generators/github/scheme_interface.go – Downloader updated to pass recursive options.
• utils/walker/git.go – File walker now respects recursion and max depth.
• Added new unit tests for recursive logic (recursive_test.go).

This fixes the configuration fragility for repositories with deeply nested component files and aligns with the desired behavior in #726 .

Notes for Reviewers

• The recursive logic uses filepath.WalkDir with max depth checks in utils/walker/git.go.
• Tests focus on option propagation and recursive field behavior; actual git clone testing may require integration testing.
• Backward compatibility with non-recursive workflows is preserved.
• Please verify the MaxDepth logic and that recursion is disabled when Recursive: false.

[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)

• Yes, I signed my commits.

---

[welcome]

Yay, your first pull request! 👍 A contributor will be by to give feedback soon. In the meantime, you can find updates in the #github-notifications channel in the community Slack.
Be sure to double-check that you have signed your commits. Here are instructions for making signing an implicit activity while performing a commit.

[gemini-code-assist]

Summary of Changes

Hello @ShigrafS, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves Meshery's ability to process component definitions from GitHub repositories by implementing recursive directory traversal. Users can now specify a root directory, and the system will automatically discover and process relevant files nested within subdirectories, overcoming previous limitations with deeply structured repositories. The new functionality is highly configurable, allowing fine-grained control over the depth of the recursive search, and has been integrated seamlessly while maintaining compatibility with existing operations.

Highlights

• Recursive Component Discovery: Introduced the capability for Meshery's generator logic to recursively discover component definition files within GitHub repositories, addressing the limitation of only processing files directly under a specified directory.
• Configurable Recursion Options: Added configurable options for recursive search, including a boolean flag to enable/disable recursion and an integer for maximum recursion depth (MaxDepth).
• Option Propagation and Integration: Updated the generator and github packages to correctly propagate the new Recursive and MaxDepth options through the system, from the initial generator creation down to the GitRepo and the underlying git file walker.
• Enhanced File Walker: The git file walker (utils/walker/git.go) has been enhanced to respect the MaxDepth setting during recursive directory traversal, using filepath.SkipDir to prune directories that exceed the specified depth.
• Backward Compatibility and Testing: Ensured backward compatibility for existing non-recursive workflows and added new unit tests to validate the correct propagation and behavior of the new recursion options.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces recursive component discovery for GitHub repositories, a valuable enhancement. The implementation adds options for recursion and depth control, and updates relevant parts of the generator logic. My review has identified a couple of high-severity issues: one is a bug in the recursion handling logic that can cause incorrect behavior, and the other is an off-by-one error in the max depth calculation which leads to counter-intuitive results. Additionally, the new tests for the recursive functionality are currently quite minimal and should be expanded to ensure the feature is robust. Overall, the changes are in the right direction, but these key issues should be addressed before merging.