Project focused nupm add/update backed by git repos

[vyadh]

It seems worth detailing what I plan to start working on so peeps can chip in early if I'm going a bit off the rails. Essentially, I thought I would start with the add/update part of the design document, which would help avoid touching the existing install functionality until we have something more fleshed out.

This project-centric mechanism, if it works well, could be widened back out to a user-level context, with perhaps a "global" project metadata file inside $env.XDG_CONFIG_HOME/nushell/nupm.

Here's some ideas of how the eventual functionality might work. The command interface here being just a placeholder for illustrative purposes.

# nupm add <url> [--dev] [--ref branch/tag/commit/version]
nupm add https://github.com/vyadh/nutest.git --dev

This will:

• Clone the git repo into nupm's cache directory if it doesn't exist, or do a fetch if it does
• Read version and commit hash information by querying Git tags and selecting the latest via normal semver rules (that could be fun) unless an explicit version has been specified
• Read project information from that version's nupm.nuon
• Add the package in in the relevant dependency section of the project's nupm.nuon (development dependencies or release)
• Add resolved versions and git hashes to the 'nupm.lock', which will be needed to "fix" resolved transitive dependencies
• Create a link to the repo at the associated commit for use by nuun and Nushell overlay activation
  • We could use git worktree for this, which would allow all usage to leverage a single git repository (only one to store/clone/update)

# nupm update [package name] [--upgrade]
nupm update nutest  # need to think about naming collisions

This will:

• Fetch any updates to the cached nutest git repo(s)
• Re-scan for tags and reevaluate latest version based on bounds set in project nupm.nuon
• If --upgrade is specified, ignore version bounds and update nupm.nuon dependencies to latest version(s) found
• Reflect the specific versions found as well as the associated git hashes in nupm.lock
• Add/reuse a working copy from git worktree for the version required (these do need to be garbage collected somehow, so maybe best done later)

The details of above may change as I flagrantly copy existing package manager ideas.

Initial deliverable:
A basic implementation of add/update to prove out the functionality that we can hopefully slot into nupm if it works out. Unlikely all of above or even finish what I started to be honest 😬. I could also raise PRs against nupm directly, but would seem a bit noisy while the details are being worked out?

Out of initial scope:

• nuun virtual environments to activate, though it seems good to integrate this soon for the functionality to become usable
• Transitive dependencies
• Version wildcards (e.g. latest bugfix version - another reason we will need nupm.lock to track the actual resolved versions)
• Script or plugin-based repos
• Registries, though we will probably need eventually. Perhaps the above mechanism achieves the same thing as personally-maintained registries? You conceivably just need to add the repo to any project to add it to your personal registry.
• Monorepos. We should design for them to slot in seemlessly. Perhaps by providing some kind of optional context on add, that being / if unspecified. The awkward bit as discussed on Discord would be how to express sub-module versions with git tags for simple lookup and addressing. A quick look suggests a common version tagging approach for monorepos is path-based tags such as <submodule/...>/<version>. If that's how it's normally done, seems a good way to start.

[vyadh]

I'm going to close this issue for now. There are a lot of things to consider about a distributed dependency manager that the above doesn't really account for. I think I need to map things out in a design document before I jump too far into implementation. That way we can carve out useful increments that are progress in mind towards a cohesive whole.