Proposal to allow the use of Pixi workspaces through a named registry of workspaces

[ruben-arts]

Problem description

Pixi always works in a directory of a workspace, you can avoid that logic by defining it's --manifest-path however this still forces the users to take care of the location of that workspace.
conda users have shared the need for the manual creation of an environment and be able to activate it from anywhere on demand.

For example, conda users just want to create an environment and reuse it in multiple directories.

> conda create -n env python numpy scipy
> cd dir
> conda activate env
(env) > python main.py

This is not as easy with pixi as you first need to init a workspace and then manage that location yourself.

Initially we thought about recreating the full conda workflow using pixi's solve capabilities, but that would mean we had to maintain a complete extra set of commands and management workflow.

Proposal

Next to --manifest-path have --name to specify the workspace you want to work with.

Creating new named workspaces

Create $PIXI_HOME/workspaces/workspace and adds it to the named-workspaces
And adds the packages to the workspace and installs them.
Similar to conda create -n env mamba create -n workspace

pixi init --name env python numpy rattler-build

Adding packages to a named workspace

Adds python to the workspace in $PIXI_HOME/workspaces/env
Similar to conda install -n env python

pixi add --name env python

Register existing workspaces

Adds env to $PIXI_HOME/config.toml named-workspaces = { env = "/path/to/env" }

pixi workspace register env

Using named workspaces

Run a task in the env workspace

pixi run -n env task

Activate a shell in the env workspace

pixi shell -n env

Using activated named workspaces

After activating a shell in the workspace, that workspace is the one used by pixi.

> pixi shell --name env

(env) > pixi run task

The task is run in the env workspace

This solves:

Access your favorite environment from anywhere on your machine, pixi will keep the model of which name links to which workspace.
This would allow for a similar workflow experience to conda in pixi while keeping all the cool features pixi offers like pypi-dependencies and conda source dependencies with pixi build.

Questions:

• Does this give you everything that makes it hard to switch from conda to Pixi?
• Should we create a pixi create command or modify the pixi init command?

Related

Related to #4457 as it would allow you to immediately get the best binaries for your platform regardless of the default system-requirements we define for normal workspaces.

[synapticarbors]

This is great. It fits the request I made a long time ago and I think solves one of the sticking points for conda users. I've worked around the issue with a combination of custom shell scripts, or just by using atuin to quickly navigate my shell history to find the correct pixi shell --manifest-path command. But having a native solution will lower the barrier for on-boarding people who are hesitant.

[dhirschfeld]

I think this is great!

My current workaround involves an activate shell function which will create projects in a well-known location which I think is similar in concept.

activate() {
    local name="$1"
    local env="${2:-default}"
    history -a  # ensure the new shell starts with the same history
    local manifest_path="$HOME/.pixi/shared/${name}"
    pixi shell --manifest-path "${manifest_path}" --environment "${env}"
}

Note

I keep the shared environments under ~/.pixi/shared/ as ~/.pixi/envs is taken by global environments. I think it might make more sense to have global environments under ~/.pixi/global/.

[mfisher87]

Initially we thought about recreating the full conda workflow using pixi's solve capabilities, but that would mean we had to maintain a complete extra set of commands and management workflow.

My initial reaction to this path was negative, but I do think there's value to uv's approach of providing a new and better interface plus compatibility with the way folks are used to doing things. There are a lot of people that I think would be quickly convinced to switch if Pixi can offer "conda's interface but up to 10s of times faster", and then those folks can incrementally discover the benefits of the project-driven workflow without having to install a new thing. tldr; I like the idea of both pixi init --name myenv python and pixi conda create -n myenv python basically doing the same thing.

There are so many JupyterHubs out there which run on conda environments, and many of them build very broad environments to support broad scientific use cases. E.g. CryoCloud. Some of those images take 40-60 minutes to build with conda/mamba, and I've built the same environment with pixi in 60-90 seconds. Having a conda-like named environment experience and interface would enable more seamless transitions for those hubs to a faster and more environmentally-conscious build system.

EDIT: To be more specific about what I don't like -- users building environments without a manifest. This is really common in the conda world, and I've seen people get in trouble for this reason too many times :) Managing an environment.yml by hand is extremely unattractive to some users, and without one it's really hard to have a coherent mental model for what's in the environment. It becomes an "out of sight, out of mind" thing ... until it doesn't work anymore, at which point it's hard to determine why it failed and hard to rebuild it. This is what drives me in trying to convert this category of user over to pixi ... it's good for their sanity (and mine 😉).

[dhirschfeld]

I'm personally not fussed on replicating the exact conda CLI as I've fully switched to pixi, but I can see how it could ease transitioning for others with existing workflows. It could potentially be implemented as a pixi-conda extension.

[noamgot]

First thoughts:

1. looks like a nice idea that will make transforming from conda easier
2. pixi create sounds reasonable in this case, keeping the init for the usual use-cases
3. where do we store the manifests and lock files of the named environments?

Another tiny thing that I want to point out - -n is currently a short flag of pixi run. I added it when I added the dry-run feature :)

TBH, at the time I did not like it, but I defined this short flag to keep in line with other pixi commands that use this as the short flag of --dry-run; My practical suggestion would be replacing it with -d (which is, of course, a breaking change...); This will free the -n for the use-case described here

[ruben-arts]

Hey @noamgot

where do we store the manifests and lock files of the named environments?

They will be named workspaces similar to your current workflow but in a global location. e.g.:

.pixi/workspaces/my-env
├── .gitattributes
├── .gitignore
├── .pixi
│   ├── .condapackageignore
│   ├── .gitignore
│   └── envs
│       └── default
│           ├── bin
│           ├── conda-meta
│           ├── include
│           ├── lib
│           ├── man
│           ├── share
│           └── ssl
├── pixi.lock
└── pixi.toml

[jaimergp]

That's awesome. One piece of advice from conda though: let's not use the name of the directory as the name of the environment. It brings little value and complicates environment renames and whatnot. We should try to resurrect this idea: conda/ceps#67.

An alternative idea is to let go of "names" as the primitive, and instead allow arbitrary target prefixes. Then one pre-configured location (but customizable) could be aliased to a shorthand.

$ pixi init --prefix my/arbitrary/path --with python
$ pixi init --prefix ~/.pixi/named-environments/name --with python 
$ pixi init --prefix @name --with python  # same as above, the value for @ is taken from the config

I'm suggesting this from the perspective of sometimes needing an environment to be in a particular path so it integrates with specific infrastructure (e.g. in some conda-forge workflows, the environment is expected in a particular mount point).

[elazarcoh]

I got here from #3541, but hopefully what I'm writing is related here as well.

Our team workflow & setup (which may or may not be optimal, but that's what we do), please bear with me:

1. Multiple machines sharing filesystem using NFS
2. Currently we are using conda, and each member has it's own conda env and own repo clone.
3. Both conda envs and source-code are placed in a some NFS shared path, e.g. /path/to/envs, /path/to/repos/ (note, we do not share the actual code/envs, only it's "parent" folder).
4. Projects (e.g. the repo clone) is installed to respective conda env as editable,
   e.g. conda activate /path/to/envs/mine; pip install -e /path/to/repos/mine

The repos and envs do not sit on the same filesystem, so the solution to just let pixi use the project folder for envs won't suffice. Think mount machine1:/envs /path/to/envs; mount machine2:/repos /path/to/repos.

It is being hell to manage and sync the environments (I haven't talked about multi-platform even), that why I am seeking for alternatives.

So, my use-case with pixi will be to create an environment using a manifest into a custom location.
The workflow I'm envisioning:

git clone my-repo
cd my-repo
pixi create --prefix /path/to/envs/mine-pixi  # This will use the repo manifest
# to activate
pixi activate /path/to/envs
# or simply, if we have support for .env or some local .pixirc that is gitignored 
# that would have `defaultenv=/path/to/envs/mine-pixi` in it:
pixi activate

The /path/to/envs/mine-pixi env would be aware of the manifest attached to it. If the manifest is missing (e.g. deleted), pixi will throw errors when trying to add dependencies etc., though the environment itself might be still useable to run python scripts.