Update documentation

crates/pet-conda/README.md

@@ -0,0 +1,101 @@
+# Conda
+
+## Notes
+
+- Structure of conda envionment
+  - A conda install folder will `generally` have a default environment with Python (except for `micromamba` and possibly others)
+  - the `envs` sub directory in a conda installation contains all conda environments belonging to that conda installation
+- Conda environments
+  - This is computed from a list of hardcoded locations (for windows, linux, macos)
+  - This includes locations in `.condarc` file(s)
+  - This includes locations in `environments.txt` file
+  - This includes all sub directories under the `env` directory in the all known conda installation folders
+- `.condarc`
+  - The `.condarc` file can be configured to store conda environments in other locations instead of `<conda install folder>/envs`
+  - There can be multiple conda installations on a machine, each with its own set of environments
+  - There can be multiple directories defined in `.condarc` file to store conda environments
+  - There can be multiple `.condarc` files
+- `environments.txt`
+  - This contains all known conda environments on the current machine (including base env, which is the same as install location of conda)
+  - The directories returned by `conda info --json` are the directories where conda environments are stored
+- Version of conda can be found in `<conda install folder>/conda-meta/conda-<version>.json` file
+- User can install multiple versions (installations) of conda,
+  - Eg. install anaconda, miniconda, miniforge, micromamba, etc.
+  - In usual (default) locations & also in custom locations
+  - Also can be installed via pyenv, in which case the conda installations are located under `~/.pyenv/versions/<conda install version>`
+- `conda-meta\<package name>-<version>.json` files
+  - This contains information about all conda packages installed into a conda environment.
+  - The root (base) conda environment too has such a directory.
+  - Given `conda` is a package in its own right, the version of conda can also be found in the `conda-meta` directory.
+    (after all, you can update conda using `conda update conda`).
+  - Similarly if Python is installed in a conda environment, the version of Python can be found in the `conda-meta` directory.
+- `conda-meta\history` file
+  - This contains a history of all installations of this conda environment.
+  - One key information is the command used to create the conda environment itself.
+  - The entry looks like so `# cmd: /Users/donjayamanne/.pyenv/versions/miniconda3-latest/bin/conda create -n myenv python=3.10`
+  - Thus using the `history` file we can find the conda installation folder.
+    This is useful in cases where conda environments are created using `-p` option.
+
+## Known issues
+
+- Note: pixi seems to use conda envs internall, hence its possible to falsely identify a pixi env as a conda env.
+- However pixi is not supported by this tool, hence thats not a concern.
+
+## Miscellanous
+
+- What if conda is installed in some custom locations that we have no idea about?
+
+In such cases the assumption is that the `environments.txt` file will contain an entry to the base env.
+Using that information we can get the conda directory and get the conda exe and version info.
+
+Even if `environments.txt` file is empty, we will look for environments in known locations and from there we can find the conda install folder (recall `history` file).
+
+- What if we have a custom conda env created in current workspace folder, and we do not know where Conda is installed?
+
+In such cases we can just inspect the `conda-meta/history` file in the conda env folder and get the conda installation folder.
+
+- How do we generate command to run Python in an environment?
+
+If the Conda env is in the `envs` folder, then use `<conda exe> -n <env name> python`.
+If the Conda env is the root (base) folder, then use `<conda exe> -n base python`.
+For all other cases use `<conda exe> -p <fully qualified path to folder> python`.
+
+## Pseduo code for algorithm
+
+```rust
+// Step 1
+// Get a list of all known directories where conda environments can be found
+// 1. environments.txt file
+// 2. .condarc file(s)
+// 3. Other known locations
+
+// Step 2
+// We hardcode some of the commonly known install directories of conda, miniconda, miniforge, etc for all platforms.
+for known_install_folder in [<home>/anaconda3, <home>/miniconda3, etc]:
+    conda_exe = "<known_install_folder>/bin/conda" // (windows is slightly different)
+    conda_version = "<known_install_folder>/conda_meta/conda-<version>.json"
+    python_exe = "<known_install_folder>/bin/python" // (windows is slightly different)
+    python_version = "<known_install_folder>/conda_meta/conda-<version>.json"
+
+    // Step 2.1
+    // We now have conda exe, version, default python information
+    // Conda run command is computed as `[<fully qualified path to conda_exe>, run, -n, <name> python]`
+    for env in `<known_install_folder>/envs`:
+        // env is a conda environment
+        // Find python exe and version in the conda-meta directory
+        // If python is not found, then this is a conda env without Python.
+        // These are named environments that are activated (run) using the `-n` option.
+
+    // Previously we captured a list of all known conda envs
+    // Go through those one by one and inspect the conda-meta/history file
+    // And check whether that env was created by this current conda installation
+    // If so, then associate that env with this conda installation
+    // Next remove that env folder from the list captured in step 1.
+
+// Step 3
+// Finally go through all of the remaining conda envs that were captured in step 1
+// & did not get removed by step 2.1.
+// Go into the env folder one by one
+// Inspect the conda-meta/history file and try to find the installation location of the conda by parsing the `cmd:` line.
+// If we find a conda installation, then process that folder as we did inside step 2.1
+```

crates/pet-homebrew/README.md

@@ -0,0 +1,37 @@
+# Hombrew
+
+## Notes
+
+- Install folders & symlinks documented here https://formulae.brew.sh/formula/[email protected]
+- Homebrew install locations documented here https://docs.brew.sh/Installation
+
+## Known Issues
+
+- `sys.prefix` is not easy to identify. Hence left empty for now.
+  If we can find a way to identify `sys.prefix` consistenly for all platforms, then we can populate this field.
+
+## Pseduo code for algorithm
+
+```rust
+homebrew_dir = // find this folder (either "HOMEBREW_PREFIX" or default to directory defined here https://docs.brew.sh/Installation)
+for file under "<homebrew_dir>/bin":
+    if we have a python executable and its a symlink, then proceed
+    if not, then skip this file
+
+    resolve the symlink and verify the file is in one of the known homebrew directories.
+    if not, then skip this file
+
+    Extract the version from the file path.
+
+    Compute the env_path by extracting the version information.
+    The env_path is known directories on MacOS (Intel & Silicon) and Linux.
+
+    Identify all known Symlinks for the python executable.
+    There are a number of symlinks for each Python environment executable. Best to identify them all.
+    As we have no idea which ones will be used by users.
+
+    Note: Identifying `sys_prefix` is not easy, hence left empty for now.
+```
+
+</details>
+</details>

crates/pet-mac-commandlinetools/README.md

@@ -0,0 +1,19 @@
+# Python from Mac Command Line Tools
+
+## Notes
+
+- Look for Python in
+  - /Library/Developer/CommandLineTools/usr/bin
+  - /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions
+- Sometimes, `/usr/bin/python3` can be a copy of Python for the above locations.
+  - Unfortunately these are not symlinks, but we need to `spawn` the process to get the actual `sys.executable` information.
+- `/Library/Developer/CommandLineTools/usr/bin/python?` are generally symlinks to the Python executable in `/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/<version>/bin/python?`.
+- Version
+  - If we spawn such as `/usr/bin/python3`, then we have the version (for at least one of the Python versions).
+  - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+  - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+
+## Known Issues
+
+- `/usr/bin/python3` can be a copy of Python for Python from one of the above locations.
+  Unfortunately these are not symlinks, but we need to `spawn` the process to get the actual `sys.executable` information.

crates/pet-mac-python-org/README.md

@@ -0,0 +1,12 @@
+# Python from Python.org on Mac
+
+## Notes
+
+- Look for Python in
+  - /Library/Frameworks/Python.framework/Versions/
+- Sometimes, `/usr/local/bin/python?` can be a symlink to Python in one of the above locations.
+  - Why `sometimes`, thats because any other installation can end up overwrite the symlink in `/usr/local/bin/python?` with something else.
+- `/Library/Frameworks/Python.framework/Versions/Current/bin/python?` can be a symlink to Python in one of the above locations.
+- Version
+  - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+  - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`

crates/pet-mac-xcode/README.md

@@ -0,0 +1,22 @@
+# Python from Mac XCode
+
+## Notes
+
+- Look for Python in such as:
+  - /Applications/Xcode.app/Contents/Developer/usr/bin/python3
+  - /Applications/Xcode_15.0.1.app/Contents/Developer/usr/bin/python3 (such paths are on CI, see here https://github.com/microsoft/python-environment-tools/issues/38)
+- Sometimes, `/usr/bin/python3` can be a copy of Python for the above locations.
+  - Unfortunately these are not symlinks, but we need to `spawn` the process to get the actual `sys.executable` information.
+- Version
+  - If we spawn such as `/usr/bin/python3`, then we have the version (for at least one of the Python versions).
+  - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+  - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+
+## Known Issues
+
+- `/usr/bin/python3` can be a copy of Python for Python from one of the above locations.
+  Unfortunately these are not symlinks, but we need to `spawn` the process to get the actual `sys.executable` information.
+- `find` will never return python installed in XCode.
+  - The assumption is that if users are using Python installed in XCode, then it will be in `/usr/bin/python3`.
+  - I.e. its very unlikely users will be using Python by looking for it in `/Applications/Xcode.app/Contents/Developer/usr/bin/python3`.
+  - If this is not true, then we can always search for Python in such directories and list them.

crates/pet-pipenv/README.md

@@ -0,0 +1,26 @@
+# Pipenv
+
+## Notes
+
+- Where are pipenv envionments located?
+  - Pipenv environments are generally located in the `WORKON_HOME`, `~/.venvs` directory.
+  - Locations are computed from the `WORKON_HOME` environment variable.
+  - However there are other commonly known locations, found in https://github.com/pypa/pipenv/blob/main/pipenv/utils/shell.py#L184
+  - And then there are other locations found in documentation on the pipenv https://pipenv.pypa.io/en/latest/virtualenv.html
+  - Note: Its possible that `WORKON_HOME` variable is not accessible for whatever reason (e.g. its setup in a shell script that is not sourced). Hence its safer to look for all known locations.
+- `.project`
+  - A Python environment is `pipenv` enviornment if:
+    - A Pytohn envioronment contains a `.project` (this contains the path to the project that the environment is associated with.)
+    - The project directory contains a `Pipfile`
+- Version
+  - Follow the symlink of the Python file and identify the Pthon install location
+    - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+    - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+  - On Windows => Extract the version from `pyvenv.cfg` if the python exe and the `pyvenv.cfg` file were created at the same time (max 60s difference).
+    - Its possible for the Python environment used to create virtual envs to change.
+    - E.g. assume we create a virtual env using Python 3.9, now `pyvenv.cfg` would contain the version 3.9.0.
+    - Now assume we upgrade Python 3.9 to Python 3.10, now the `pyvenv.cfg` file still contains 3.9.0.
+    - However the python executable in the virtual env would point to the same original path and now we have pyhon 3.10 exe there.
+- Find is not implemented for this locator
+  - This is because all environments are located in known locations, hence there is no need to search for them.
+  - For each environment the callers will invoke the `try_from` method to see if the environment is a `pipenv` environment.

crates/pet-poetry/README.md

@@ -0,0 +1,23 @@
+# Poetry
+
+## Notes
+
+- Where are poetry envionments located?
+  - The paths were determined by inspecting the source for Poetry code.
+- `config.toml` is the global Poetry config file
+- `poetry.toml` is the local Poetry config/project file
+- Using the `config.toml` and known global locations, enumerate all known Poetry Environments
+  - Given a project directory (workspace folder), compute the hash for that directory
+  - Look for an environment with the same hash.
+- Given a project directory (workspace folder), determine the local poetry config (`poetry.toml`)
+  - Check the setting `virtualenvs.in-project` and `POETRY_VIRTUALENVS_IN_PROJECT`
+  - Based on the above value any existing `.venv` directory in the project directory will be treated as a Poetry environment.
+- Version
+  - Follow the symlink of the Python file and identify the Pthon install location
+    - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+    - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+  - On Windows => Extract the version from `pyvenv.cfg` if the python exe and the `pyvenv.cfg` file were created at the same time (max 60s difference).
+    - Its possible for the Python environment used to create virtual envs to change.
+    - E.g. assume we create a virtual env using Python 3.9, now `pyvenv.cfg` would contain the version 3.9.0.
+    - Now assume we upgrade Python 3.9 to Python 3.10, now the `pyvenv.cfg` file still contains 3.9.0.
+    - However the python executable in the virtual env would point to the same original path and now we have pyhon 3.10 exe there.

crates/pet-pyenv/README.md

@@ -0,0 +1,11 @@
+# Pyenv
+
+## Notes
+
+- Where is pyenv located?
+  - For windows its generally `~/.pyenv/pyenv-win` else `~/.pyenv`
+- `versions` sub directory for `pyenv` contains all Python versions installed using pyenv
+- On windows, if the directory ends with `-win32`, then its a 32bit Windows Python installation.
+- Version
+  - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+  - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`

crates/pet-venv/README.md

@@ -0,0 +1,16 @@
+# Venv
+
+## Notes
+
+- A Python environment containing a `pyvenv.cfg` file is a virtual environment.
+- Note: `poetry`, `pipenv`, `virtualenvwrapper`, etc all create virtual environments that have a `pyvenv.cfg` file.
+  - Hence when determining whether an Environment is a `venv` environment, we need to first if it is some other type like `poetry` or the like.
+- Version
+  - Follow the symlink of the Python file and identify the Pthon install location
+    - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+    - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+  - On Windows => Extract the version from `pyvenv.cfg` if the python exe and the `pyvenv.cfg` file were created at the same time (max 60s difference).
+    - Its possible for the Python environment used to create virtual envs to change.
+    - E.g. assume we create a virtual env using Python 3.9, now `pyvenv.cfg` would contain the version 3.9.0.
+    - Now assume we upgrade Python 3.9 to Python 3.10, now the `pyvenv.cfg` file still contains 3.9.0.
+    - However the python executable in the virtual env would point to the same original path and now we have pyhon 3.10 exe there.

crates/pet-virtualenv/README.md

@@ -0,0 +1,17 @@
+# Virtualenv
+
+## Notes
+
+- A Python environment containing a activation scripts such as `/bin/activate` or `Scripts/activate.bat` or the like is a virtual environment.
+- Note: `poetry`, `pipenv`, `virtualenvwrapper`, etc all create virtual environments that have a `pyvenv.cfg` file.
+  - Hence when determining whether an Environment is a `virtualenv` environment, we need to first if it is some other type like `poetry` or the like.
+- Similarly an environment containing `pyvenv.cfg` is generally not considered a `virtualenv`, as it could be a `venv`, `poetry`, `pipenv`, etc.
+- Version
+  - Follow the symlink of the Python file and identify the Pthon install location
+    - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+    - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+  - On Windows => Extract the version from `pyvenv.cfg` if the python exe and the `pyvenv.cfg` file were created at the same time (max 60s difference).
+    - Its possible for the Python environment used to create virtual envs to change.
+    - E.g. assume we create a virtual env using Python 3.9, now `pyvenv.cfg` would contain the version 3.9.0.
+    - Now assume we upgrade Python 3.9 to Python 3.10, now the `pyvenv.cfg` file still contains 3.9.0.
+    - However the python executable in the virtual env would point to the same original path and now we have pyhon 3.10 exe there.

crates/pet-virtualenvwrapper/README.md

@@ -0,0 +1,23 @@
+# Virtualenvwrapper
+
+## Notes
+
+- They are regular Python environments created a specific location
+- The location is defined in the `WORKON_HOME` environment variable.
+- Else defaults to `~/.virtualenvs`
+- They too have a have a `.project` file in the root of the environment
+  This file contains the path to the project directory thats associated with this environment.
+- They have a `.pyvenv.cfg` file in the root of the environment
+  This file contains the version of Python used in this environment.
+- A Python environment containing a `pyvenv.cfg` file is a virtual environment.
+- Note: `poetry`, `pipenv`, `virtualenvwrapper`, etc all create virtual environments that have a `pyvenv.cfg` file.
+  - Hence when determining whether an Environment is a `virtualenvwrapper` environment, we need to first if it is some other type like `poetry` or the like.
+- Version
+  - Follow the symlink of the Python file and identify the Pthon install location
+    - Extract the version of Python from the `patchlevel.h` file from the entry `#define PY_VERSION`
+    - These files are located in `<sys prefix>/include/patchlevel.h` or `<sys prefix>/Headers/patchlevel.h`
+  - On Windows => Extract the version from `pyvenv.cfg` if the python exe and the `pyvenv.cfg` file were created at the same time (max 60s difference).
+    - Its possible for the Python environment used to create virtual envs to change.
+    - E.g. assume we create a virtual env using Python 3.9, now `pyvenv.cfg` would contain the version 3.9.0.
+    - Now assume we upgrade Python 3.9 to Python 3.10, now the `pyvenv.cfg` file still contains 3.9.0.
+    - However the python executable in the virtual env would point to the same original path and now we have pyhon 3.10 exe there.