Repository navigation
docs: clarify how pixi resolves conda vs PyPI dependencies #819
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -8,7 +8,6 @@ | |||||
| - **Installing CLI tools globally?** → pipx | ||||||
| - **Need conda packages or cross-language dependencies?** → **pixi** (recommended) or conda/mamba | ||||||
| - **Creating a Python package?** → Use Hatch -- with UV as a dependency manager | ||||||
|
|
||||||
| You can mix tools! For example, use **pipx** to install tools you use often (at the command line) like Hatch, ruff or pre-commit, then use **uv** within your projects for package and environment management. | ||||||
| ::: | ||||||
|
|
||||||
|
|
@@ -18,18 +17,15 @@ Package and environment managers are important tools in your Python packaging wo | |||||
| Your packaging experience when selecting a tool will be easier if you understand the difference between the two. | ||||||
|
|
||||||
| 1. A **package manager** is used to install, update, and remove Python packages (libraries and tools) and their dependencies in your environment. When you use a package manager, you are often downloading packages from a repository like PyPI (Python Package Index) or a local repository like GitHub / GitLab. | ||||||
|
|
||||||
| :::{note} | ||||||
| When you run `pip install numpy`, pip acts as a package manager and installs numpy from PyPI. Pip's default repository when you install a package is PyPI, but it can be used to install packages from other repositories such as GitHub. | ||||||
| ::: | ||||||
|
|
||||||
| 2. An **environment manager** creates isolated spaces (environments) for your Python projects. Each environment has its own Python installation and its own installed packages. Using isolated environments for different projects reduces the change of environment conflicts when using the same environment across different projects with different dependencies. | ||||||
|
|
||||||
| There are many tools listed below, but if you're short on time, you may want to consider | ||||||
|
|
||||||
| 1. Hatch combined with UV if you are managing a Python package. [Check out our tutorials for more on this workflow.](create-pure-python-package) | ||||||
| 2. Pixi or mamba as faster alternatives to conda if you are working in the non-Pure Python packaging space. | ||||||
|
|
||||||
| ## Where environment managers save your environment | ||||||
|
|
||||||
| Environment managers save environments in different locations by default. For instance, `venv`, an environment manager that ships with Python, saves an environment by default in your current working directory. UV has the same native behavior. In contrast, conda and mamba save environments in a global location, allowing you to access them easily across projects. | ||||||
|
|
@@ -71,7 +67,6 @@ Some modern tools handle both package installation and environment management. F | |||||
| * Pip is Python's standard package installer. It is included with Python by default. | ||||||
| * Pip is great for installing packages from PyPI and GitHub / GitLab into existing environments. | ||||||
| * It is also great for development if you want to install your package locally in editable mode. | ||||||
|
|
||||||
| **Basic usage:** | ||||||
| ```bash | ||||||
| pip install numpy | ||||||
|
|
@@ -81,7 +76,6 @@ pip install -e . # Install your package in editable mode | |||||
| ### pipx | ||||||
|
|
||||||
| * Pipx is can be used to install a tool that you need to use across projects (like `riff`, `pytest`, `sphinx`, `nox`), globally. | ||||||
|
|
||||||
| Why use it: You might use it to avoid reinstalling the same tool over and over on your machine. | ||||||
|
|
||||||
| **Basic usage:** | ||||||
|
|
@@ -243,10 +237,10 @@ pip install -e ".[dev]" | |||||
| # Initialize project (creates environment automatically) | ||||||
| pixi init | ||||||
|
|
||||||
| # Add packages | ||||||
| # Add packages from conda-forge (pixi's default) | ||||||
| pixi add numpy | ||||||
|
|
||||||
| # Install your package in dev mode | ||||||
| # Install your package in dev mode from PyPI | ||||||
| pixi add --pypi --editable "package-name[dev] @ file:///absolute/path/to/package" | ||||||
| ``` | ||||||
| ::: | ||||||
|
|
@@ -275,26 +269,51 @@ mamba activate myenv | |||||
| For projects needing conda packages, we recommend **pixi** over conda/mamba. It's faster, uses lock files for reproducibility, and works cross-platform. | ||||||
| ``` | ||||||
|
|
||||||
| Pixi is a modern, fast package and environment manager built on conda ecosystems. Similar to UV, Pixi uses lock files for reproducible environments. Pixi is best suited for scientific projects that require conda packages, teams that require exact reproducibility, or cross-platform development. | ||||||
| Pixi is a modern, fast package and environment manager built on the conda ecosystem. Similar to uv, Pixi uses lock files for reproducible environments. Pixi is best suited for scientific projects that require conda packages, teams that require exact reproducibility, or projects that need one lock file covering multiple platforms. | ||||||
|
|
||||||
| **How Pixi resolves packages:** | ||||||
|
|
||||||
| Pixi is "conda-first." By default, `pixi add` installs packages from conda channels (conda-forge, unless you configure otherwise). If you need a package that is only available on PyPI, use `pixi add --pypi`. Pixi records each dependency in your manifest (`pixi.toml` or `pyproject.toml`) under either `[dependencies]` (conda) or `[pypi-dependencies]` (PyPI), and it resolves everything together into a single `pixi.lock` file. When resolving PyPI dependencies, Pixi [uses the already-resolved conda packages where it can](https://pixi.sh/latest/python/tutorial/#replacing-pypi-packages-with-conda-packages), so a PyPI package's dependencies won't pull in a second copy of something that conda already provides. | ||||||
|
|
||||||
| **Basic usage:** | ||||||
|
|
||||||
| ```bash | ||||||
| # Initialize new project | ||||||
| pixi init | ||||||
|
|
||||||
| # Add Python packages | ||||||
| pixi add numpy pandas | ||||||
| # Add packages from conda-forge (the default) | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| pixi add numpy gdal | ||||||
|
|
||||||
| # Add conda packages (like GDAL) | ||||||
| pixi add gdal | ||||||
| # Add packages from PyPI (only if a package isn't on conda-forge) | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am not sure, but i wonder if this is a best practice - so only install from pypi if it's not on conda-forge, and this would be because of channel mixing being "dangerous" to do in environments? ignore if i am wrong as i am not an active pixi user! |
||||||
| pixi add --pypi scipy | ||||||
|
|
||||||
| # Run commands in the environment | ||||||
| pixi run python script.py | ||||||
|
|
||||||
| # Activate environment in an interactive shell | ||||||
| pixi shell | ||||||
| ``` | ||||||
|
|
||||||
| The commands above produce a manifest like this: | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ```toml | ||||||
| [workspace] | ||||||
| channels = ["conda-forge"] | ||||||
| name = "example" | ||||||
| platforms = ["linux-64"] | ||||||
|
|
||||||
| [dependencies] | ||||||
| numpy = ">=2.3.4,<3" | ||||||
| gdal = ">=3.11,<4" | ||||||
|
|
||||||
| [pypi-dependencies] | ||||||
| scipy = ">=1.16.3, <2" | ||||||
| ``` | ||||||
|
|
||||||
| Here, Pixi installs `numpy` and `gdal` from conda-forge and only `scipy` from PyPI. You can confirm where each package came from with `pixi list`. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| This is the same outcome as creating a conda environment, installing `numpy` and `gdal` with conda, and then running `pip install scipy` inside it. The difference is that the conda-vs-PyPI split is declared explicitly in the manifest and captured in the lock file, so there is no ambiguity about how a given dependency will be resolved at install time. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This statement is actually confusing too. You can add a pip: section to a conda environment. yml file. and that is explicit. Can you clarify what this actually does that is different from using an environment.yml file that has a pip section? i think maybe leaning into the speed gains might be a better summary saying - this is similar to creating a conda environment with a pip section - so it's declarative but pixi adds x, y, z value to. the process Emphasize the similarities, but also how they differ and how Pixi gives you more power. @ruben-arts I'd love your input here because I am not a Pixi user, but this statement doesn't clearly articulate the value I think Pixi provides to a user! |
||||||
|
|
||||||
| ```{note} | ||||||
| Pixi automatically creates a lock file (`pixi.lock`) ensuring everyone on your team gets identical environments. | ||||||
| ``` | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey there, @InessaPawson, I have a question. Installing in dev mode frm PyPI doesn't fit into my mental model of editable mode because I think editable mode is when you work on the package locally so it's a local link to the package's flat files, and you can edit those files and restart the "kernel" or python session to see the new changes live. So it's not from PyPI