{"slug":"pixi-environment-builder","title":"pixi-environment-builder","summary":"Use when creating, migrating, or debugging pixi environments, especially for scientific Python, bioinformatics, single-cell analysis, CUDA/PyTorch, Jupyter/VS Code kernels, conda-to-pixi migration, or conda + PyPI mixed dependency issues.","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-08-23T08:58:31.66471Z","repo":{"url":"https://github.com/xuzhougeng/wisp-science","stars":1170,"forks":122,"license":"AGPL-3.0","updatedAt":"2026-09-25T06:13:53Z"},"bodyHtml":"<hr>\n<h2>name: pixi-environment-builder\ndescription: Use when creating, migrating, or debugging pixi environments, especially for scientific Python, bioinformatics, single-cell analysis, CUDA/PyTorch, Jupyter/VS Code kernels, conda-to-pixi migration, or conda + PyPI mixed dependency issues.</h2>\n<h1>Pixi Environment Builder</h1>\n<h2>Overview</h2>\n<p>Use this skill to design, migrate, and debug pixi-managed environments. The core principle is to clarify environment intent before editing <code>pixi.toml</code>: version constraints, project scope, package source priority, mirror/network policy, special packages, cache location, and validation tasks.</p>\n<p>Pixi can solve dependencies automatically, but mixed conda + PyPI environments need deliberate package ownership. Most hard failures come from unclear ownership, unconstrained top-level packages, inaccessible mirrors, or non-registry packages.</p>\n<h2>Preflight Questions</h2>\n<p>Before creating or changing a pixi environment, ask these questions unless the answer is already known from repo files, user context, or error logs:</p>\n<ol>\n<li><p><strong>Required versions</strong></p>\n<ul>\n<li>Are any package versions fixed by previous results, notebooks, papers, models, CUDA drivers, or collaborators?</li>\n<li>Examples: <code>python</code>, <code>cuda</code>, <code>pytorch</code>, domain packages, model libraries, analysis frameworks.</li>\n</ul>\n</li>\n<li><p><strong>Environment scope</strong></p>\n<ul>\n<li>Is this project-level, user/global-level, or temporary?</li>\n<li>Project-level: create or edit repo <code>pixi.toml</code>.</li>\n<li>User/global-level: prefer <code>pixi global</code> for reusable CLI tools, not complex project workflows.</li>\n</ul>\n</li>\n<li><p><strong>Package source priority</strong></p>\n<ul>\n<li>Should conda or PyPI own the main dependency graph?</li>\n<li>Which packages should be installed from conda, PyPI, Git, local path, or system modules?</li>\n<li>Avoid specifying the same package unconstrained in both conda and PyPI.</li>\n</ul>\n</li>\n<li><p><strong>Multiple environments or kernels</strong></p>\n<ul>\n<li>Does the user need separate named environments, solve groups, or Jupyter/VS Code kernels?</li>\n<li>Clarify whether they need identical packages in separate prefixes or different feature sets.</li>\n</ul>\n</li>\n<li><p><strong>Mirror and network policy</strong></p>\n<ul>\n<li>Which conda channels and mirrors are reachable?</li>\n<li>Which PyPI index is reachable?</li>\n<li>Can the machine access GitHub, <code>pypi.org</code>, <code>files.pythonhosted.org</code>, <code>prefix.dev</code>, or internal mirrors?</li>\n</ul>\n</li>\n<li><p><strong>Non-registry packages</strong></p>\n<ul>\n<li>Are any packages installed from local source, private Git repos, wheels, editable paths, or unpublished projects?</li>\n</ul>\n</li>\n<li><p><strong>Cache and storage</strong></p>\n<ul>\n<li>Use pixi defaults unless there is a permissions, quota, or sharing requirement.</li>\n<li>If custom cache is needed, ask where writable shared cache should live.</li>\n</ul>\n</li>\n<li><p><strong>Validation</strong></p>\n<ul>\n<li>What imports, version checks, CLI commands, GPU checks, or kernel registration prove the environment works?</li>\n</ul>\n</li>\n</ol>\n<h2>Design Rules</h2>\n<h3>Prefer project-level manifests for project workflows</h3>\n<p>For analysis projects, put environment definition in the repo:</p>\n<pre><code>[workspace]\nname = \"project-name\"\nchannels = [\"conda-forge\"]\nplatforms = [\"linux-64\"]\n</code></pre>\n<p>Use user/global environments mostly for standalone tools.</p>\n<h3>Assign package ownership</h3>\n<p>Choose one owner for each important package family.</p>\n<p>Prefer conda for:</p>\n<ul>\n<li>Python interpreter</li>\n<li>compiled libraries and hard-to-build scientific packages</li>\n<li>CUDA/PyTorch stacks when conda binaries are desired</li>\n<li>R, rpy2, system libraries, CLI bioinformatics tools</li>\n<li>packages requiring consistent native ABI</li>\n</ul>\n<p>Prefer PyPI for:</p>\n<ul>\n<li>packages whose canonical release is PyPI</li>\n<li>fast-moving Python-only libraries</li>\n<li>packages unavailable or stale on conda</li>\n<li>top-level frameworks that expect pip-style dependency resolution</li>\n<li>headless/server variants such as <code>opencv-python-headless</code></li>\n</ul>\n<p>Avoid this pattern:</p>\n<pre><code>[dependencies]\nscanpy = \"*\"\nanndata = \"*\"\nscipy = \"*\"\n\n[pypi-dependencies]\nsome-framework-that-also-depends-on-scanpy = \"*\"\n</code></pre>\n<p>This can make conda pin versions before PyPI solves, causing conflicts.</p>\n<h3>Start minimal, then add constraints only when evidence requires them</h3>\n<p>Do not mechanically copy an entire old conda environment. Start from:</p>\n<ul>\n<li>interpreter/runtime</li>\n<li>top-level packages the user directly uses</li>\n<li>hardware/runtime packages</li>\n<li>Jupyter/kernel tooling if needed</li>\n<li>non-registry packages</li>\n</ul>\n<p>Add transitive pins only when solver output or runtime validation proves they are needed.</p>\n<h2>Migrating From Conda</h2>\n<p>When migrating an existing conda/mamba environment:</p>\n<ol>\n<li>Inspect by path if env-name lookup is unreliable:</li>\n</ol>\n<pre><code>conda list -p /path/to/env\nconda env export -p /path/to/env --no-builds\n</code></pre>\n<ol start=\"2\">\n<li>Identify direct imports and notebook evidence:</li>\n</ol>\n<pre><code>rg -n \"^(import|from) \" scripts src notebooks tests --glob '*.py' --glob '*.ipynb'\nrg -n \"Version:|__version__|import \" notebooks scripts --glob '*.ipynb'\n</code></pre>\n<ol start=\"3\">\n<li><p>Classify packages:</p>\n<ul>\n<li>direct user dependencies</li>\n<li>transitive dependencies</li>\n<li>runtime/system dependencies</li>\n<li>local/Git/private packages</li>\n<li>packages only needed for old experiments</li>\n</ul>\n</li>\n<li><p>Preserve known compatibility anchors:</p>\n<ul>\n<li>versions printed in notebook outputs</li>\n<li>versions required by published workflow</li>\n<li>CUDA/PyTorch compatibility</li>\n<li>package versions known to affect results</li>\n</ul>\n</li>\n<li><p>Leave unrelated transitive packages out of <code>pixi.toml</code>.</p>\n</li>\n</ol>\n<h2>Multiple Environments And Kernels</h2>\n<p>Use multiple named environments when the user needs isolation or parallel notebooks.</p>\n<p>Use one solve group when environments should have identical package versions:</p>\n<pre><code>[environments]\nworker-1 = { solve-group = \"analysis\" }\nworker-2 = { solve-group = \"analysis\" }\nworker-3 = { solve-group = \"analysis\" }\n</code></pre>\n<p>Use separate features when environments differ:</p>\n<pre><code>[feature.gpu.dependencies]\npytorch-cuda = \"*\"\n\n[feature.r.dependencies]\nr-base = \"*\"\n\n[environments]\ncpu = []\ngpu = [\"gpu\"]\nr-analysis = [\"r\"]\n</code></pre>\n<p>For VS Code/Jupyter kernels, add explicit kernel tasks:</p>\n<pre><code>[tasks]\nkernel-1 = \"python -m ipykernel install --user --name worker-1 --display-name 'Python (worker-1)'\"\nkernel-2 = \"python -m ipykernel install --user --name worker-2 --display-name 'Python (worker-2)'\"\nkernels = \"pixi run -e worker-1 kernel-1 &amp;&amp; pixi run -e worker-2 kernel-2\"\n</code></pre>\n<p>Tell VS Code users: after registration, select the kernel in VS Code; they do not need to launch notebooks through <code>pixi run</code>.</p>\n<h2>Mirrors And Network</h2>\n<p>Use mirrors deliberately. Do not assume a mirror works for all package types.</p>\n<p>Recommended checks:</p>\n<pre><code>pixi config list\nsed -n '1,120p' ~/.config/uv/uv.toml 2&gt;/dev/null\nsed -n '1,120p' ~/.config/pip/pip.conf 2&gt;/dev/null\nenv | rg \"PIP|UV|PIXI|RATTLER|HTTP|HTTPS|PROXY\"\n</code></pre>\n<p>For PyPI, prefer setting only the index URL in <code>pixi.toml</code>:</p>\n<pre><code>[pypi-options]\nindex-url = \"https://example-mirror/simple\"\n</code></pre>\n<p>Avoid unnecessary <code>files.pythonhosted.org</code> mirror rewrites unless verified. Some mirrors serve simple index pages but fail wheel metadata URLs.</p>\n<p>If official PyPI times out, switch to a reachable mirror. If a mirror gives 404 for metadata, try another mirror or the official file server directly.</p>\n<p>For conda, use <code>.pixi/config.toml</code> mirrors when needed:</p>\n<pre><code>[mirrors]\n\"https://conda.anaconda.org/conda-forge\" = [\n  \"https://your-conda-mirror/anaconda/cloud/conda-forge\",\n  \"https://conda.anaconda.org/conda-forge\"\n]\n</code></pre>\n<h2>Conda + PyPI Mapping</h2>\n<p>Pixi needs conda-to-PyPI name mapping when conda and PyPI dependencies are mixed. If fetching mapping from <code>prefix.dev</code> fails, use a local mapping file.</p>\n<p>In <code>pixi.toml</code>:</p>\n<pre><code>[workspace]\nconda-pypi-map = { \"conda-forge\" = \"config/conda-pypi-map.json\" }\n</code></pre>\n<p>Example <code>config/conda-pypi-map.json</code>:</p>\n<pre><code>{\n  \"scikit-learn\": \"scikit-learn\",\n  \"matplotlib-base\": \"matplotlib\",\n  \"pytorch\": \"torch\",\n  \"torchvision\": \"torchvision\",\n  \"torchaudio\": \"torchaudio\"\n}\n</code></pre>\n<p>Validate it:</p>\n<pre><code>python -m json.tool config/conda-pypi-map.json\n</code></pre>\n<p>Keep this mapping small and project-specific. Add entries only for packages relevant to mixed solving.</p>\n<h2>Non-Registry Packages</h2>\n<p>If a package is not found on PyPI or conda, inspect how it was installed before guessing.</p>\n<p>Check old environment metadata:</p>\n<pre><code>find /path/to/env/lib/python*/site-packages -maxdepth 3 \\\n  \\( -iname '*dist-info' -o -path '*dist-info/direct_url.json' \\)\n</code></pre>\n<p>Use local path dependency when reproducible on this machine:</p>\n<pre><code>[pypi-dependencies]\nmy-package = { path = \"/absolute/path/to/source\" }\n</code></pre>\n<p>Use Git dependency when portability matters:</p>\n<pre><code>[pypi-dependencies]\nmy-package = { git = \"https://github.com/org/repo.git\", rev = \"commit-sha\" }\n</code></pre>\n<p>Prefer a fixed commit/tag for reproducibility.</p>\n<h2>Common Failure Patterns</h2>\n<h3>Package not found in registry</h3>\n<p>Root cause: package is unpublished, private, named differently, or only installed from source.</p>\n<p>Actions:</p>\n<ul>\n<li>Inspect old <code>direct_url.json</code>.</li>\n<li>Search project docs for install command.</li>\n<li>Use path or Git dependency.</li>\n<li>Do not keep retrying PyPI.</li>\n</ul>\n<h3>Version conflict after conda solve</h3>\n<p>Root cause: conda pinned a transitive package version that conflicts with PyPI requirements.</p>\n<p>Actions:</p>\n<ul>\n<li>Read the solver message for pinned packages.</li>\n<li>Decide whether top-level package should own those dependencies.</li>\n<li>Remove conda-side transitive packages, or bound them to a compatible range.</li>\n<li>Pin only compatibility anchors, not every transitive package.</li>\n</ul>\n<h3>Network timeout fetching PyPI package</h3>\n<p>Root cause: inaccessible PyPI index, file host, proxy, or mirror.</p>\n<p>Actions:</p>\n<ul>\n<li>Check current <code>[pypi-options] index-url</code>.</li>\n<li>Check user uv/pip config for reachable mirrors.</li>\n<li>Change only PyPI index first.</li>\n<li>Clean PyPI cache and retry.</li>\n</ul>\n<pre><code>pixi clean cache --pypi -y\npixi install --all\n</code></pre>\n<h3>Mirror metadata 404</h3>\n<p>Root cause: simple index mirror works but wheel metadata/file mirror is incomplete.</p>\n<p>Actions:</p>\n<ul>\n<li>Remove <code>files.pythonhosted.org</code> mirror rewrites.</li>\n<li>Use a different PyPI index.</li>\n<li>Avoid mixing multiple PyPI mirror layers unless verified.</li>\n</ul>\n<h3>OpenCV solve conflict</h3>\n<p>Root cause: conda <code>opencv</code> pulls GUI/Qt/Python ABI-specific builds.</p>\n<p>Server/headless fix:</p>\n<pre><code>[pypi-dependencies]\nopencv-python-headless = \"&gt;=4.10,&lt;5\"\n</code></pre>\n<p>Use conda <code>opencv</code> only when GUI functionality is required.</p>\n<h3>CUDA/PyTorch mismatch</h3>\n<p>Root cause: CUDA runtime, driver, PyTorch build, and channel priorities disagree.</p>\n<p>Actions:</p>\n<ul>\n<li>Ask for <code>nvidia-smi</code>.</li>\n<li>Pin CUDA runtime intentionally.</li>\n<li>Use a single coherent PyTorch source.</li>\n<li>Validate with <code>torch.cuda.is_available()</code>.</li>\n</ul>\n<h2>Cache Guidance</h2>\n<p>Pixi has default cache. Do not create custom cache directories unless the user requests shared cache, has permission errors, or needs a specific storage path.</p>\n<p>Useful commands:</p>\n<pre><code>pixi clean cache --pypi -y\npixi clean cache --repodata -y\npixi clean cache --mapping -y\n</code></pre>\n<p>If custom cache is needed:</p>\n<pre><code>PIXI_CACHE_DIR=/path/to/pixi-cache \\\nRATTLER_CACHE_DIR=/path/to/rattler-cache \\\npixi install --all\n</code></pre>\n<h2>Verification Tasks</h2>\n<p>Add a <code>check</code> task for complex environments. It should verify the actual success criteria, not just installation.</p>\n<p>Examples:</p>\n<pre><code>[tasks]\ncheck = \"python -c \\\"import sys; print(sys.version)\\\"\"\n</code></pre>\n<p>For GPU Python environments:</p>\n<pre><code>check = \"python -c \\\"import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NO GPU')\\\"\"\n</code></pre>\n<p>Run:</p>\n<pre><code>pixi run -e &lt;environment&gt; check\n</code></pre>\n<p>For Jupyter/VS Code workflows, verify kernel registration separately:</p>\n<pre><code>pixi run kernels\njupyter kernelspec list\n</code></pre>\n<h2>OpenBLAS Thread Tuning For R Environments</h2>\n<p>conda-forge <code>r-base</code> ships with OpenBLAS, but when <code>OPENBLAS_NUM_THREADS</code> is unset on a\nhigh-core server, the default thread scheduling is extremely poor — SVD on 96 cores\nwithout explicit thread count is <em>slower</em> than single-threaded. This directly impacts\nSeurat <code>RunPCA()</code> (backed by <code>irlba()</code> randomized SVD).</p>\n<h3>When To Apply</h3>\n<ul>\n<li>User reports RunPCA / SVD / matrix operations are slow in a pixi R environment</li>\n<li><code>OPENBLAS_NUM_THREADS</code> and <code>OMP_NUM_THREADS</code> are both unset</li>\n<li>Multi-core Linux server (&gt;16 cores)</li>\n</ul>\n<h3>Diagnostic Flow</h3>\n<h4>Step 1 - Confirm BLAS Implementation</h4>\n<pre><code>PREFIX=$(pixi info --manifest-path pixi-workspaces/&lt;env&gt;/pixi.toml 2&gt;/dev/null | grep \"Prefix location\" | awk '{print $NF}')\nreadlink -f \"$PREFIX/lib/libblas.so.3\"\n</code></pre>\n<ul>\n<li>Points to <code>libopenblasp-*.so</code> → OpenBLAS, this section applies</li>\n<li>Points to <code>libflexiblas.so</code> → different approach needed (FlexiBLAS backend switching)</li>\n</ul>\n<h4>Step 2 - Baseline Benchmark</h4>\n<pre><code>pixi run --manifest-path pixi-workspaces/&lt;env&gt;/pixi.toml \\\n  Rscript -e '\n  cat(\"OPENBLAS_NUM_THREADS =\", Sys.getenv(\"OPENBLAS_NUM_THREADS\"), \"\\n\")\n  set.seed(42); n &lt;- 3000; X &lt;- matrix(rnorm(n*n), n, n)\n  t &lt;- system.time({ svd(X, nu=10, nv=0) })\n  cat(\"SVD(3000) time:\", round(t[\"elapsed\"], 3), \"sec\\n\")\n  cat(\"Detected cores:\", parallel::detectCores(), \"\\n\")\n  '\n</code></pre>\n<ul>\n<li>SVD(3000) &gt; 15 sec on 96 cores → confirmed thread scheduling problem</li>\n<li>SVD(3000) &lt; 5 sec → already optimized, no action needed</li>\n</ul>\n<h4>Step 3 - Thread Count Sweep</h4>\n<pre><code>for threads in 1 8 16 32 64; do\n    echo \"=== OPENBLAS_NUM_THREADS=$threads ===\"\n    OPENBLAS_NUM_THREADS=$threads OMP_NUM_THREADS=$threads \\\n    pixi run --manifest-path pixi-workspaces/&lt;env&gt;/pixi.toml \\\n    Rscript -e '\n    set.seed(42); n &lt;- 3000; X &lt;- matrix(rnorm(n*n), n, n)\n    t &lt;- system.time({ svd(X, nu=10, nv=0) })\n    cat(\"SVD(3000) time:\", round(t[\"elapsed\"],3), \"sec\\n\")\n    ' 2&gt;/dev/null\ndone\n</code></pre>\n<h4>Step 4 - Pick Optimal Thread Count</h4>\n<p>Benchmark results on AMD EPYC 7K62 (96 cores, Zen2):</p>\n<table>\n<thead>\n<tr>\n<th>Threads</th>\n<th>SVD(3000) Time</th>\n<th>Speedup</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Default (unset)</td>\n<td>~20 sec</td>\n<td>1x (baseline)</td>\n</tr>\n<tr>\n<td>1</td>\n<td>~12 sec</td>\n<td>1.6x</td>\n</tr>\n<tr>\n<td>8</td>\n<td>~3.8 sec</td>\n<td>5.2x</td>\n</tr>\n<tr>\n<td>16</td>\n<td>~3.2 sec</td>\n<td>6.2x</td>\n</tr>\n<tr>\n<td><strong>32</strong></td>\n<td><strong>~2.9 sec</strong></td>\n<td><strong>6.8x (optimal)</strong></td>\n</tr>\n<tr>\n<td>64</td>\n<td>~3.3 sec</td>\n<td>6.0x (overhead degrades)</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Rule of thumb</strong>: optimal thread count ≈ 1/3 of total cores (96→32, 64→16-24, 32→8-16).\nBeyond the sweet spot, thread synchronization overhead degrades performance.</p>\n<h3>Fix - pixi.toml activation.env (Recommended)</h3>\n<p>Add <code>[activation.env]</code> to the workspace <code>pixi.toml</code>:</p>\n<pre><code>[activation.env]\nOPENBLAS_NUM_THREADS = \"32\"\nOMP_NUM_THREADS = \"32\"\n</code></pre>\n<p>Takes effect on every <code>pixi run</code> or environment activation. Jupyter kernels (IRkernel)\nregistered via <code>pixi run kernel</code> also inherit these variables.</p>\n<h3>Gotchas</h3>\n<ul>\n<li><strong>Do NOT use all cores</strong>: 96 cores fully loaded is ~15% slower than 32 threads</li>\n<li><strong>Set OMP_NUM_THREADS too</strong>: some R packages (data.table, RcppParallel) use OpenMP</li>\n<li><strong>conda-forge r-base already links OpenBLAS</strong>: unlike system R (/opt/R), no need to\nmanually replace libRblas.so symlinks</li>\n<li><strong>Optimal thread count varies by CPU</strong>: AMD EPYC (Zen2) vs Intel Xeon may differ</li>\n</ul>\n<h2>Debugging Discipline</h2>\n<ul>\n<li>Treat each error type separately: network timeout, package not found, version conflict, mapping failure, runtime import failure.</li>\n<li>Change one thing per solver error where possible.</li>\n<li>Do not rewrite the manifest blindly after every failure.</li>\n<li>Do not run install commands if the user asked only for diagnosis or commands.</li>\n<li>Explain whether warnings are harmless or actionable.</li>\n</ul>\n","files":[{"path":"SKILL.md","sizeBytes":14990,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-08-23T09:02:33.355119Z","sha256":"21B7B282343CA0BB03321B310115EB882EFE5765FAFF6470545813825D627028","sizeBytes":6220},"review":null,"source":{"repositoryUrl":"https://github.com/xuzhougeng/wisp-science","path":"skills/pixi-environment-builder","license":"AGPL-3.0","commit":"79e64163262196611dd390d06713c465b906b299","subtreeSha":"5975B99460392B4A85E7C9612DE0FBE290F487319FDC57FC2DB2CAB52EBBC695","lastSyncedAt":"2026-09-25T07:37:55.552497Z"},"reviewedAt":"2026-08-23T09:10:05.253515Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/xuzhougeng/wisp-science/tree/main/skills/pixi-environment-builder"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install xuzhougeng-wisp-science@llmmart"},{"target":"git","command":"git clone https://github.com/xuzhougeng/wisp-science.git"}]}