# Spark Environment Setup

> Configura un entorno de entrenamiento/inferencia de ML en NVIDIA DGX Spark (GB10, aarch64, CUDA 13): instalación de PyTorch/Unsloth/TRL/vLLM, errores ABI de libcudart y NGC vs pip.

Fuente: https://skillsagentes.com/skills/wshobson/agents/spark-environment-setup
Markdown: https://skillsagentes.com/skills/wshobson/agents/spark-environment-setup.md

Repositorio: https://github.com/wshobson/agents
Autor: wshobson
Licencia: MIT
Actualizado: hace 2 meses
Coste de contexto: 66 tok instalada, 2k tok al activarse, 4.8k tok con todos los archivos del bundle
Bundle: 3 archivos, 19 KB
Permisos que pide: ninguno declarado

## Instalación

Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina.

```bash
# Claude Code
npx -y skills add wshobson/agents --skill spark-environment-setup --agent claude-code
# Cursor
npx -y skills add wshobson/agents --skill spark-environment-setup --agent cursor
# Codex
npx -y skills add wshobson/agents --skill spark-environment-setup --agent codex
# Gemini CLI
npx -y skills add wshobson/agents --skill spark-environment-setup --agent gemini
# Windsurf
npx -y skills add wshobson/agents --skill spark-environment-setup --agent windsurf
# Cline
npx -y skills add wshobson/agents --skill spark-environment-setup --agent cline
```

## Qué hace

- Guía la instalación de PyTorch/Unsloth/TRL/vLLM en DGX Spark (GB10, aarch64, CUDA 13)
- Recomienda usar contenedores NGC/Unsloth antes que pip directo
- Da la secuencia exacta de pip install con versiones fijadas cuando el contenedor no aplica
- Diagnostica errores de ABI entre CUDA 12/13 (libcudart, símbolos indefinidos)
- Incluye comandos de verificación para confirmar que el entorno ve la GPU correctamente

## Cuándo usarla

- Configurar una Spark box nueva para entrenamiento o inferencia
- Aparece un error de importación con libcudart, un símbolo faltante o un wheel que no carga
- Falla, se cuelga o cae a CPU una instalación de PyTorch, Unsloth, TRL, vLLM o xformers
- Decidir entre usar un contenedor NGC o pip directo, o restaurar el entorno tras reinstalar el SO

## Qué la activa

- "Configura el entorno de PyTorch en mi DGX Spark"
- "Me da un error de libcudart al importar torch en aarch64"
- "¿Uso el contenedor NGC o pip para instalar Unsloth en Spark?"

## Antes de instalar

- Requiere acceso a una máquina DGX Spark (GB10, aarch64, CUDA 13) y, para la vía contenedor, Docker con runtime nvidia.
- Necesita en el PATH: docker, pip, python3

## Archivos

- SKILL.md — 8 KB
- references/container-workflow.md — 4 KB
- references/stack-matrix.md — 7 KB

## SKILL.md

Reproducido tal cual desde wshobson/agents bajo MIT. Esta sección es el documento original y está en inglés.

# Spark Environment Setup

DGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121
GPU, 128GB unified memory, CUDA 13. This is a narrower and
younger platform than a standard x86 CUDA 12 box, so package
selection and ABI matching matter more than usual — the wheel
ecosystem for aarch64 + CUDA 13 is still filling in.

## When to Use This Skill

- Setting up a fresh Spark box for training or inference.
- Hitting an import error mentioning `libcudart`, a missing
  symbol, or a wheel that "installed fine but won't load."
- A framework install (PyTorch, Unsloth, TRL, vLLM, xformers)
  fails, hangs, or silently falls back to CPU.
- Deciding whether to use an NGC container or bare pip.
- Restoring a working setup after an OS reinstall or a
  base-image update, needing to re-verify from scratch.

Each of these accepts the same general fix: match the
container/wheel combination to CUDA 13 and SM121, don't fight
the ABI.

## Container-First Rule

Quick decision, before the detail below:

- Standard training/inference work → NGC PyTorch container.
- Unsloth-centric fine-tuning → Unsloth container (it ships
  the pinned Triton/xformers/transformers combination already
  validated for that path).
- Neither fits (custom system package, local IDE interpreter)
  → bare pip, following the exact sequence further down.

Default to a container. Use `nvcr.io/nvidia/pytorch:25.09-py3`
as the base for general work — the newest tag confirmed working
on this hardware; pull a newer blessed tag if locally available
rather than hard-blocking on `25.11-py3`. NGC's tag is dated, so
running it directly is fine:

```bash
docker run --runtime=nvidia --gpus all -it --rm \
  nvcr.io/nvidia/pytorch:25.09-py3
```

`unsloth/unsloth:dgxspark-latest` is a *moving* tag by
contrast — resolve and pin its digest before running it for
anything reproducible; the bare tag is a discovery step only,
not the default invocation. Full pull-inspect-pin sequence and
flag rationale/volume mounts for `finetuning/` run dirs:
`references/container-workflow.md`. Treat bare pip as the exception.

The reason for the container-first stance is pinning, not
convenience. Triton, xformers, and transformers versions
interact narrowly with GB10's SM121 target and CUDA 13; a
container locks all of them together against a combination
already validated on this hardware. Bare pip leaves that
resolution to you, one broken import at a time.

When bare pip is warranted, follow the NVIDIA playbook's
install sequence verbatim and in order:

```bash
pip install "transformers==5.13.1" "peft==0.19.1" "hf_transfer==0.1.9" "datasets==4.3.0" "trl==1.8.0"
pip install --no-deps "unsloth==2026.7.2" "unsloth_zoo==2026.7.2" "bitsandbytes==0.49.2"
pip install -U "torchao==0.17.0"
```

The second command's `--no-deps` flag is not optional —
letting pip re-resolve Unsloth's dependency tree on aarch64 is
a common way to pull in an incompatible torch or triton build.
The third line is not optional either: the NGC base image's
bundled `torchao` is too old for current `peft`'s LoRA-attach
path (`ImportError: ... torchao ... only versions above 0.16.0
are supported`) — a hard blocker, not a warning. Every `==` pin
above is load-bearing, taken from the dated known-good version
matrix in `references/stack-matrix.md` (its `Last verified` date
governs staleness) — an unpinned install resolves current PyPI
versions well outside what this Unsloth release supports.

Pull a fresh tag when a new blessed release is announced.
Rebuild locally from one of the two bases only when a project
needs an extra system package layered in — not to "upgrade" a
component the image already pins. Details on both paths:
`references/container-workflow.md`.

One more preflight: official DGX Spark playbooks have shipped
broken before. Check recent issues on
`github.com/NVIDIA/dgx-spark-playbooks` (and the other
resources in `references/stack-matrix.md`) before trusting a
recipe verbatim for a long run.

## The ABI Rule

The single most common failure on Spark is a CUDA 12/13 ABI
mismatch: a wheel built against `libcudart.so.12` loaded on a
system that only has `libcudart.so.13`. The install usually
succeeds; the failure surfaces later as a missing-symbol error
or a segfault that doesn't obviously point at CUDA.

Fix: pull wheels from `download.pytorch.org/whl/cu130` (the
cu130-tagged aarch64 builds), or use one of the containers
above, which already carry a matched build. Before chasing a
stack trace that mentions a CUDA symbol, check which CUDA tag
the installed wheel was built against:

```bash
python3 -c "import torch; print(torch.version.cuda)"
```

If that output doesn't start with `13`, the ABI mismatch is the
first thing to fix. NGC container builds (e.g.
`nvcr.io/nvidia/pytorch:25.09-py3`) build torch internally
against CUDA 13 with no `+cu130` wheel tag — `pip show torch`
won't say `cu130` there, and that absence alone is not a failure.

Typical symptoms:

- `ImportError: undefined symbol` referencing a CUDA runtime
  function.
- A segfault on the first `.cuda()` call, no useful traceback.
- A wheel that installs cleanly, then fails at import time —
  pip's resolver doesn't check CUDA ABI, only version constraints.
- Two "identical" environments behaving differently — usually one
  has a cu130 wheel, the other a cu121/cu124 leftover.

The fix is the same regardless of symptom: match the wheel's
CUDA tag to the system, or use a container that already does.

## Component Quick Table

Condensed status for the components most likely to come up.
Full table with wheel URLs, build flags, the sm_121 vs sm_121a
distinction, and the dated known-good version matrix:
`references/stack-matrix.md`.

| Component | Status |
|---|---|
| PyTorch | ✅ official cu130 aarch64 wheels |
| bitsandbytes | ✅ works out of the box |
| Triton | ✅ needs the `TRITON_PTXAS_PATH` parameter set |
| flash-attn | ❌ skip pip build; NGC bundles a working one — see `spark-training-gotchas` G2 |
| xformers | source build only (`TORCH_CUDA_ARCH_LIST=12.1`) |
| vLLM | nightly wheels only |
| TransformerEngine / NVFP4 train | container-only |

Everything else — Unsloth, Axolotl, TRL, PEFT — installs
cleanly through the container-first path above. LLaMA-Factory
and NeMo are fragile on Spark; check upstream issues first.

## Verification Commands

Confirm the environment can actually see the GPU before
running anything expensive:

```python
import torch
print(torch.cuda.is_available(), torch.version.cuda)
```

This call returns two values; the exact output format is one
line, `<bool> <cuda-version>`:

```text
True 13.0
```

If it prints `False` instead, don't jump straight to a wheel
reinstall — ABI mismatch is one cause among several:

| Hypothesis | Quick check |
|---|---|
| Runtime/flags | `nvidia-smi` fails in-container too |
| Device visibility | `echo $CUDA_VISIBLE_DEVICES` |
| Permissions | `ls -l /dev/nvidia*` |
| CUDA init state | wedged process; retry fresh shell/container |
| ABI mismatch (usual culprit) | `torch.version.cuda` not `13.x` |

Check `nvidia-smi` first — if it doesn't show the GPU, it's one
of the first three, not ABI. Reinstall a wheel only once ABI is
confirmed. Per-hypothesis detail: `references/stack-matrix.md`.
Run right after the container starts, before installing
project-specific packages.

One more check: if Triton kernel compilation fails once
training starts, set
`TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas` and retry — see
`references/stack-matrix.md` for the full workaround list.

## Next Steps

A verified environment is only the starting point. See also:
`spark-training-gotchas` for failure preflights before a
training run, and `spark-memory-thermal-ops` for unified-memory
OOMs and thermal throttling during long ones.

## Dónde encaja

- Categoría: [DevOps e infraestructura](https://skillsagentes.com/categorias/devops-infraestructura.md) — Despliegues, contenedores, IaC y flujos de gestión de incidentes.
- Creador: [wshobson](https://skillsagentes.com/creators/wshobson.md) — 183 skills en el directorio
- [Todas las skills](https://skillsagentes.com/skills.md)
- [Ranking de instalaciones](https://skillsagentes.com/ranking.md)

## Otras skills del mismo repositorio

- [Hermes Tweet](https://skillsagentes.com/skills/wshobson/agents/hermes-tweet.md): Instala y opera Hermes Tweet, un plugin de Hermes Agent para investigar X/Twitter, leer timelines, analizar tweets y ejecutar operaciones privadas o de cambio de estado con aprobación previa.
- [Superself](https://skillsagentes.com/skills/wshobson/agents/superself.md): Úsalo cuando un proyecto guarda su estado en Superself: lee `self context` al iniciar sesión, vincula el trabajo a una work unit, reporta con evidencia y registra decisiones confirmadas.
- [Grounded Vault](https://skillsagentes.com/skills/wshobson/agents/grounded-vault.md): Úsalo para mantener un almacén Markdown de conocimiento donde cada afirmación compilada se rastrea hasta una fuente inmutable y el drift se detecta con git diff sin gastar tokens.
- [Postgresql Table Design](https://skillsagentes.com/skills/wshobson/agents/postgresql-table-design.md): Úsalo al diseñar o revisar un esquema específico de PostgreSQL: buenas prácticas, tipos de datos, indexación, restricciones, patrones de rendimiento y funciones avanzadas.
- [Prompt Engineering Patterns](https://skillsagentes.com/skills/wshobson/agents/prompt-engineering-patterns.md): Úsalo cuando pidan optimizar un prompt, mejorar su rendimiento, diseñar una plantilla, aplicar chain-of-thought, few-shot prompting o técnicas avanzadas de prompt engineering para producción.

## Skills relacionadas

- [Bazel Build Optimization](https://skillsagentes.com/skills/wshobson/agents/bazel-build-optimization.md): Optimiza builds de Bazel en monorepos a gran escala. Úsalo al configurar Bazel, implementar ejecución remota u optimizar el rendimiento de builds en codebases empresariales.
- [Workflow Orchestration Patterns](https://skillsagentes.com/skills/wshobson/agents/workflow-orchestration-patterns.md): Diseña workflows duraderos con Temporal para sistemas distribuidos: separación workflow/activity, patrones saga, gestión de estado y restricciones de determinismo.
- [Terraform Module Library](https://skillsagentes.com/skills/wshobson/agents/terraform-module-library.md): Construye módulos Terraform reutilizables para infraestructura AWS, Azure, GCP y OCI siguiendo buenas prácticas de infraestructura como código.
- [Spark Training Gotchas](https://skillsagentes.com/skills/wshobson/agents/spark-training-gotchas.md): Preflight y diagnóstico de los diez fallos conocidos de entrenamiento ML en NVIDIA DGX Spark; úsala ante fallos de arranque, OOM bajo el límite de 128GB, caídas de throughput o antes de runs largos en GB10.
- [Slo Implementation](https://skillsagentes.com/skills/wshobson/agents/slo-implementation.md): Define e implementa Indicadores (SLI) y Objetivos (SLO) de nivel de servicio con error budgets y alertas, para establecer metas de fiabilidad y prácticas SRE.

---

Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)