# Spark Memory Thermal Ops

> Gestiona memoria unificada y térmicas durante jobs de ML largos en NVIDIA DGX Spark: headroom en GB10, OOMs en memoria unificada y monitoreo de temperatura/potencia.

Fuente: https://skillsagentes.com/skills/wshobson/agents/spark-memory-thermal-ops
Markdown: https://skillsagentes.com/skills/wshobson/agents/spark-memory-thermal-ops.md

Repositorio: https://github.com/wshobson/agents
Autor: wshobson
Licencia: MIT
Actualizado: hace 2 meses
Coste de contexto: 62 tok instalada, 2k tok al activarse, 3.4k tok con todos los archivos del bundle
Bundle: 3 archivos, 13 KB
Permisos que pide: ninguno declarado

## Instalación

Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina.

```bash
# Claude Code
npx -y skills add wshobson/agents --skill spark-memory-thermal-ops --agent claude-code
# Cursor
npx -y skills add wshobson/agents --skill spark-memory-thermal-ops --agent cursor
# Codex
npx -y skills add wshobson/agents --skill spark-memory-thermal-ops --agent codex
# Gemini CLI
npx -y skills add wshobson/agents --skill spark-memory-thermal-ops --agent gemini
# Windsurf
npx -y skills add wshobson/agents --skill spark-memory-thermal-ops --agent windsurf
# Cline
npx -y skills add wshobson/agents --skill spark-memory-thermal-ops --agent cline
```

## Qué hace

- Da un procedimiento para calcular headroom de memoria unificada antes de lanzar un entrenamiento en GB10
- Define una escalera de pasos ordenados para resolver un OOM sobre memoria unificada
- Explica cómo monitorear temperatura y potencia durante jobs largos con `assets/thermal-sample.sh`
- Aconseja cómo compartir el pool de memoria entre un trainer y un servidor de inferencia sin conflictos

## Cuándo usarla

- Dimensionar un entrenamiento contra el pool de 128GB antes de lanzarlo
- Un run hace OOM a mitad de carga o de step y hay que decidir el orden de remediación
- Vigilar temperatura y potencia en un job de varias horas para saber si hay throttling
- Planear correr un trainer junto a un servidor de inferencia (vLLM, Ollama) en la misma máquina

## Cuándo no

- Fallos en el momento de lanzamiento (ABI mismatches, flash-attn, playbooks rotos) — usar spark-training-gotchas en su lugar

## Qué la activa

- "¿Cuánta memoria libre necesito para un QLoRA de 70B en mi Spark?"
- "Mi entrenamiento hace OOM en memoria unificada, ¿qué pruebo primero?"
- "¿Por qué mi run se hizo más lento a mitad de camino, es térmico?"
- "¿Puedo correr vLLM y un fine-tune al mismo tiempo en el DGX Spark?"

## Antes de instalar

- Asume DGX Spark con chip GB10 y acceso a `free -g`; el flush de caché requiere privilegios de root.
- Necesita en el PATH: awk

## Archivos

- SKILL.md — 8 KB
- assets/thermal-sample.sh — 1017 B
- references/uma-accounting.md — 4 KB

## SKILL.md

Reproducido tal cual desde wshobson/agents bajo MIT. Esta sección es el documento original y está en inglés.

# Spark Memory & Thermal Ops

DGX Spark's GB10 chip has one 128GB unified
memory (UMA) pool shared by CPU and GPU, and a
sustained power ceiling well below its rated
figure. Both break discrete-GPU assumptions:
headroom isn't what `nvidia-smi` reports, and a
run that starts fast will slow down mid-job
with nothing misconfigured. This skill covers
planning memory headroom, working an actual
OOM, and watching thermals across a long job.
For launch-time failure modes (ABI mismatches,
flash-attn, playbook breakage), see
`spark-training-gotchas` — this skill assumes
the job starts.

## Common Issues Quick Reference

| Situation | Do this |
|---|---|
| Planning headroom before launch | Budget against `free -g`, not `nvidia-smi` — see UMA Memory Model |
| Job OOMs on unified memory | Work the OOM Ladder in order: flush, then batch/pack, then method downgrade |
| Throughput drops mid-run | Check the power/temp log before assuming a config bug — see Thermal Monitoring |
| Trainer + inference server both wanted | Run one at a time — see Concurrent Workloads |

## When to Use This Skill

- Sizing a training run against the 128GB pool
  before launch — will this model, method, and
  batch/pack combination fit.
- A run OOMs mid-load or mid-step and the
  remediation order matters — what to try first,
  second, third.
- Watching temperature and power during a
  multi-hour job, deciding whether a slowdown is
  thermal throttling or something else.
- Planning to run a trainer alongside an
  inference server (vLLM, Ollama) on the same box.

## UMA Memory Model

Spark has no separate GPU VRAM — the GPU and
CPU share one 128GB pool. Two consequences:

- **`nvidia-smi` and `cudaMemGetInfo`
  underreport pressure — or report nothing at
  all.** Both report CUDA-allocator-visible
  memory, not the pool's actual state — a box can
  show headroom in `nvidia-smi` and still OOM,
  because page-cache and mmap'd pages the
  allocator doesn't see consume the same pool. On
  some driver/setups, the memory query returns
  `[N/A], [N/A]` outright instead of a number — a
  script grepping for a numeric value there gets
  nothing, not a misleading undercount (see
  `spark-training-gotchas` gotcha G3).

- **Model load is a transient peak, not the
  steady state.** Loading safetensors weights
  mmaps the file, then copies into CUDA
  tensors — for a window during load, both the
  mmap'd pages and the CUDA copy count against
  the pool at once. A model that fits while
  training can still OOM during load if headroom
  was sized for the post-load footprint instead
  of this doubled transient.

Plan and diagnose with `free -g`, not
`nvidia-smi`:

```bash
free -g | awk 'NR==2 {print "free:", $4, "GB"}'
```

Rule of thumb: take that free figure, subtract a
few GB for OS/driver overhead, and budget against
the result — not the 128GB spec number.
The worksheet in `references/uma-accounting.md`
accepts parameter count, dtype, and method as
input, and returns a memory estimate to compare
against known anchors.

### Planning Sequence

Before launch, work through these in order:

1. Read `free -g`; subtract OS/driver overhead
   for the budget.
2. Estimate weights + optimizer + gradients +
   activations from `references/uma-accounting.md`.
3. Compare against the closest anchor (70B
   QLoRA, 27B LoRA, 9B full FT), not the
   estimate alone.
4. If the estimate is close to the budget, start
   with shorter packing or a smaller batch —
   cheaper than hitting the OOM Ladder mid-run.

### Example: Sizing a 70B QLoRA Run

A sanity check of the worksheet formula against
the ≈40GB anchor:

```python
params = 70e9
weights_gb = params * 0.5 / 1e9      # NF4, step 1
adapter_gb = 0.5                     # step 5, negligible
total_gb = weights_gb + adapter_gb   # + activations
print(f"{total_gb:.0f}GB before activations")
```

Weights alone land near the ≈40GB anchor — a plan
estimating far above that for the same model
class is a signal to recheck dtype and method.

## The OOM Ladder

When a job OOMs on unified memory, work this
ladder in order. Each step is more disruptive
than the last — don't skip ahead:
**reducing batch size is never step 1.**

1. **Flush the buffer cache.** Page cache from a
   previous run or a large dataset read often
   accounts for GB of the "missing" headroom.
   This costs nothing but a rerun and doesn't
   touch the job's configuration:

   ```bash
   sync; echo 3 > /proc/sys/vm/drop_caches
   ```

   Needs root; a between-run reset, not a
   mid-training step. See
   `spark-training-gotchas` (gotcha G3) for the
   full diagnostic behind this step.

2. **Reduce batch size or packing length.** Only
   after a flush fails to free enough headroom,
   cut batch size or packing length — the first
   step that changes what the run does. Prefer
   packing length first; it drives activation
   footprint more directly at long context.

3. **Downgrade the method: bf16 LoRA before
   QLoRA.** If flushing and shrinking batch/pack
   still OOM, drop the method a tier — bf16 LoRA
   is next, not the reverse. QLoRA's bitsandbytes
   dequantization buffers are transient CUDA-side
   allocations that can OOM before an equivalent
   bf16 LoRA run would, even though QLoRA's
   steady-state footprint is smaller. A QLoRA OOM
   is not proof the model doesn't fit.

Fall back further (smaller model, multi-Spark)
only after all three steps and the job still
won't fit.

## Thermal Monitoring

Multi-hour runs push into Spark's sustained
power ceiling, well under the rated figure —
expected platform behavior, not a symptom to
explain away:

- Sample temperature and power alongside the
  training logs, not after a slowdown is
  noticed — every 30-60 seconds correlates a
  throughput drop with a thermal event. Keep
  the CSV output format `assets/thermal-sample.sh`
  writes, so timestamps line up against the log:

  ```bash
  bash assets/thermal-sample.sh 30 thermal.log
  ```

- **A sustained ~100W power draw is the platform
  cap, not a configuration bug.** Don't re-tune
  batch size or precision to "fix" a plateau
  that's the box behaving normally under load.
  If temperature climbs while power stays flat
  under the rated 240W figure, that's the
  signature to recognize.

- Log throttle events explicitly instead of
  letting a run silently slow down unrecorded. A
  run whose per-step time doubles two hours in
  should show that in the log, correlated against
  the thermal sample at that timestamp. Full
  throttling diagnostics: `spark-training-gotchas`
  (gotcha G4).

## Concurrent Workloads

Because the 128GB pool is global, eviction
happens without either process's logs showing
an OOM:

- The one-heavy-job rule applies to **uncapped or
  near-capacity** workloads — an uncapped trainer
  and inference server (vLLM, Ollama) compete for
  the same pool. A small, capped workload doesn't:
  a <4GB LoRA fine-tune coexists fine alongside
  vLLM capped at `gpu-memory-utilization<=0.5` —
  check the other process's cap, not just its
  presence, before stopping it.

- Inference servers evict trainer pages silently
  under uncapped/near-capacity contention, and
  vice versa — neither logs an error, so a slow
  run or lost KV cache is a contention symptom to
  check for. Stop unrelated *uncapped* servers
  before a long or full-pool run.

Check for GPU-resident processes first:

```bash
ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep
```

This procedure complements `spark-training-gotchas`
(gotchas G3, G4, G6) — that skill covers launch-time
failures; this one, the running job.

Memory math worksheets:
`references/uma-accounting.md`.

## Dónde encaja

- Categoría: [DevOps e infraestructura](https://skillsagentes.com/categorias/devops-infraestructura.md) — Despliegues, contenedores, IaC y flujos de gestión de incidentes.
- Creador: [wshobson](https://skillsagentes.com/creators/wshobson.md) — 183 skills en el directorio
- [Todas las skills](https://skillsagentes.com/skills.md)
- [Ranking de instalaciones](https://skillsagentes.com/ranking.md)

## Otras skills del mismo repositorio

- [Hermes Tweet](https://skillsagentes.com/skills/wshobson/agents/hermes-tweet.md): Instala y opera Hermes Tweet, un plugin de Hermes Agent para investigar X/Twitter, leer timelines, analizar tweets y ejecutar operaciones privadas o de cambio de estado con aprobación previa.
- [Superself](https://skillsagentes.com/skills/wshobson/agents/superself.md): Úsalo cuando un proyecto guarda su estado en Superself: lee `self context` al iniciar sesión, vincula el trabajo a una work unit, reporta con evidencia y registra decisiones confirmadas.
- [Grounded Vault](https://skillsagentes.com/skills/wshobson/agents/grounded-vault.md): Úsalo para mantener un almacén Markdown de conocimiento donde cada afirmación compilada se rastrea hasta una fuente inmutable y el drift se detecta con git diff sin gastar tokens.
- [Postgresql Table Design](https://skillsagentes.com/skills/wshobson/agents/postgresql-table-design.md): Úsalo al diseñar o revisar un esquema específico de PostgreSQL: buenas prácticas, tipos de datos, indexación, restricciones, patrones de rendimiento y funciones avanzadas.
- [Prompt Engineering Patterns](https://skillsagentes.com/skills/wshobson/agents/prompt-engineering-patterns.md): Úsalo cuando pidan optimizar un prompt, mejorar su rendimiento, diseñar una plantilla, aplicar chain-of-thought, few-shot prompting o técnicas avanzadas de prompt engineering para producción.

---

Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)