# Test Flakiness

> Detecta tests no deterministas (flaky) leyendo logs de CI o histórico de resultados; agrega tasas de aprobación, recomienda cuarentena o fix y mantiene un registro de tests flaky.

Fuente: https://skillsagentes.com/skills/donchitos/claude-code-game-studios/test-flakiness
Markdown: https://skillsagentes.com/skills/donchitos/claude-code-game-studios/test-flakiness.md

Repositorio: https://github.com/Donchitos/Claude-Code-Game-Studios
Autor: Donchitos
Licencia: MIT
Actualizado: hace 3 meses
Coste de contexto: 69 tok instalada, 2k tok al activarse, 2k tok con todos los archivos del bundle
Bundle: 1 archivo, 8 KB
Permisos que pide: read, glob, grep, write, edit, bash

## Instalación

Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina.

```bash
# Claude Code
npx -y skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness --agent claude-code
# Cursor
npx -y skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness --agent cursor
# Codex
npx -y skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness --agent codex
# Gemini CLI
npx -y skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness --agent gemini
# Windsurf
npx -y skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness --agent windsurf
# Cline
npx -y skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness --agent cline
```

## Qué hace

- Parsea logs de CI o históricos de resultados de tests para detectar tests inestables (flaky)
- Calcula tasas de fallo por test y clasifica la causa probable (timing, orden, seed, fuga de recursos, etc.)
- Recomienda cuarentena, corrección o monitoreo según el nivel de flakiness
- Actualiza la sección de cuarentena en tests/regression-suite.md
- Genera un informe opcional en production/qa/flakiness-report-[date].md

## Cuándo usarla

- En la fase de Polish, cuando ya hay muchas ejecuciones de CI con señal estadística fiable
- Cuando los desarrolladores empiezan a descartar fallos de CI como 'probablemente flaky'
- Después de que /regression-suite identifique tests en cuarentena que necesitan diagnóstico

## Qué la activa

- "Analiza este log de CI para detectar tests flaky"
- "/test-flakiness scan"
- "/test-flakiness registry"
- "Revisa los tests en cuarentena de regression-suite.md"

## Antes de instalar

- Necesita historial de resultados de al menos varias ejecuciones de test (logs de CI en .github/, test-results/ o Saved/Logs/).
- runs shell commands
- writes to your files

## Archivos

- SKILL.md — 8 KB

## SKILL.md

Reproducido tal cual desde Donchitos/Claude-Code-Game-Studios bajo MIT. Esta sección es el documento original y está en inglés.

# Test Flakiness Detection

A flaky test is one that sometimes passes and sometimes fails without any code
change. Flaky tests are worse than no tests in some ways — they train the team
to ignore red CI runs, masking genuine failures. This skill identifies them,
explains likely causes, and recommends whether to quarantine or fix each one.

**Output:** Updated `tests/regression-suite.md` quarantine section + optional
`production/qa/flakiness-report-[date].md`

**When to run:**
- Polish phase (tests have had many runs; statistical signal is reliable)
- When developers start dismissing CI failures as "probably flaky"
- After `/regression-suite` identifies quarantined tests that need diagnosis

---

## 1. Parse Arguments

**Modes:**
- `/test-flakiness [ci-log-path]` — analyse a specific CI run log file
- `/test-flakiness scan` — scan all available CI logs in `.github/` or
  standard log output directories
- `/test-flakiness registry` — read existing regression-suite.md quarantine
  section and provide remediation guidance for already-known flaky tests
- No argument — auto-detect: run `scan` if CI logs are accessible, else
  `registry`

---

## 2. Locate CI Log Data

### Option A — GitHub Actions (preferred)

Check for test result artifacts:
```bash
ls -t .github/ 2>/dev/null
ls -t test-results/ 2>/dev/null
```

For Godot projects: GdUnit4 outputs XML results compatible with JUnit format.
Check `test-results/` for `.xml` files.

For Unity projects: game-ci test runner outputs NUnit XML to `test-results/`
by default.

For Unreal projects: automation logs go to `Saved/Logs/`. Grep for
`Result: Success` and `Result: Fail` patterns.

### Option B — Local log files

If a path argument is provided, read that file directly.

### Option C — No log data available

If no logs found:
> "No CI log data found. To detect flaky tests, this skill needs test result
> history from multiple runs. Options:
> 1. Run the test suite at least 3 times and collect the output logs
> 2. Check CI pipeline output and save a log to `test-results/`
> 3. Run `/test-flakiness registry` to review tests already flagged as flaky
>    in `tests/regression-suite.md`"

Stop and ask the user which option to pursue.

---

## 3. Parse Test Results

For each CI log or result file found, parse:

**JUnit XML format** (GdUnit4 / Unity):
- Grep for `<testcase name=` to get test names
- Grep for `<failure` or `<error` to identify failures
- Parse `classname` and `name` attributes for full test identifiers

**Plain text logs**:
- Grep for pass/fail patterns:
  - Godot: `PASSED` / `FAILED` adjacent to test names
  - Unreal: `Result: Success` / `Result: Fail`
  - Unity: `Test passed` / `Test failed`

Build a table: `test_id → [run1_result, run2_result, run3_result, ...]`

---

## 4. Identify Flaky Tests

A test is **flaky** if it appears in the result history with both PASS and
FAIL outcomes across runs with no code changes between them.

Flakiness thresholds:
- **High flakiness**: Fails in >25% of runs — quarantine immediately
- **Moderate flakiness**: Fails in 5–25% of runs — investigate and fix soon
- **Low/suspected flakiness**: Fails in 1–5% of runs — monitor; may be
  genuinely rare failure

For each flaky test, classify the likely cause:

### Cause classification

| Cause | Symptoms | Fix direction |
|-------|----------|---------------|
| **Timing / async** | Fails after awaiting signals or timers; pass rate correlates with system load | Add explicit await/synchronisation; avoid time-based delays |
| **Order dependency** | Fails when run after specific other tests; passes in isolation | Add proper setup/teardown; ensure test isolation |
| **Random seed** | Fails intermittently with no pattern; involves RNG | Pass explicit seed; don't use `randf()` in tests |
| **Resource leak** | Fails more often later in a test run | Fix cleanup in teardown; check orphan nodes (Godot) or object disposal (Unity) |
| **External state** | Fails when a file, scene, or global exists from a prior test | Isolate test from file system; use in-memory mocks |
| **Floating point** | Fails on comparisons like `== 0.5` | Use epsilon comparison (`is_equal_approx`, `Assert.AreApproximately`) |
| **Scene/prefab load race** | Fails when scenes are not yet ready | Await one frame after instantiation; use `await get_tree().process_frame` |

Use Grep to check the test file for timing calls, randf, global state access,
or equality comparisons on floats to narrow down the cause.

---

## 5. Recommend Action

For each flaky test:

**Quarantine (High flakiness):**
> "Quarantine this test immediately. Disable it in CI by adding
> `@pytest.mark.skip` / `[Ignore]` / `GdUnitSkip` annotation. Log it in
> `tests/regression-suite.md` quarantine section. The test is now opt-in only.
> Fix the root cause before removing quarantine."

**Investigate and fix soon (Moderate):**
> "This test is intermittently unreliable. Root cause appears to be [cause].
> Suggested fix: [specific fix based on cause classification]. Do not quarantine
> yet — fix the test directly."

**Monitor (Low/suspected):**
> "This test shows suspected flakiness. Collect more run data before
> quarantining. Note it as 'suspected' in the regression suite."

---

## 6. Generate Reports

### In-conversation summary

```
## Flakiness Detection Results

**Runs analysed**: [N]
**Tests tracked**: [N]

### Flaky Tests Found

| Test | System | Fail Rate | Likely Cause | Recommendation |
|------|--------|-----------|--------------|----------------|
| [test_name] | [system] | [N]% | Timing | Quarantine + fix async |
| [test_name] | [system] | [N]% | Float comparison | Fix: use epsilon compare |
| [test_name] | [system] | [N]% | Order dependency | Investigate teardown |

### Clean Tests (no flakiness detected)

[N] tests ran across [N] runs with consistent results — no flakiness detected.

### Data Limitations

[Note if fewer than 5 runs were available — fewer runs = less statistical confidence]
```

---

## 7. Update Regression Suite + Optional Report File

Ask: "May I update the quarantine section of `tests/regression-suite.md`
with the flaky tests found?"

If yes: use `Edit` to append entries to the Quarantined Tests table.
Never remove existing quarantine entries — only add new ones.

Ask (separately): "May I write a full flakiness report to
`production/qa/flakiness-report-[date].md`?"

The full report includes per-test analysis with cause details and
engine-specific fix snippets.

After writing:

- For each quarantined test: "Add the engine-specific skip annotation to
  disable this test in CI. Re-enable after the root cause is fixed."
- For fix-eligible tests: "The fix for [test] is straightforward —
  change the equality comparison on line [N] to use `is_equal_approx`."
- Summary: "Once all quarantine annotations are applied, CI should run green.
  Schedule fix work for the [N] quarantined tests before the release gate."

---

## Collaborative Protocol

- **Never delete test files** — quarantine means annotate + list, not remove
- **Statistical confidence matters** — with < 3 runs, flag findings as
  "suspected" not "confirmed"; ask if more run data is available
- **Fix is always the goal** — quarantine is temporary; surface the fix
  direction even when recommending quarantine
- **Ask before writing** — both the regression-suite update and the report
  file require explicit approval. On write: Verdict: **COMPLETE** — flakiness report written. On decline: Verdict: **BLOCKED** — user declined write.
- **Flakiness in CI is a team problem** — surface the list and recommended
  actions clearly; do not just silently quarantine without the team knowing

## Dónde encaja

- Categoría: [Testing y QA](https://skillsagentes.com/categorias/testing-qa.md) — Flujos de testing unitario, de integración y end-to-end.
- Creador: [Donchitos](https://skillsagentes.com/creators/donchitos.md) — 73 skills en el directorio
- [Todas las skills](https://skillsagentes.com/skills.md)
- [Ranking de instalaciones](https://skillsagentes.com/ranking.md)

## Otras skills del mismo repositorio

- [Team Qa](https://skillsagentes.com/skills/donchitos/claude-code-game-studios/team-qa.md): Orquesta al equipo de QA (qa-lead y qa-tester) para producir un paquete completo de QA: plan de pruebas, smoke check, casos de prueba, ejecución manual y reporte de sign-off.
- [Consistency Check](https://skillsagentes.com/skills/donchitos/claude-code-game-studios/consistency-check.md): Compara todos los GDDs contra el registro de entidades para detectar inconsistencias entre documentos: mismo stat, ítem o fórmula con valores distintos, usando un enfoque grep-first.
- [Create Architecture](https://skillsagentes.com/skills/donchitos/claude-code-game-studios/create-architecture.md): Redacción guiada, sección por sección, del documento de arquitectura maestro del juego, consciente de la versión del motor y de sus posibles brechas de conocimiento.
- [Day One Patch](https://skillsagentes.com/skills/donchitos/claude-code-game-studios/day-one-patch.md): Prepara un patch day-one para el lanzamiento de un juego: define alcance, prioriza, implementa y valida con un QA gate ligero, como un mini-sprint con plan de rollback.
- [Hotfix](https://skillsagentes.com/skills/donchitos/claude-code-game-studios/hotfix.md): Flujo de arreglo urgente que se salta el proceso normal de sprint pero deja rastro de auditoría completo: crea rama hotfix, registra aprobaciones y asegura el backport correcto.

---

Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)