# Skill Autobench > Redacta un eval para una skill existente a partir de su historial real de uso (no de su spec), etiqueta casos como SPEC-DERIVED o HISTORY-IMPLIED y lo deja pendiente de aprobación humana. Fuente: https://skillsagentes.com/skills/garrytan/gbrain/skill-autobench Markdown: https://skillsagentes.com/skills/garrytan/gbrain/skill-autobench.md Repositorio: https://github.com/garrytan/gbrain Autor: garrytan Licencia: MIT Actualizado: hace 9 días Coste de contexto: 172 tok instalada, 3.1k tok al activarse, 3.3k tok con todos los archivos del bundle Bundle: 2 archivos, 13 KB Permisos que pide: ninguno declarado ## Instalación Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina. ```bash # Claude Code npx -y skills add garrytan/gbrain --skill skill-autobench --agent claude-code # Cursor npx -y skills add garrytan/gbrain --skill skill-autobench --agent cursor # Codex npx -y skills add garrytan/gbrain --skill skill-autobench --agent codex # Gemini CLI npx -y skills add garrytan/gbrain --skill skill-autobench --agent gemini # Windsurf npx -y skills add garrytan/gbrain --skill skill-autobench --agent windsurf # Cline npx -y skills add garrytan/gbrain --skill skill-autobench --agent cline ``` ## Qué hace - Mina invocaciones reales de una skill desde el archivo de conversaciones y transcripciones por harness, priorizando correcciones del usuario - Sintetiza un eval_contract y de 4 a 8 casos reproducibles con etiquetas HISTORY-IMPLIED o SPEC-DERIVED - Escribe el resultado en skills//eval/autobench-.md con status: PENDING-HUMAN-APPROVAL, sin tocar SKILL.md - Verifica la integridad del panel multi-modelo (respuestas distintas, endpoints distintos, sin duplicados byte a byte) - Clasifica fallos minados según la taxonomía fail-improve (DETERMINISTIC-CODIFIABLE, PROMPT-FIXABLE, SPEC-GAP, ROUTING-MISS) ## Cuándo usarla - Quieres escribir un eval para una skill existente basado en su uso real, no en su spec - Necesitas verificar que un panel de jueces multi-modelo realmente respondió con proveedores distintos - Quieres convertir correcciones de usuario repetidas en casos de prueba permanentes ## Cuándo no - La skill objetivo no tiene ninguna historia de invocación real (usa gbrain skillopt --bootstrap-from-skill en su lugar) ## Qué la activa - "skill autobench para skill-x" - "escribe el eval a partir del historial de uso" - "sintetiza un eval para esta skill" - "verifica el panel de eval" - "¿respondieron todos los proveedores?" ## Antes de instalar - Requiere el directorio conversations/ ya poblado (por la skill conversation-archive) para minar historial real. ## Archivos - SKILL.md — 12 KB - routing-eval.jsonl — 801 B ## SKILL.md Reproducido tal cual desde garrytan/gbrain bajo MIT. Esta sección es el documento original y está en inglés. # skill-autobench — write the eval from lived usage > **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) — > mining starts in the brain. Search the conversation archive before touching > raw transcript files, and never declare "no history" without having queried > the brain first. > > **Convention:** see [conventions/model-routing.md](../conventions/model-routing.md) — > mining and synthesis run on the cheap tier by default. The full multi-model > judging pass is an explicit opt-in (see Contract). The self-improving loop has three legs: an **eval**, a **variant generator** (SkillOpt), and a **replay + judge harness** (`gbrain eval cross-modal`). The generator and the judge ship with gbrain. The persistently missing leg is the **eval author** — someone has to WRITE the eval, and a spec-derived benchmark only tests what the skill promised, not what users actually asked for or what actually went wrong. This skill writes the eval from reality instead of imagination. ## Pipeline ### 1. MINE — extract real invocation windows Substrates, in priority order: 1. **Brain conversation archive** — pages under `conversations/`, populated by the conversation-archive skill (hard dependency for this substrate: if it hasn't ingested your history yet, run it first). Search for the target skill's name, trigger phrases, and output shapes: ```bash gbrain search "" gbrain query "when did I use and what did I ask for" ``` 2. **Per-harness session transcripts, as available** — use what the harness exposes; do not assume a layout. `gbrain transcripts recent --full` reads the configured local transcript corpus (local-only by design). Claude Code keeps per-project session JSONL under `~/.claude/projects/`; other harnesses have their own session stores. Absent stores are simply skipped. From each hit, extract an **invocation window**: the user ask before the invocation, the invocation turn itself, and the 2 turns after — because that is where corrections live. **A user correction after an invocation is the gold signal**: it is a real, observed failure mode, and it becomes a `hard_fail` plus a replayable case. **FAIL-CLOSED:** if no substrate yields a single real invocation of the target skill, emit an honest no-history report (substrates checked, queries run, windows scanned, zero matches) and stop. Do NOT invent "typical" invocations. For a skill with no history, the right tool is `gbrain skillopt --bootstrap-from-skill` (spec-derived, and honest about it) — see Dedup. ### 2. SYNTH — turn windows into a proposed eval From the mined windows plus the current SKILL.md, produce: - A proposed **eval_contract**: goal, dimensions, hard_fails. Dimensions come from observed asks; hard_fails encode observed corrections. - **4-8 replayable cases**, each shaped `{input, expected_behavior, failure_mode_to_catch}` — realistic input, a checkable expected behavior, and the named failure mode the case exists to catch. - **Spec-vs-usage gaps**: "the spec says X, users consistently ask Y." **HONESTY LABELS are mandatory.** Every dimension and every case is labeled: - `HISTORY-IMPLIED` — a real mined window backs it; cite which one. - `SPEC-DERIVED` — inferred from SKILL.md only; no usage evidence. Never conflate the two. If history is thin or off-target, say so prominently at the top of the staged file ("GROUNDING WARNING: only N windows found, none exercised the core path") instead of padding with fabricated evidence. **Privacy scrub before staging:** staged evals live in the skill repo and are distributable. Mined windows contain real names, companies, and deals — rewrite every case onto placeholder slugs (`alice-example`, `acme-example`) before writing the file. A history-grounded case keeps its shape and failure mode, never its real entities. ### 3. STAGE — human gate, always Write `skills//eval/autobench-.md` with frontmatter `status: PENDING-HUMAN-APPROVAL`. **This skill NEVER rewrites SKILL.md** — not the eval_contract, not the body, not the triggers. Merging the staged eval is the human's decision. (This is a workflow contract the agent must honor, not a mechanically-enforced gate.) ## The loop (after approval) 1. Human reviews, edits, and approves the staged eval; the approved eval_contract is merged into the skill's frontmatter explicitly. 2. Convert approved cases into `skills//skillopt-benchmark.jsonl` lines and run `gbrain skillopt ` — this is the SkillOpt surface extension: a history-grounded benchmark replacing the spec-derived bootstrap. 3. Judge outputs through the native gate: ```bash gbrain eval cross-modal --task "" --output ``` 4. Re-run autobench after more usage accumulates; diff against the prior staged baseline. ## Panel integrity — trust no aggregate Multi-model judging is only as good as the panel being real. The silent failure class: a model-id normalizer strips provider prefixes and every "different model" call lands on one host, so a "3-frontier consensus" is one model's opinion in a trench coat. Before trusting any multi-model verdict, assert over the result object: 1. **Each named model returned a non-empty response.** An empty or failed slot is the first tell of a collapse. 2. **Responses came from DISTINCT provider endpoints.** If three "different models" all report the same provider, the panel collapsed to one host. 3. **No two "different models" returned byte-identical output.** If two differently-named models return the same bytes, they are the same model. This catches a collapse even when provider metadata is missing or faked. `gbrain eval cross-modal` already exits 2 (INCONCLUSIVE) when fewer than 2/3 models return parseable scores; the byte-identical duplicate check and the distinct-endpoint check are the independent backstops this skill layers on top. Run them over the receipt JSON (written to the receipt dir) before treating a PASS/FAIL as authoritative. The integrity check is pure assertion logic over an existing result — it never calls a model itself: no network, no cost. ## Fail-improve taxonomy — what mined failures become Classify each mined correction/failure case by its cheapest durable fix: | Class | Signal in history | Durable fix | |---|---|---| | DETERMINISTIC-CODIFIABLE | An LLM fallback repeatedly handles the same input shape (regex, parsing, slugs, dates) | Convert to deterministic code + a permanent test case. The LLM is not the solution; it is the training-data generator for the code that replaces it. | | PROMPT-FIXABLE | The correction targets tone, format, or an omission the SKILL.md could specify | An eval case + a `gbrain skillopt` run | | SPEC-GAP | Users consistently ask for something the spec never promised | A spec-vs-usage gap observation for the human | | ROUTING-MISS | The skill fired on the wrong ask, or failed to fire | A `routing-eval.jsonl` case, not a benchmark case | Direction of travel: every fixed failure becomes a permanent test, the deterministic share rises, and the LLM-fallback share falls. Log and improve; never silently drop a mined failure. ## Contract - **Input:** a skill name that exists under `skills/`. - **Substrates:** `conversations/` archive pages (brain-first), then per-harness session transcripts as available (`gbrain transcripts recent` is local-only by design). No usable substrate → honest no-history report, never fabricated evidence. - **Output:** one staged file at `skills//eval/autobench-.md`, `status: PENDING-HUMAN-APPROVAL` — or the no-history report. Never an edit to SKILL.md, triggers, or any routing surface. - **Cost posture:** cheap-model default for mining and synthesis. The full multi-model judging pass (3 provider slots per cycle) is an explicit opt-in, and judging goes through native `gbrain eval cross-modal` — no bespoke judging harness. - **Honesty:** every dimension and case carries a SPEC-DERIVED or HISTORY-IMPLIED label; thin history is flagged, not papered over. - **Privacy:** mined cases are rewritten onto placeholder entities before staging. ## Output Format ```markdown --- skill: status: PENDING-HUMAN-APPROVAL generated: substrate: { conversation_pages: N, transcript_files: M, windows: K, corrections: C } --- # Autobench: — ## Grounding ## Proposed eval_contract goal / dimensions (each labeled HISTORY-IMPLIED|SPEC-DERIVED) / hard_fails ## Cases (4-8) ### case-01 [HISTORY-IMPLIED — window ref] input: ... expected_behavior: ... failure_mode_to_catch: ... ## Spec-vs-usage gaps - spec says X; users ask Y (windows: ...) ## Fail-improve classification - case-03 → DETERMINISTIC-CODIFIABLE (same date-format fallback, 4 windows) ``` ## Anti-Patterns - ❌ Auto-merging a synthesized eval into SKILL.md. The human gate is the contract. - ❌ Presenting SPEC-DERIVED dimensions as history-grounded — fabricating usage evidence is the cardinal sin. - ❌ Mining nothing and still emitting a confident eval. Fail loudly or label honestly. - ❌ Trusting "3 models scored it 8/10" without checking that three providers actually returned distinct, non-identical responses. - ❌ Calling a model inside the panel-integrity check — it is pure assertion logic over a result object. - ❌ Building a bespoke judging harness when `gbrain eval cross-modal` is the native gate. - ❌ Staging mined cases with real people/companies in them. Placeholders only. ## Dedup (sharp boundaries) - **skill-optimizer (SkillOpt, host-side)** — optimizes a skill's body against an EXISTING benchmark; its `--bootstrap-from-skill` derives tasks from the spec. THIS skill authors the benchmark from lived usage and feeds it into `skills//skillopt-benchmark.jsonl` — it extends SkillOpt's surface, never duplicates it. Read the skill-optimizer SKILL.md (on the host, where its engine lives) before the handoff. No history at all → use `--bootstrap-from-skill`, not this. - **BrainBench (`gbrain eval` suites)** — evals the ENGINE (retrieval, memory conformance, calibration). This evals SKILLS over their history. - **`skills/skillify/SKILL.md` / `skills/skill-creator/SKILL.md`** — create skills from descriptions; they don't mine lived usage. - **`skills/cross-modal-review/SKILL.md` / `gbrain eval cross-modal`** — RUN judging panels; they don't author evals. The panel-integrity assertions here verify their panels were real. - **`skills/skillpack-check/SKILL.md`** — audits skill structure/conformance, not behavior quality. - **`routing-eval.jsonl`** — tests dispatch (does the right skill fire); autobench tests behavior after dispatch. ROUTING-MISS findings route there. ## Dónde encaja - Categoría: [Testing y QA](https://skillsagentes.com/categorias/testing-qa.md) — Flujos de testing unitario, de integración y end-to-end. - Creador: [garrytan](https://skillsagentes.com/creators/garrytan.md) — 134 skills en el directorio - [Todas las skills](https://skillsagentes.com/skills.md) - [Ranking de instalaciones](https://skillsagentes.com/ranking.md) ## Otras skills del mismo repositorio - [Setup](https://skillsagentes.com/skills/garrytan/gbrain/setup.md): Configura GBrain con auto-aprovisionamiento de Supabase o PGLite, inyección en AGENTS.md y primera importación. - [Maintain](https://skillsagentes.com/skills/garrytan/gbrain/maintain.md): Chequeos de salud del brain: aplicación de back-links, auditoría de citas, validación de filing, detección de info obsoleta, páginas huérfanas y benchmarks. - [Schema Unify](https://skillsagentes.com/skills/garrytan/gbrain/schema-unify.md): Migra un brain de gbrain-base a la taxonomía de 14 tipos canónicos de gbrain-base-v2 usando gbrain onboard --check y el handler Minion unify-types. - [Retrieval Reflex](https://skillsagentes.com/skills/garrytan/gbrain/retrieval-reflex.md): Cuándo y qué recuperar: abre la página del brain de una entidad relevante antes de responder desde memoria. - [Minion Orchestrator](https://skillsagentes.com/skills/garrytan/gbrain/minion-orchestrator.md): Skill unificado de Minions para jobs deterministas de shell y orquestación de subagentes LLM: cola durable, observable y controlable, más la doctrina de ejecución durable para operaciones largas. --- Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)