# Vercel Plugin Eval

> Run live eval sessions against the vercel-plugin to verify hook behavior, skill injection, dedup correctness, and coverage. Launches real Claude Code sessions via WezTerm, monitors debug logs, and produces a structured coverage report.

Fuente: https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-plugin-eval
Markdown: https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-plugin-eval.md

Repositorio: https://github.com/vercel/vercel-plugin
Autor: vercel
Licencia: NOASSERTION
Actualizado: hace 5 meses
Coste de contexto: 59 tok instalada, 1.2k tok al activarse, 1.2k tok con todos los archivos del bundle
Bundle: 1 archivo, 5 KB
Permisos que pide: ninguno declarado

## Instalación

Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina.

```bash
# Claude Code
npx -y skills add vercel/vercel-plugin --skill vercel-plugin-eval --agent claude-code
# Cursor
npx -y skills add vercel/vercel-plugin --skill vercel-plugin-eval --agent cursor
# Codex
npx -y skills add vercel/vercel-plugin --skill vercel-plugin-eval --agent codex
# Gemini CLI
npx -y skills add vercel/vercel-plugin --skill vercel-plugin-eval --agent gemini
# Windsurf
npx -y skills add vercel/vercel-plugin --skill vercel-plugin-eval --agent windsurf
# Cline
npx -y skills add vercel/vercel-plugin --skill vercel-plugin-eval --agent cline
```

## Antes de instalar

- Necesita en el PATH: node, npx
- Variables de entorno: CLAIMDIR, SLUG
- makes network requests
- reads environment config

## Archivos

- SKILL.md — 5 KB

## SKILL.md

Reproducido tal cual desde vercel/vercel-plugin bajo NOASSERTION. Esta sección es el documento original y está en inglés.

# Plugin Eval

Launch real Claude Code sessions with the plugin installed, monitor debug logs in real-time, and verify every hook fires correctly with proper dedup.

## DO NOT (Hard Rules)

- **DO NOT** use `claude --print` or `-p` — hooks don't fire, no files created
- **DO NOT** use `--dangerously-skip-permissions`
- **DO NOT** create projects in `/tmp/` — always use `~/dev/vercel-plugin-testing/`
- **DO NOT** manually wire hooks or create `settings.local.json` — use `npx add-plugin`
- **DO NOT** set `CLAUDE_PLUGIN_ROOT` manually
- **DO NOT** use `bash -c` in WezTerm — use `/bin/zsh -ic`
- **DO NOT** use full path to claude — use the `x` alias
- **DO NOT** write eval scripts — do everything as Bash tool calls in the conversation

**Copy the exact commands below. Do not improvise.**

## Quick Start

**Always append a timestamp** to directory names so reruns don't overwrite old projects:

```bash
# 1. Create test dir & install plugin (with timestamp)
TS=$(date +%Y%m%d-%H%M)
SLUG="my-eval-$TS"
mkdir -p ~/dev/vercel-plugin-testing/$SLUG
cd ~/dev/vercel-plugin-testing/$SLUG
npx add-plugin https://github.com/vercel/vercel-plugin -s project -y

# 2. Launch session via WezTerm
wezterm cli spawn --cwd /Users/johnlindquist/dev/vercel-plugin-testing/$SLUG -- /bin/zsh -ic \
  "unset CLAUDECODE; VERCEL_PLUGIN_LOG_LEVEL=debug x '<PROMPT>' --settings .claude/settings.json; exec zsh"

# 3. Find debug log (wait ~25s for session start)
find ~/.claude/debug -name "*.txt" -mmin -2 -exec grep -l "$SLUG" {} +
```

## What to Monitor

### Hook firing (all 8 registered hooks)

```bash
LOG=~/.claude/debug/<session-id>.txt

# SessionStart (3 hooks)
grep "SessionStart.*success" "$LOG"

# PreToolUse skill injection
grep -c "executePreToolHooks" "$LOG"        # total calls
grep -c "provided additionalContext" "$LOG"  # injections

# UserPromptSubmit
grep "UserPromptSubmit.*success" "$LOG"

# PostToolUse validate + shadcn font-fix
grep "posttooluse-validate.*provided" "$LOG"
grep "PostToolUse:Bash.*success" "$LOG"

# SessionEnd cleanup
grep "SessionEnd" "$LOG"
```

### Dedup correctness (the key metric)

```bash
TMPDIR=$(node -e "import {tmpdir} from 'os'; console.log(tmpdir())" --input-type=module)
CLAIMDIR="$TMPDIR/vercel-plugin-<session-id>-seen-skills.d"

# Claim files = one per skill, atomic O_EXCL
ls "$CLAIMDIR"

# Compare: injections should equal claims
inject_meta=$(grep -c "skillInjection:" "$LOG")
claims=$(ls "$CLAIMDIR" 2>/dev/null | wc -l | tr -d ' ')
echo "Injections: $((inject_meta / 3)) | Claims: $claims"
```

`skillInjection:` appears 3x per actual injection in the debug log (initial check, parsed, success). Divide by 3.

### PostToolUse validate quality

Look for real catches — API key bypass, outdated models, wrong patterns:

```bash
grep "VALIDATION" "$LOG" | head -10
```

## Scenario Design

Describe **products and features**, never name specific technologies. Let the plugin infer which skills to inject. Always end prompts with: "Link the project to my vercel-labs team so we can deploy it later. Skip any planning and just build it. Get the dev server running."

### Coverage targets by scenario type

| Scenario Type | Skills Exercised |
|--------------|-----------------|
| AI chat app | ai-sdk, ai-gateway, nextjs, ai-elements |
| Durable workflow | workflow, ai-sdk, vercel-queues |
| Monorepo | turborepo, turbopack, nextjs |
| Edge auth + routing | routing-middleware, auth, sign-in-with-vercel |
| Chat bot (multi-platform) | chat-sdk, ai-sdk, vercel-storage |
| Feature flags + CRM | vercel-flags, vercel-queues, ai-sdk |
| Email pipeline | email, satori, ai-sdk, vercel-storage |
| Marketplace/payments | payments, marketplace, cms |
| Kitchen sink | micro, ncc, all niche skills |

### Hard-to-trigger skills (8 of 44)

These need explicit technology references in the prompt because agents don't naturally reach for them:

- `ai-elements` — say "use the AI Elements component registry"
- `v0-dev` — say "generate components with v0"
- `vercel-firewall` — say "use Vercel Firewall for rate limiting"
- `marketplace` — say "publish to the Vercel Marketplace"
- `geist` — say "install the geist font package"
- `json-render` — name files `components/chat-*.tsx`

## Coverage Report

Write results to `.notes/COVERAGE.md` with:

1. **Session index** — slug, session ID, unique skills, dedup status
2. **Hook coverage matrix** — which hooks fired in which sessions
3. **Skill injection table** — which of the 44 skills triggered
4. **Dedup stats** — injections vs claims per session
5. **Issues found** — bugs, pattern gaps, validation findings

## Cleanup

```bash
rm -rf ~/dev/vercel-plugin-testing
```

## Dónde encaja

- Categoría: [Testing y QA](https://skillsagentes.com/categorias/testing-qa.md) — Flujos de testing unitario, de integración y end-to-end.
- Creador: [vercel](https://skillsagentes.com/creators/vercel.md) — 79 skills en el directorio
- [Todas las skills](https://skillsagentes.com/skills.md)
- [Ranking de instalaciones](https://skillsagentes.com/ranking.md)

## Otras skills del mismo repositorio

- [Knowledge Update](https://skillsagentes.com/skills/vercel/vercel-plugin/knowledge-update.md): Corrects outdated LLM knowledge about the Vercel platform and introduces new products. Injected at session start.
- [Vercel Connect](https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-connect.md): Vercel Connect expert guidance — securely obtain scoped OAuth tokens for third-party services (Slack, GitHub, MCP servers, OAuth, Snowflake) on behalf of apps or users via Vercel OIDC. Use when wiring up third-party API access, connecting to MCP servers, sending Slack messages, accessing GitHub APIs, receiving webhook events from Slack/Linear/GitHub and forwarding them to your agents and apps, or building eve agent connections.
- [Vercel Functions](https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-functions.md): Vercel Functions expert guidance — Serverless Functions, Edge Functions, Fluid Compute, streaming, Cron Jobs, and runtime configuration. Use when configuring, debugging, or optimizing server-side code running on Vercel.
- [Cdn Caching](https://skillsagentes.com/skills/vercel/vercel-plugin/cdn-caching.md): Debug Vercel CDN caching — cache hit rate, stale content, revalidation behavior, ISR + PPR, per-request cache reasons (cacheReason) and PPR state (ppr_state), and costs.
- [Eve](https://skillsagentes.com/skills/vercel/vercel-plugin/eve.md): eve framework guidance for durable AI agents and agent-powered applications. Use when creating, editing, or debugging an eve project, when the user explicitly asks for eve, or when the build-agents skill has selected eve as the default framework. Covers eve's filesystem-first runtime, durable sessions, tools, skills, connections, channels, sandboxes, subagents, schedules, evals, frontend clients, and Agent Runs observability. Do not use for incidental agent mentions, generic agent-building prompts, or established non-eve stacks unless the user asks for comparison or migration.

## Skills relacionadas

- [Benchmark Agents](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-agents.md): Advanced AI agent benchmark scenarios that push Vercel's cutting-edge platform features — Workflow SDK, AI Gateway, MCP, Chat SDK, Queues, Flags, Sandbox, and multi-agent orchestration. Designed to stress-test skill injection for complex, multi-system builds.
- [Benchmark E2e](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-e2e.md): End-to-end benchmark suite for vercel-plugin. Runs realistic projects through skill injection, launches dev servers, verifies everything works, analyzes conversation logs, and produces an improvement report for overnight self-improvement loops.
- [Plugin Audit](https://skillsagentes.com/skills/vercel/vercel-plugin/plugin-audit.md): Audit vercel-plugin performance on real-world projects. Extracts tool calls from Claude Code conversation logs, tests hook matching against actual inputs, identifies pattern coverage gaps, and checks plugin cache staleness. Use when asked to audit, test, or investigate plugin skill injection on a real project.
- [Benchmark Sandbox](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-sandbox.md): Run vercel-plugin eval scenarios in Vercel Sandboxes instead of local WezTerm panels. Provisions ephemeral microVMs with Claude Code + plugin pre-installed, runs benchmark prompts, extracts hook artifacts, and produces coverage reports.
- [Benchmark Testing](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-testing.md): Create and launch benchmark test projects to exercise vercel-plugin skill injection across realistic scenarios. Sets up isolated directories, installs the plugin, and spawns WezTerm panes running Claude Code with crafted prompts.

---

Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)