# Benchmark E2e

> End-to-end benchmark suite for vercel-plugin. Runs realistic projects through skill injection, launches dev servers, verifies everything works, analyzes conversation logs, and produces an improvement report for overnight self-improvement loops.

Fuente: https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-e2e
Markdown: https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-e2e.md

Repositorio: https://github.com/vercel/vercel-plugin
Autor: vercel
Licencia: NOASSERTION
Actualizado: hace 5 meses
Coste de contexto: 61 tok instalada, 1.3k tok al activarse, 1.3k tok con todos los archivos del bundle
Bundle: 1 archivo, 5 KB
Permisos que pide: ninguno declarado

## Instalación

Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina.

```bash
# Claude Code
npx -y skills add vercel/vercel-plugin --skill benchmark-e2e --agent claude-code
# Cursor
npx -y skills add vercel/vercel-plugin --skill benchmark-e2e --agent cursor
# Codex
npx -y skills add vercel/vercel-plugin --skill benchmark-e2e --agent codex
# Gemini CLI
npx -y skills add vercel/vercel-plugin --skill benchmark-e2e --agent gemini
# Windsurf
npx -y skills add vercel/vercel-plugin --skill benchmark-e2e --agent windsurf
# Cline
npx -y skills add vercel/vercel-plugin --skill benchmark-e2e --agent cline
```

## Antes de instalar

- Necesita en el PATH: bun

## Archivos

- SKILL.md — 5 KB

## SKILL.md

Reproducido tal cual desde vercel/vercel-plugin bajo NOASSERTION. Esta sección es el documento original y está en inglés.

# Benchmark E2E

Single-command pipeline that creates projects, exercises skill injection via `claude --print`, launches dev servers, verifies they work, analyzes conversation logs, and generates actionable improvement reports.

## Quick Start

```bash
# Full suite (9 projects, ~2-3 hours)
bun run scripts/benchmark-e2e.ts

# Quick mode (first 3 projects, ~30-45 min)
bun run scripts/benchmark-e2e.ts --quick
```

Options:

| Flag | Description | Default |
|------|-------------|---------|
| `--quick` | Run only first 3 projects | `false` |
| `--base <path>` | Override base directory | `~/dev/vercel-plugin-testing` |
| `--timeout <ms>` | Per-project timeout (forwarded to runner) | `900000` (15 min) |

## Pipeline Stages

The orchestrator chains four stages sequentially, aborting on failure:

1. **runner** — Creates test dirs, installs plugin, runs `claude --print` with `VERCEL_PLUGIN_LOG_LEVEL=trace`
2. **verify** — Detects package manager, launches dev server, polls for 200 with non-empty HTML
3. **analyze** — Matches JSONL sessions to projects via `run-manifest.json`, extracts metrics
4. **report** — Generates `report.md` and `report.json` with scorecards and recommendations

## Contracts

### `run-manifest.json`

Written by the runner at `<base>/results/run-manifest.json`. Links all downstream stages to the same run.

```typescript
interface BenchmarkRunManifest {
  runId: string;           // UUID for this pipeline run
  timestamp: string;       // ISO 8601
  baseDir: string;         // Absolute path to base directory
  projects: Array<{
    slug: string;          // e.g. "01-recipe-platform"
    cwd: string;           // Absolute path to project dir
    promptHash: string;    // SHA hash of the prompt text
    expectedSkills: string[];
  }>;
}
```

The analyzer and verifier read this manifest to correlate sessions precisely instead of guessing from directory listings.

### `events.jsonl`

The orchestrator writes NDJSON events to `<base>/results/events.jsonl` tracking pipeline lifecycle:

```jsonc
// Each line is one JSON object:
{ "stage": "pipeline", "event": "start", "timestamp": "...", "data": { "baseDir": "...", "quick": false } }
{ "stage": "runner",   "event": "start", "timestamp": "...", "data": { "script": "...", "args": [...] } }
{ "stage": "runner",   "event": "complete", "timestamp": "...", "data": { "exitCode": 0, "durationMs": 120000 } }
// On failure:
{ "stage": "verify",   "event": "error", "timestamp": "...", "data": { "exitCode": 1, "durationMs": 5000, "slug": "04-conference-tickets" } }
{ "stage": "pipeline", "event": "abort", "timestamp": "...", "data": { "failedStage": "verify", "exitCode": 1, "slug": "04-conference-tickets" } }
```

### `report.json`

Machine-readable report at `<base>/results/report.json` for programmatic consumption:

```typescript
interface ReportJson {
  runId: string | null;
  timestamp: string;
  verdict: "pass" | "partial" | "fail";
  gaps: Array<{
    slug: string;
    expected: string[];
    actual: string[];
    missing: string[];
  }>;
  recommendations: string[];
  suggestedPatterns: Array<{
    skill: string;   // Skill that was expected but not injected
    glob: string;    // Suggested pathPattern glob
    tool: string;    // Tool name that should trigger injection
  }>;
}
```

## Overnight Automation Loop

Run the pipeline repeatedly with a cooldown between iterations:

```bash
while true; do
  bun run scripts/benchmark-e2e.ts
  sleep 3600
done
```

Each run produces timestamped `report.json` and `report.md` files. Compare across runs to track improvement.

## Self-Improvement Cycle

The pipeline enables a closed feedback loop:

1. **Run** — `bun run scripts/benchmark-e2e.ts` exercises the plugin against realistic projects
2. **Read gaps** — `report.json` lists which skills were expected but never injected, with exact slugs
3. **Apply fixes** — Use `suggestedPatterns` entries (copy-pasteable YAML) to add missing frontmatter patterns; use `recommendations` to fix hook logic
4. **Re-run** — Execute the pipeline again to verify the gaps are closed
5. **Compare** — Diff `report.json` across runs: `verdict` should trend from `"fail"` → `"partial"` → `"pass"`

For overnight automation, combine with the loop above. Wake up to reports showing exactly what improved and what still needs work.

## Prompt Table

Prompts never name specific technologies — they describe the product and features, letting the plugin infer which skills to inject.

| # | Slug | Expected Skills |
|---|------|----------------|
| 01 | recipe-platform | auth, vercel-storage, nextjs |
| 02 | trivia-game | vercel-storage, nextjs |
| 03 | code-review-bot | ai-sdk, nextjs |
| 04 | conference-tickets | payments, email, auth |
| 05 | content-aggregator | cron-jobs, ai-sdk |
| 06 | finance-tracker | cron-jobs, email |
| 07 | multi-tenant-blog | routing-middleware, cms, auth |
| 08 | status-page | cron-jobs, vercel-storage, observability |
| 09 | dog-walking-saas | payments, auth, vercel-storage, env-vars |

## Cleanup

```bash
rm -rf ~/dev/vercel-plugin-testing
```

## Dónde encaja

- Categoría: [Testing y QA](https://skillsagentes.com/categorias/testing-qa.md) — Flujos de testing unitario, de integración y end-to-end.
- Creador: [vercel](https://skillsagentes.com/creators/vercel.md) — 79 skills en el directorio
- [Todas las skills](https://skillsagentes.com/skills.md)
- [Ranking de instalaciones](https://skillsagentes.com/ranking.md)

## Otras skills del mismo repositorio

- [Knowledge Update](https://skillsagentes.com/skills/vercel/vercel-plugin/knowledge-update.md): Corrects outdated LLM knowledge about the Vercel platform and introduces new products. Injected at session start.
- [Vercel Connect](https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-connect.md): Vercel Connect expert guidance — securely obtain scoped OAuth tokens for third-party services (Slack, GitHub, MCP servers, OAuth, Snowflake) on behalf of apps or users via Vercel OIDC. Use when wiring up third-party API access, connecting to MCP servers, sending Slack messages, accessing GitHub APIs, receiving webhook events from Slack/Linear/GitHub and forwarding them to your agents and apps, or building eve agent connections.
- [Vercel Functions](https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-functions.md): Vercel Functions expert guidance — Serverless Functions, Edge Functions, Fluid Compute, streaming, Cron Jobs, and runtime configuration. Use when configuring, debugging, or optimizing server-side code running on Vercel.
- [Cdn Caching](https://skillsagentes.com/skills/vercel/vercel-plugin/cdn-caching.md): Debug Vercel CDN caching — cache hit rate, stale content, revalidation behavior, ISR + PPR, per-request cache reasons (cacheReason) and PPR state (ppr_state), and costs.
- [Eve](https://skillsagentes.com/skills/vercel/vercel-plugin/eve.md): eve framework guidance for durable AI agents and agent-powered applications. Use when creating, editing, or debugging an eve project, when the user explicitly asks for eve, or when the build-agents skill has selected eve as the default framework. Covers eve's filesystem-first runtime, durable sessions, tools, skills, connections, channels, sandboxes, subagents, schedules, evals, frontend clients, and Agent Runs observability. Do not use for incidental agent mentions, generic agent-building prompts, or established non-eve stacks unless the user asks for comparison or migration.

## Skills relacionadas

- [Benchmark Agents](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-agents.md): Advanced AI agent benchmark scenarios that push Vercel's cutting-edge platform features — Workflow SDK, AI Gateway, MCP, Chat SDK, Queues, Flags, Sandbox, and multi-agent orchestration. Designed to stress-test skill injection for complex, multi-system builds.
- [Plugin Audit](https://skillsagentes.com/skills/vercel/vercel-plugin/plugin-audit.md): Audit vercel-plugin performance on real-world projects. Extracts tool calls from Claude Code conversation logs, tests hook matching against actual inputs, identifies pattern coverage gaps, and checks plugin cache staleness. Use when asked to audit, test, or investigate plugin skill injection on a real project.
- [Vercel Plugin Eval](https://skillsagentes.com/skills/vercel/vercel-plugin/vercel-plugin-eval.md): Run live eval sessions against the vercel-plugin to verify hook behavior, skill injection, dedup correctness, and coverage. Launches real Claude Code sessions via WezTerm, monitors debug logs, and produces a structured coverage report.
- [Benchmark Sandbox](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-sandbox.md): Run vercel-plugin eval scenarios in Vercel Sandboxes instead of local WezTerm panels. Provisions ephemeral microVMs with Claude Code + plugin pre-installed, runs benchmark prompts, extracts hook artifacts, and produces coverage reports.
- [Benchmark Testing](https://skillsagentes.com/skills/vercel/vercel-plugin/benchmark-testing.md): Create and launch benchmark test projects to exercise vercel-plugin skill injection across realistic scenarios. Sets up isolated directories, installs the plugin, and spawns WezTerm panes running Claude Code with crafted prompts.

---

Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)