# Vision Sft > Haz fine-tuning supervisado de modelos visión-lenguaje (VLMs) con datos de imagen+texto. Úsalo al adaptar un VLM a un dominio visual, configurar LoRA con torre de visión congelada, o depurar un fine-tune que entrena sin aprender. Fuente: https://skillsagentes.com/skills/wshobson/agents/vision-sft Markdown: https://skillsagentes.com/skills/wshobson/agents/vision-sft.md Repositorio: https://github.com/wshobson/agents Autor: wshobson Licencia: MIT Actualizado: hace 2 meses Coste de contexto: 58 tok instalada, 2k tok al activarse, 3.5k tok con todos los archivos del bundle Bundle: 2 archivos, 14 KB Permisos que pide: ninguno declarado ## Instalación Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina. ```bash # Claude Code npx -y skills add wshobson/agents --skill vision-sft --agent claude-code # Cursor npx -y skills add wshobson/agents --skill vision-sft --agent cursor # Codex npx -y skills add wshobson/agents --skill vision-sft --agent codex # Gemini CLI npx -y skills add wshobson/agents --skill vision-sft --agent gemini # Windsurf npx -y skills add wshobson/agents --skill vision-sft --agent windsurf # Cline npx -y skills add wshobson/agents --skill vision-sft --agent cline ``` ## Qué hace - Fija la receta por defecto: torre de visión y proyector congelados, LoRA solo en la LLM (r=8–16, α=16–32) - Da criterios para decidir cuándo descongelar las últimas 6 capas ViT y con qué LR relativo - Establece reglas de emparejamiento QLoRA + torre de visión y `finetune_vision_layers`/`fast_inference` - Detalla los dos fallos silenciosos: desajuste de placeholders de imagen y mal ajuste de min_pixels/max_pixels - Indica el collator correcto (`UnslothVisionDataCollator`) para datos VLM en Unsloth ## Cuándo usarla - Adaptar un VLM a un dominio o tarea visual - Configurar LoRA con torre de visión congelada - Depurar un fine-tune de VLM que entrena sin errores pero no aprende ## Cuándo no - La elección del modelo base VLM (eso vive en el catálogo de modelos de `finetuning-method-selection`) - RL sobre VLM (VLM-GRPO), que se cubre en `grpo-rlvr-training` - Preparación del dataset de imagen+texto, cubierta por `dataset-curation` ## Qué la activa - "Quiero hacer fine-tuning de un VLM con LoRA para imágenes médicas" - "Mi VLM entrena con loss normal pero no mejora en eval, ¿qué reviso?" - "¿Debo congelar la torre de visión o descongelar las últimas capas ViT?" - "Necesito configurar min_pixels y max_pixels para mi dataset de documentos" ## Antes de instalar - Asume que ya se eligió un modelo VLM base y que el dataset es de imagen+texto (no pares de preferencia ni recompensa verificable), tras pasar por `finetuning-method-selection`. ## Archivos - SKILL.md — 8 KB - references/collators-and-pitfalls.md — 6 KB ## SKILL.md Reproducido tal cual desde wshobson/agents bajo MIT. Esta sección es el documento original y está en inglés. # Vision-Language SFT This skill assumes `finetuning-method-selection` already routed here: the data shape is image+text demonstrations, not preference pairs or a verifiable reward signal, and the base is a vision-language model rather than a text-only one. `lora-qlora-recipes` covers the text-only LoRA/QLoRA recipe this skill specializes for the vision tower and projector; read that skill first if the LoRA fundamentals (rank, alpha, target modules) aren't already familiar. **Input:** an image+text dataset and a VLM base model already picked from the model catalog. **Output format:** a validated adapter config — which components are frozen, LoRA target modules, and a `min_pixels`/`max_pixels` budget — that `llm-finetuning-training-engineer` consumes directly when it generates a runnable script. ## Quick Reference | Situation | Default | |---|---| | Adapting behavior on familiar images | Frozen tower+projector, LoRA r=8–16, α=16–32 | | Visual domain shift | Unfreeze last-6 ViT layers, vision LR 5–10x lower | | Doesn't fit in bf16 at target rank | QLoRA — frozen vision tower only | | `fast_inference=True` | `finetune_vision_layers=False` | | Loss normal, eval not improving | Check the Two Silent Killers below first | ## The Consensus Recipe Freeze the vision tower and the projector. Put LoRA on the LLM only, all-linear (the same attention + MLP target list as text-only SFT — see `lora-qlora-recipes`), at **r=8–16, α=16–32**. This is the settled default for adapting a VLM's behavior without disturbing how it sees. - **The vision tower and projector stay frozen by default.** They already encode a general visual representation; retraining them is rarely necessary and adds risk without adding capability for most tasks. - **LoRA rank runs lower than the text-only general default** (r=8–16 here vs r=16–32 for text-only SFT) because the LLM-only adapter is adapting behavior, not injecting new visual knowledge. - **QLoRA is permitted only with a frozen vision tower.** Quantizing the base while also unfreezing and training vision layers is unsupported and unstable — treat this as a hard pairing rule, not a tunable. If the vision tower needs to unfreeze, drop QLoRA and use bf16 LoRA instead. ```python # freeze tower + projector; LoRA on LLM only for name, param in model.named_parameters(): if "vision_tower" in name or "projector" in name: param.requires_grad = False target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ] # LLM-only, all-linear — r=8-16, alpha=16-32 ``` ## When to Unfreeze Unfreezing vision layers is a deliberate escalation, not a default decision — reach for it only when the domain shift is visual, not textual. - **Unfreeze only for visual domain shift.** If the task is teaching new behavior on images the tower already understands (charts, everyday photos), the frozen-tower recipe above is sufficient. Unfreeze when the visual domain itself is unfamiliar to the tower — satellite imagery, medical scans, dense technical diagrams — and the frozen-tower recipe plateaus. - **Last-6 ViT layers is the sweet spot.** Unfreezing the final six vision-transformer layers (not the whole tower) measured **+1.7pt DocVQA at ~1.75x training cost** over the frozen baseline. Treat six layers as the ceiling worth paying for; going further spends compute without a matched result. - **Vision LR must run 5–10x lower than the LLM LR when unfrozen.** The vision tower's pretrained representation is more fragile than the LLM's adapter; the same LR for both risks overwriting the visual representation faster than the LLM adapter can compensate. - **High LoRA rank on the patch- embedding layer risks NaN.** If patch embedding is in the unfrozen set, keep its rank low and watch early-step loss closely — one of the most fragile places to apply LoRA in a VLM. ## The Two Silent Killers Both produce a run that trains without error and without learning: the loss curve looks normal, the model doesn't improve, and neither throws an exception — both need an explicit pre-training check, not just a clean training log. - **Image-tag/count mismatch.** Every image placeholder token in the templated text must map 1:1 to a media item actually passed to the collator. A mismatch (one placeholder, zero or two images attached; or an image with no placeholder) doesn't error in most collators — it silently misaligns image and text, and the model "trains but learns nothing." Validate the 1:1 placeholder-to-media mapping before training starts, on every example, not just a sample. Full validation-checklist detail: `references/collators-and-pitfalls.md`. - **`min_pixels`/`max_pixels` resolution budget.** This pair is the single most consequential hyperparameter for quality and memory in VLM SFT — more than rank, alpha, or LR. Too low silently downsamples images below what the task needs (small document text becomes unreadable even though training "succeeds"); too high blows the activation memory budget or forces too small a batch to train stably. Set it deliberately per dataset, don't leave it at a framework default. ## Unsloth Specifics - **`UnslothVisionDataCollator`** is the collator Unsloth expects for VLM SFT — it handles the image-tag alignment and per-architecture processor contract described in `references/collators-and-pitfalls.md`. Don't substitute a text-only collator for VLM data. - **`finetune_vision_layers=False` is required when `fast_inference=True`.** vLLM cannot serve LoRA adapters on vision layers, so a fast- inference setup that also unfreezes vision layers fails at serve time even if training succeeds. If the recipe calls for unfreezing the last-6 ViT layers (see When to Unfreeze above), fast inference is off the table for that run — choose one or the other, not both. ## Model Choice Base VLM choice is out of scope for this skill — it lives in one place, the model catalog at `finetuning-method-selection`'s `references/model-catalog.md`. This skill and its references describe recipes by architecture family only, never by recommending one model over another. VLM reinforcement learning (VLM-GRPO) is reference-only in this plugin — the fragmented tooling and reward-hacking failure modes specific to VLM-RL are covered in `grpo-rlvr-training`, not here. This skill's scope stops at supervised fine-tuning. ## Failure Modes The recurring mistake across every section above is treating a clean loss curve as proof the run is healthy. A normal-looking curve is consistent with **both** a working run **and** either silent killer, since the model trains on *something* either way — just not the aligned image-text signal when a killer is present. A flat eval score next to a normal loss curve means re-run the checklist in `references/collators-and-pitfalls.md` before touching any hyperparameter. ## References - `references/collators-and-pitfalls.md` — per- architecture collator table, dataset-format examples with image placeholders, a pre- training validation checklist, and the two- stage projector-alignment recipe as an advanced pattern. Related skills: `finetuning-method-selection` routes here; `lora-qlora-recipes` covers the text-only LoRA fundamentals this skill specializes; `grpo-rlvr-training` covers VLM-RL (reference-only); `dataset-curation` covers image+text dataset preparation this skill doesn't. ## Dónde encaja - Categoría: [Aprendizaje](https://skillsagentes.com/categorias/aprendizaje.md) — Explicaciones, tutorías y flujos de estudio estructurado. - Creador: [wshobson](https://skillsagentes.com/creators/wshobson.md) — 183 skills en el directorio - [Todas las skills](https://skillsagentes.com/skills.md) - [Ranking de instalaciones](https://skillsagentes.com/ranking.md) ## Otras skills del mismo repositorio - [Hermes Tweet](https://skillsagentes.com/skills/wshobson/agents/hermes-tweet.md): Instala y opera Hermes Tweet, un plugin de Hermes Agent para investigar X/Twitter, leer timelines, analizar tweets y ejecutar operaciones privadas o de cambio de estado con aprobación previa. - [Superself](https://skillsagentes.com/skills/wshobson/agents/superself.md): Úsalo cuando un proyecto guarda su estado en Superself: lee `self context` al iniciar sesión, vincula el trabajo a una work unit, reporta con evidencia y registra decisiones confirmadas. - [Grounded Vault](https://skillsagentes.com/skills/wshobson/agents/grounded-vault.md): Úsalo para mantener un almacén Markdown de conocimiento donde cada afirmación compilada se rastrea hasta una fuente inmutable y el drift se detecta con git diff sin gastar tokens. - [Postgresql Table Design](https://skillsagentes.com/skills/wshobson/agents/postgresql-table-design.md): Úsalo al diseñar o revisar un esquema específico de PostgreSQL: buenas prácticas, tipos de datos, indexación, restricciones, patrones de rendimiento y funciones avanzadas. - [Prompt Engineering Patterns](https://skillsagentes.com/skills/wshobson/agents/prompt-engineering-patterns.md): Úsalo cuando pidan optimizar un prompt, mejorar su rendimiento, diseñar una plantilla, aplicar chain-of-thought, few-shot prompting o técnicas avanzadas de prompt engineering para producción. ## Skills relacionadas - [Grpo Rlvr Training](https://skillsagentes.com/skills/wshobson/agents/grpo-rlvr-training.md): Entrena comportamiento de razonamiento y tareas verificables con GRPO y RL a partir de recompensas verificables (RLVR): cuándo aplicar RL, la receta de referencia y el gate de inspección de rewards. - [Eval Harness First](https://skillsagentes.com/skills/wshobson/agents/eval-harness-first.md): Construye el eval harness que condiciona cada fine-tuning: golden sets, graders por modo de fallo, calibración de juez y baselines del modelo base. - [Review Agent Setup](https://skillsagentes.com/skills/wshobson/agents/review-agent-setup.md): Configura un gating humano para las acciones de revisión de agentes IA en Claude Code, con un rastro de aprobación auditable criptográficamente y gates aplicados con Cedar. - [Competitive Landscape](https://skillsagentes.com/skills/wshobson/agents/competitive-landscape.md): Analiza la competencia, identifica oportunidades de diferenciación y desarrolla posicionamiento de mercado ganador usando las Cinco Fuerzas de Porter, Blue Ocean Strategy y mapas de posicionamiento. - [Brand Landingpage](https://skillsagentes.com/skills/wshobson/agents/brand-landingpage.md): Diseñador de landing pages centrado en marca: hace una entrevista de identidad visual y luego genera e itera la página con Stitch, entregando HTML listo para desplegar. --- Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)