Claude
Cursor
Skill
vision-sft
Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or debugging a VLM fine-tune that trains without learning.
Virus-scanned
Reviewed automatically before listing.
Download
wshobson-agents-plugins_llm-finetuning_skills_vision-sft-554237f.zip · 6 KB
Install
skills CLI
npx skills add https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/vision-sft
Claude Code
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install wshobson-agents@llmmart
Git
git clone https://github.com/wshobson/agents.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole wshobson/agents collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Vision-Language SFT
This skill assumes finetuning-method-selection
already routed here: the data shape is
image+text demonstrations, not preference pairs
or a verifiable reward signal, and the base is a
vision-language model rather than a text-only
one. lora-qlora-recipes covers the text-only
LoRA/QLoRA recipe this skill specializes for the
vision tower and projector; read that skill first
if the LoRA fundamentals (rank, alpha, target
modules) aren't already familiar.
Input: an image+text dataset and a VLM base
model already picked from the model catalog.
Output format: a validated adapter config —
which components are frozen, LoRA target modules,
and a min_pixels/max_pixels budget — that
llm-finetuning-training-engineer consumes
directly when it generates a runnable script.
Quick Reference
| Situation | Default |
|---|---|
| Adapting behavior on familiar images | Frozen tower+projector, LoRA r=8–16, α=16–32 |
| Visual domain shift | Unfreeze last-6 ViT layers, vision LR 5–10x lower |
| Doesn't fit in bf16 at target rank | QLoRA — frozen vision tower only |
fast_inference=True |
finetune_vision_layers=False |
| Loss normal, eval not improving | Check the Two Silent Killers below first |
The Consensus Recipe
Freeze the vision tower and the projector. Put
LoRA on the LLM only, all-linear (the same
attention + MLP target list as text-only SFT —
see lora-qlora-recipes), at r=8–16,
α=16–32. This is the settled default for
adapting a VLM's behavior without disturbing how
it sees.
- The vision tower and projector stay frozen by default. They already encode a general visual representation; retraining them is rarely necessary and adds risk without adding capability for most tasks.
- LoRA rank runs lower than the text-only general default (r=8–16 here vs r=16–32 for text-only SFT) because the LLM-only adapter is adapting behavior, not injecting new visual knowledge.
- QLoRA is permitted only with a frozen vision tower. Quantizing the base while also unfreezing and training vision layers is unsupported and unstable — treat this as a hard pairing rule, not a tunable. If the vision tower needs to unfreeze, drop QLoRA and use bf16 LoRA instead.
# freeze tower + projector; LoRA on LLM only
for name, param in model.named_parameters():
if "vision_tower" in name or "projector" in name:
param.requires_grad = False
target_modules = [
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
] # LLM-only, all-linear — r=8-16, alpha=16-32
When to Unfreeze
Unfreezing vision layers is a deliberate escalation, not a default decision — reach for it only when the domain shift is visual, not textual.
- Unfreeze only for visual domain shift. If the task is teaching new behavior on images the tower already understands (charts, everyday photos), the frozen-tower recipe above is sufficient. Unfreeze when the visual domain itself is unfamiliar to the tower — satellite imagery, medical scans, dense technical diagrams — and the frozen-tower recipe plateaus.
- Last-6 ViT layers is the sweet spot. Unfreezing the final six vision-transformer layers (not the whole tower) measured +1.7pt DocVQA at ~1.75x training cost over the frozen baseline. Treat six layers as the ceiling worth paying for; going further spends compute without a matched result.
- Vision LR must run 5–10x lower than the LLM LR when unfrozen. The vision tower's pretrained representation is more fragile than the LLM's adapter; the same LR for both risks overwriting the visual representation faster than the LLM adapter can compensate.
- High LoRA rank on the patch- embedding layer risks NaN. If patch embedding is in the unfrozen set, keep its rank low and watch early-step loss closely — one of the most fragile places to apply LoRA in a VLM.
The Two Silent Killers
Both produce a run that trains without error and without learning: the loss curve looks normal, the model doesn't improve, and neither throws an exception — both need an explicit pre-training check, not just a clean training log.
- Image-tag/count mismatch. Every image
placeholder token in the templated text must
map 1:1 to a media item actually passed to the
collator. A mismatch (one placeholder, zero or
two images attached; or an image with no
placeholder) doesn't error in most collators —
it silently misaligns image and text, and the
model "trains but learns nothing." Validate the
1:1 placeholder-to-media mapping before training
starts, on every example, not just a sample.
Full validation-checklist detail:
references/collators-and-pitfalls.md. min_pixels/max_pixelsresolution budget. This pair is the single most consequential hyperparameter for quality and memory in VLM SFT — more than rank, alpha, or LR. Too low silently downsamples images below what the task needs (small document text becomes unreadable even though training "succeeds"); too high blows the activation memory budget or forces too small a batch to train stably. Set it deliberately per dataset, don't leave it at a framework default.
Unsloth Specifics
UnslothVisionDataCollatoris the collator Unsloth expects for VLM SFT — it handles the image-tag alignment and per-architecture processor contract described inreferences/collators-and-pitfalls.md. Don't substitute a text-only collator for VLM data.finetune_vision_layers=Falseis required whenfast_inference=True. vLLM cannot serve LoRA adapters on vision layers, so a fast- inference setup that also unfreezes vision layers fails at serve time even if training succeeds. If the recipe calls for unfreezing the last-6 ViT layers (see When to Unfreeze above), fast inference is off the table for that run — choose one or the other, not both.
Model Choice
Base VLM choice is out of scope for this skill —
it lives in one place, the model catalog at
finetuning-method-selection's
references/model-catalog.md. This skill and its
references describe recipes by architecture
family only, never by recommending one model over
another.
VLM reinforcement learning (VLM-GRPO) is
reference-only in this plugin — the fragmented
tooling and reward-hacking failure modes specific
to VLM-RL are covered in grpo-rlvr-training,
not here. This skill's scope stops at supervised
fine-tuning.
Failure Modes
The recurring mistake across every section above
is treating a clean loss curve as proof the run
is healthy. A normal-looking curve is consistent
with both a working run and either silent
killer, since the model trains on something
either way — just not the aligned image-text
signal when a killer is present. A flat eval score
next to a normal loss curve means re-run the
checklist in references/collators-and-pitfalls.md
before touching any hyperparameter.
References
references/collators-and-pitfalls.md— per- architecture collator table, dataset-format examples with image placeholders, a pre- training validation checklist, and the two- stage projector-alignment recipe as an advanced pattern.
Related skills: finetuning-method-selection
routes here; lora-qlora-recipes covers the
text-only LoRA fundamentals this skill
specializes; grpo-rlvr-training covers VLM-RL
(reference-only); dataset-curation covers
image+text dataset preparation this skill doesn't.
Files (agents)
-
references
-
collators-and-pitfalls.md 5.8 KB
Last verified: 2026-07-13 # VLM Collators, Dataset Format, and Pitfalls Full detail backing the summary in `SKILL.md`. Base models are never named here as recommendations — the collator table below names architecture families only because the processor contract (which tensors a collator must produce) is a technical property of that family, not a model choice. For which actual model to fine-tune at a given size class, see `finetuning-method-selection`'s `references/model-catalog.md`. ## Per-Architecture Collator Table Collators are **not interchangeable** across VLM architecture families — each family's processor expects a different tensor contract, and using the wrong collator produces either a hard error or (worse) silently wrong tensors that train without learning. Each row below describes an architecture family's processor contract, not a model recommendation. | Architecture family | Tensor contract | Notes | |---|---|---| | Qwen-VL family | `pixel_values` + `image_grid_thw` | The grid tensor encodes the patch layout per image; a collator that drops it or mismatches its shape against `pixel_values` silently corrupts the vision-token layout. | | InternVL family | Variable-length pixel-value lists | Images can each contribute a different number of tiles/patches; the collator must pad or batch these variable-length lists per example rather than assuming a fixed tensor shape. | | Gemma 3 family | `token_type_ids` for loss masking | Loss masking between image and text spans is driven by `token_type_ids`, not just the usual assistant-turn attention mask — a collator built for a different family's masking convention silently masks the wrong spans. | Two practical consequences: - Picking a collator is an architecture-family decision, made once per base model, not a free parameter to tune. - A collator built for one family will often *run* against another family's data without erroring — the shapes are superficially compatible — which is exactly how a mismatched collator becomes a silent-failure run instead of a crash. ## Dataset Format: Messages with Image Placeholders VLM SFT datasets are typically a messages list per example, with an explicit image placeholder token in the content that the processor later expands to the architecture's actual vision-token span: ```python example = { "messages": [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "What does this chart show?"}, ], }, { "role": "assistant", "content": [ {"type": "text", "text": "Quarterly revenue trending upward."}, ], }, ], "images": [<PIL.Image or path>], } ``` The count of `{"type": "image"}` placeholder entries in `messages` must equal the count of entries in `images`, in order, for every single example — this 1:1 mapping is exactly the first silent killer from `SKILL.md`. A dataset-level `assert` on this count, run over every example before training starts, catches the mismatch at data-prep time instead of after a wasted training run. ## Pre-Training Validation Checklist Run this checklist against one collated batch before launching a full training run. All three checks are cheap (seconds, one batch) relative to the cost of discovering a silent failure after hours of training: 1. **Decode one collated batch back to text.** Pull a batch from the dataloader, decode the `input_ids` with the tokenizer, and read it. Confirm the image placeholder tokens appear where expected and the surrounding text matches the source example — this catches template or collator bugs that reshuffle content. 2. **Count image tokens per example.** Compare the number of vision tokens the processor actually inserted against the expected count for that image's resolution under the configured `min_pixels`/`max_pixels` budget (the second silent killer from `SKILL.md`). A count that doesn't match the expected budget means the resolution budget isn't being applied the way it's configured. 3. **Verify the loss mask covers assistant turns only.** Inspect the labels tensor (or `token_type_ids` for Gemma-3-family collators) and confirm masked (`-100`) positions cover the system/user turns and image tokens, with only assistant-turn text contributing to the loss. A loss mask that leaks onto image tokens or user turns trains the model to predict input it should only be conditioning on. If any of the three checks fails, fix the collator or dataset before starting the full run — none of these are the kind of thing a training curve reveals on its own. ## Advanced Pattern: Two-Stage Projector-Alignment Recipe The consensus recipe in `SKILL.md` freezes the projector. When adapting to a base model or dataset far enough from the projector's original alignment that the frozen-projector recipe underperforms, a two-stage LLaVA-style alignment recipe is the advanced fallback: 1. **Stage 1 — projector-only alignment.** Freeze both the vision tower and the LLM. Train only the projector (no LoRA involved yet) on a broad, simple image-caption-style dataset. The goal is purely to re-align the projector's output space with the current LLM's embedding space — this stage does not teach the target task. 2. **Stage 2 — task LoRA on top.** With the realigned projector now frozen again, apply the standard consensus recipe from `SKILL.md` (LoRA on the LLM only, all-linear, r=8–16, α=16–32) using the actual task dataset. This two-stage recipe is an escalation path, not a default — reach for it only when the single-stage frozen-projector recipe measurably underperforms, since it roughly doubles the number of training runs required. Most VLM SFT tasks in this plugin's scope stay on the single-stage consensus recipe.
-
-
SKILL.md 7.7 KB
--- name: vision-sft description: Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or debugging a VLM fine-tune that trains without learning. --- # Vision-Language SFT This skill assumes `finetuning-method-selection` already routed here: the data shape is image+text demonstrations, not preference pairs or a verifiable reward signal, and the base is a vision-language model rather than a text-only one. `lora-qlora-recipes` covers the text-only LoRA/QLoRA recipe this skill specializes for the vision tower and projector; read that skill first if the LoRA fundamentals (rank, alpha, target modules) aren't already familiar. **Input:** an image+text dataset and a VLM base model already picked from the model catalog. **Output format:** a validated adapter config — which components are frozen, LoRA target modules, and a `min_pixels`/`max_pixels` budget — that `llm-finetuning-training-engineer` consumes directly when it generates a runnable script. ## Quick Reference | Situation | Default | |---|---| | Adapting behavior on familiar images | Frozen tower+projector, LoRA r=8–16, α=16–32 | | Visual domain shift | Unfreeze last-6 ViT layers, vision LR 5–10x lower | | Doesn't fit in bf16 at target rank | QLoRA — frozen vision tower only | | `fast_inference=True` | `finetune_vision_layers=False` | | Loss normal, eval not improving | Check the Two Silent Killers below first | ## The Consensus Recipe Freeze the vision tower and the projector. Put LoRA on the LLM only, all-linear (the same attention + MLP target list as text-only SFT — see `lora-qlora-recipes`), at **r=8–16, α=16–32**. This is the settled default for adapting a VLM's behavior without disturbing how it sees. - **The vision tower and projector stay frozen by default.** They already encode a general visual representation; retraining them is rarely necessary and adds risk without adding capability for most tasks. - **LoRA rank runs lower than the text-only general default** (r=8–16 here vs r=16–32 for text-only SFT) because the LLM-only adapter is adapting behavior, not injecting new visual knowledge. - **QLoRA is permitted only with a frozen vision tower.** Quantizing the base while also unfreezing and training vision layers is unsupported and unstable — treat this as a hard pairing rule, not a tunable. If the vision tower needs to unfreeze, drop QLoRA and use bf16 LoRA instead. ```python # freeze tower + projector; LoRA on LLM only for name, param in model.named_parameters(): if "vision_tower" in name or "projector" in name: param.requires_grad = False target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ] # LLM-only, all-linear — r=8-16, alpha=16-32 ``` ## When to Unfreeze Unfreezing vision layers is a deliberate escalation, not a default decision — reach for it only when the domain shift is visual, not textual. - **Unfreeze only for visual domain shift.** If the task is teaching new behavior on images the tower already understands (charts, everyday photos), the frozen-tower recipe above is sufficient. Unfreeze when the visual domain itself is unfamiliar to the tower — satellite imagery, medical scans, dense technical diagrams — and the frozen-tower recipe plateaus. - **Last-6 ViT layers is the sweet spot.** Unfreezing the final six vision-transformer layers (not the whole tower) measured **+1.7pt DocVQA at ~1.75x training cost** over the frozen baseline. Treat six layers as the ceiling worth paying for; going further spends compute without a matched result. - **Vision LR must run 5–10x lower than the LLM LR when unfrozen.** The vision tower's pretrained representation is more fragile than the LLM's adapter; the same LR for both risks overwriting the visual representation faster than the LLM adapter can compensate. - **High LoRA rank on the patch- embedding layer risks NaN.** If patch embedding is in the unfrozen set, keep its rank low and watch early-step loss closely — one of the most fragile places to apply LoRA in a VLM. ## The Two Silent Killers Both produce a run that trains without error and without learning: the loss curve looks normal, the model doesn't improve, and neither throws an exception — both need an explicit pre-training check, not just a clean training log. - **Image-tag/count mismatch.** Every image placeholder token in the templated text must map 1:1 to a media item actually passed to the collator. A mismatch (one placeholder, zero or two images attached; or an image with no placeholder) doesn't error in most collators — it silently misaligns image and text, and the model "trains but learns nothing." Validate the 1:1 placeholder-to-media mapping before training starts, on every example, not just a sample. Full validation-checklist detail: `references/collators-and-pitfalls.md`. - **`min_pixels`/`max_pixels` resolution budget.** This pair is the single most consequential hyperparameter for quality and memory in VLM SFT — more than rank, alpha, or LR. Too low silently downsamples images below what the task needs (small document text becomes unreadable even though training "succeeds"); too high blows the activation memory budget or forces too small a batch to train stably. Set it deliberately per dataset, don't leave it at a framework default. ## Unsloth Specifics - **`UnslothVisionDataCollator`** is the collator Unsloth expects for VLM SFT — it handles the image-tag alignment and per-architecture processor contract described in `references/collators-and-pitfalls.md`. Don't substitute a text-only collator for VLM data. - **`finetune_vision_layers=False` is required when `fast_inference=True`.** vLLM cannot serve LoRA adapters on vision layers, so a fast- inference setup that also unfreezes vision layers fails at serve time even if training succeeds. If the recipe calls for unfreezing the last-6 ViT layers (see When to Unfreeze above), fast inference is off the table for that run — choose one or the other, not both. ## Model Choice Base VLM choice is out of scope for this skill — it lives in one place, the model catalog at `finetuning-method-selection`'s `references/model-catalog.md`. This skill and its references describe recipes by architecture family only, never by recommending one model over another. VLM reinforcement learning (VLM-GRPO) is reference-only in this plugin — the fragmented tooling and reward-hacking failure modes specific to VLM-RL are covered in `grpo-rlvr-training`, not here. This skill's scope stops at supervised fine-tuning. ## Failure Modes The recurring mistake across every section above is treating a clean loss curve as proof the run is healthy. A normal-looking curve is consistent with **both** a working run **and** either silent killer, since the model trains on *something* either way — just not the aligned image-text signal when a killer is present. A flat eval score next to a normal loss curve means re-run the checklist in `references/collators-and-pitfalls.md` before touching any hyperparameter. ## References - `references/collators-and-pitfalls.md` — per- architecture collator table, dataset-format examples with image placeholders, a pre- training validation checklist, and the two- stage projector-alignment recipe as an advanced pattern. Related skills: `finetuning-method-selection` routes here; `lora-qlora-recipes` covers the text-only LoRA fundamentals this skill specializes; `grpo-rlvr-training` covers VLM-RL (reference-only); `dataset-curation` covers image+text dataset preparation this skill doesn't.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.