{
  "skill": "alterlab-transformers",
  "evals": [
    {
      "id": "pipeline-quick-inference",
      "prompt": "I have a list of customer review strings and just want sentiment labels quickly without setting up any training. What's the fastest way to do this with Hugging Face?",
      "expected_output": "Invokes alterlab-transformers and uses the Pipeline API for quick inference: pipeline('text-classification') (or sentiment-analysis) applied over the list of reviews, with no manual model/tokenizer wiring. Notes that pipelines are the fast path for simple inference and can take a model='model-id' argument to pin a specific checkpoint.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "pipeline" },
        { "type": "behavior", "value": "Uses the transformers pipeline() API for quick text-classification inference without manual configuration or training." }
      ]
    },
    {
      "id": "fine-tune-trainer",
      "prompt": "I want to fine-tune a pre-trained BERT model on my own labeled text dataset for a domain-specific classification task. How do I set up the training loop?",
      "expected_output": "Invokes alterlab-transformers for fine-tuning with the Trainer API: load the model with AutoModelForSequenceClassification.from_pretrained, tokenize the dataset, define TrainingArguments (output_dir, num_train_epochs, per_device_train_batch_size, eval_strategy), construct a Trainer with model/args/train_dataset and processing_class=tokenizer (the v5 name for the old tokenizer= argument), and call trainer.train(). May mention the datasets, evaluate, and accelerate packages, mixed precision, and that report_to defaults to 'none' in v5.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "Trainer" },
        { "type": "behavior", "value": "Sets up fine-tuning via TrainingArguments + Trainer on a custom dataset and calls trainer.train()." }
      ]
    },
    {
      "id": "generation-decoding-strategy",
      "prompt": "I'm loading a causal language model with AutoModelForCausalLM and want to generate longer, more diverse completions. How do I control the decoding with temperature, top-k and top-p instead of greedy output?",
      "expected_output": "Invokes alterlab-transformers for text generation: load with AutoModelForCausalLM.from_pretrained and AutoTokenizer, tokenize with return_tensors='pt', and call model.generate(**inputs, max_new_tokens=..., do_sample=True, temperature=..., top_k=..., top_p=...) to control sampling vs greedy/beam decoding, then tokenizer.decode the output. Explains the decoding-strategy parameters.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "generate" },
        { "type": "behavior", "value": "Uses AutoModelForCausalLM + model.generate with sampling parameters (temperature/top-k/top-p) to control decoding strategy." }
      ]
    },
    {
      "id": "image-classification-vision",
      "prompt": "I have a folder of product photos and want to run a pre-trained vision transformer to classify what each image contains. Which Hugging Face setup do I need?",
      "expected_output": "Invokes alterlab-transformers for a computer-vision task: uses pipeline('image-classification') with a pinned checkpoint such as google/vit-base-patch16-224 (or loads AutoModelForImageClassification with its AutoImageProcessor) on the photos, and notes the vision dependencies (pillow, plus torchvision for the default v5 image-processor backend; timm only for timm-backed models). Confirms transformers supports CV tasks beyond NLP, including image classification and object detection.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "image-classification" },
        { "type": "behavior", "value": "Handles the vision task with an image-classification pipeline/model and notes the pillow/torchvision vision dependencies." }
      ]
    },
    {
      "id": "v5-migration-dtype-removed-pipelines",
      "prompt": "After upgrading to transformers 5 my old analysis script warns that torch_dtype is deprecated and pipeline('summarization', model='facebook/bart-large-cnn') no longer exists. How should I rewrite the model loading and the summarization step?",
      "expected_output": "Invokes alterlab-transformers for a v4-to-v5 migration: replaces torch_dtype= with dtype= (noting the v5 default is dtype='auto', the checkpoint dtype), explains that the summarization/translation/question-answering pipelines were removed in v5, and rewrites the step as pipeline('text-generation') with a small open chat model (e.g. Qwen/Qwen3-0.6B) prompted with an instruction and chat messages, reading the reply from generated_text[-1]['content'] and using max_new_tokens rather than max_length. May mention that TensorFlow/Flax support was removed.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "dtype" },
        { "type": "behavior", "value": "Replaces torch_dtype with dtype and replaces the removed summarization pipeline with a text-generation pipeline driven by a chat model and max_new_tokens." }
      ]
    },
    {
      "id": "near-miss-scikit-learn",
      "prompt": "I have a tabular CSV of numeric and categorical features and want to train a RandomForest classifier with a preprocessing pipeline and cross-validated hyperparameter tuning. How do I build that?",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-scikit-learn. The task is classical machine learning on tabular features (RandomForest, ColumnTransformer preprocessing, GridSearchCV), which has nothing to do with pre-trained transformer models. The response should point to alterlab-scikit-learn for the tabular ML pipeline.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-scikit-learn" }
      ]
    },
    {
      "id": "near-miss-pytorch-lightning",
      "prompt": "I'm writing a custom convolutional neural network from scratch and want to structure the training loop, multi-GPU strategy, and checkpointing cleanly with a LightningModule and Trainer. How should I organize it?",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-pytorch-lightning. The user is building a custom-architecture CNN and wants Lightning's LightningModule/Trainer training-loop organization, not loading or fine-tuning a pre-trained Hugging Face transformer via the transformers Trainer. The response should point to alterlab-pytorch-lightning.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-pytorch-lightning" }
      ]
    },
    {
      "id": "near-miss-text-as-data",
      "prompt": "For my political-science paper I want to discover the main themes in 20,000 parliamentary speeches with BERTopic and then show reviewers the topics are valid and stable across seeds. How should I design and validate that analysis?",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-text-as-data. The request is a social-science text-as-data measurement design (topic discovery with BERTopic, topic validity and stability checks for a paper), not loading, running, or fine-tuning a Hugging Face transformers checkpoint. The response should point to alterlab-text-as-data.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-text-as-data" }
      ]
    }
  ]
}

# Text Generation (transformers v5)

## Overview

Generate text with `model.generate()`. Decoding strategy and parameters (length, temperature, top-k/top-p, beams, repetition controls) shape output quality and diversity. Examples use `Qwen/Qwen3-0.6B`, a small open chat model that runs on CPU; swap in any `AutoModelForCausalLM` checkpoint.

## Basic Generation

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)

# Decode only the newly generated tokens
text = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(text)
```

Pass `**inputs` (not just `input_ids`) so the attention mask reaches `generate()`.

## Generation Strategies

### Greedy Decoding

Highest-probability token at each step (deterministic):

```python
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
```

**Use for**: extraction, short factual answers, reproducible runs. Note that many chat checkpoints ship a `generation_config.json` that turns sampling on by default, so pass `do_sample=False` explicitly when you need determinism. Some reasoning models (e.g. Qwen3 in thinking mode) degrade or loop under greedy decoding — follow the model card's recommended settings.

### Sampling

```python
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
)
```

**Use for**: open-ended or diverse outputs. `min_p` (e.g. `min_p=0.05`) is an alternative truncation rule.

### Beam Search

```python
outputs = model.generate(**inputs, max_new_tokens=50, num_beams=5, early_stopping=True)
```

**Use for**: translation/summarization with encoder-decoder models, where a high-likelihood output matters.

### Strategies Moved to the Hub in v5

Contrastive search (`penalty_alpha`), constrained beam search (`force_words_ids`, `constraints`), group/diverse beam search (`num_beam_groups`), and DoLa now live in Hub `custom_generate` repositories. Calling them the v4 way raises an error unless you opt in to running the repository's code:

```python
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    penalty_alpha=0.6,
    top_k=4,
    custom_generate="transformers-community/contrastive-search",
    trust_remote_code=True,  # executes custom_generate/generate.py from that repo — read it first
)
```

Other repos: `transformers-community/constrained-beam-search`, `transformers-community/group-beam-search`, `transformers-community/dola`.

## Key Parameters

### Length Control

- `max_new_tokens`: maximum tokens to generate — prefer this.
- `max_length`: maximum total length (prompt + output); easy to exhaust with long prompts.
- `min_new_tokens`: force at least N new tokens.

### Temperature

Only applies with `do_sample=True`:

```python
temperature=1.0   # model distribution unchanged
temperature=0.7   # more focused
temperature=1.3   # more random
```

### Top-K / Top-P

```python
do_sample=True
top_k=50     # sample from the 50 most likely tokens
top_p=0.95   # sample from the smallest set with ≥95% cumulative probability
```

### Repetition Controls

```python
repetition_penalty=1.2   # 1.0 = none; >1 discourages repeats
no_repeat_ngram_size=3   # forbid repeating any 3-gram
```

### Output Control

```python
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    num_return_sequences=3,  # three samples per prompt
)
```

- `pad_token_id=tokenizer.eos_token_id` — silences the warning for models without a pad token.
- `eos_token_id` — stop on a specific token (or a list of ids); `stop_strings=["\n\n"]` (with `tokenizer=tokenizer`) stops on text.

## Advanced Features

### Batch Generation

Decoder-only models must be **left-padded** for batched generation; right padding corrupts the continuation.

```python
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

prompts = ["Hello, my name is", "Once upon a time"]
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)

texts = tokenizer.decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)  # v5: decode handles batches
for i, text in enumerate(texts):
    print(f"Prompt {i}: {text}\n")
```

### Streaming Generation

```python
from threading import Thread
from transformers import TextIteratorStreamer

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
generation_kwargs = dict(**inputs, streamer=streamer, max_new_tokens=100)

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()
for text in streamer:
    print(text, end="", flush=True)
thread.join()
```

For simple console output, `TextStreamer(tokenizer, skip_prompt=True)` passed as `streamer=` needs no thread.

### Blocking Tokens

```python
bad_words_ids = tokenizer(["offensive", "inappropriate"], add_special_tokens=False).input_ids
outputs = model.generate(**inputs, max_new_tokens=50, bad_words_ids=bad_words_ids)
```

To *force* words into the output, use the `transformers-community/constrained-beam-search` repo shown above.

### Generation Config

```python
from transformers import GenerationConfig

generation_config = GenerationConfig(
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
)
generation_config.save_pretrained("./my_generation_config")

generation_config = GenerationConfig.from_pretrained("./my_generation_config")
outputs = model.generate(**inputs, generation_config=generation_config)
```

The model's defaults live in `model.generation_config` (in v5 they are no longer read from `model.config`). Inspect it to see what a checkpoint does by default.

## Model-Specific Generation

### Chat Models

In v5 `apply_chat_template` returns a `BatchEncoding` (`input_ids` + `attention_mask`) by default, so unpack it into `generate()`:

```python
messages = [
    {"role": "system", "content": "You are a concise research assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,  # append the assistant header so the model answers
    return_tensors="pt",
    enable_thinking=False,       # Qwen3 template variable; omit for other models
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
```

Use `tokenize=False` to inspect the rendered prompt string. Extra keyword arguments (like `enable_thinking`) are passed to the model's chat template, so they are model-specific.

### Encoder-Decoder Models

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")
tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")

# T5 uses task prefixes
inputs = tokenizer("translate English to French: Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, num_beams=4)
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
```

## Optimization

### KV Cache

Caching is on by default (`use_cache=True`); in v5 the default cache class is chosen by the model.

### Static Cache

A fixed-size cache enables `torch.compile` speedups for repeated generation:

```python
outputs = model.generate(**inputs, max_new_tokens=100, cache_implementation="static")
```

Or build one explicitly: `StaticCache(config=model.config, max_cache_len=1024)` passed as `past_key_values=`.

### Attention Implementation

SDPA is the default; Flash Attention 2 can be faster on supported CUDA GPUs:

```python
model = AutoModelForCausalLM.from_pretrained(
    "org/model-id", attn_implementation="flash_attention_2", dtype=torch.bfloat16
)
```

### Assisted (Speculative) Decoding

A small draft model sharing the tokenizer can speed up a large one:

```python
assistant = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype="auto", device_map="auto")
outputs = model.generate(**inputs, assistant_model=assistant, max_new_tokens=100)
```

## Generation Recipes

### Creative Writing

```python
outputs = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.2,
)
```

### Factual / Reproducible Generation

```python
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False, repetition_penalty=1.1)
```

### Several Diverse Candidates

```python
outputs = model.generate(
    **inputs, max_new_tokens=100, do_sample=True,
    temperature=1.0, top_p=0.95, num_return_sequences=5,
)
```

### Translation / Summarization (encoder-decoder)

```python
outputs = model.generate(
    **inputs, max_new_tokens=100, num_beams=5, early_stopping=True, no_repeat_ngram_size=3,
)
```

## Common Issues

**Repetitive output:** raise `repetition_penalty` (1.1–1.3), set `no_repeat_ngram_size` (2–3), or switch from greedy to sampling.

**Poor quality:** use a larger or instruction-tuned checkpoint, apply its chat template, and follow the model card's recommended decoding settings.

**Too deterministic:** set `do_sample=True` and raise `temperature` (0.7–1.0).

**Garbled batched outputs:** left-pad (`padding_side="left"`) and pass the attention mask.

**Slow generation:** use a GPU with `dtype=torch.bfloat16`, a static cache, Flash Attention, assisted decoding, or fewer `max_new_tokens`.

## Best Practices

1. **Start from the checkpoint's `generation_config`**, then tune.
2. **Greedy/beam for extraction and scoring, sampling for open-ended text.**
3. **Always set `max_new_tokens`.**
4. **Seed sampled runs** with `transformers.set_seed(42)` and report decoding parameters with results.
5. **Validate generated text** against a labelled sample before using an LLM as a research instrument (coding, classification, summarization).
6. **Monitor memory**: beams and `num_return_sequences` multiply memory use.

# Model Loading and Management (transformers v5)

## Overview

`from_pretrained` detects the architecture from the checkpoint's config, downloads weights (safetensors) from the Hub or reads them from disk, and places them on devices according to `device_map`. v5 is PyTorch-only.

## Loading Models

### AutoModel Classes

```python
from transformers import (AutoModel, AutoModelForCausalLM, AutoModelForMaskedLM,
                          AutoModelForSeq2SeqLM, AutoModelForSequenceClassification)

# Base model (no task head) — hidden states / embeddings
model = AutoModel.from_pretrained("google-bert/bert-base-uncased")

# Sequence classification (the head is newly initialized unless the checkpoint has one)
model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased", num_labels=2)

# Causal language modeling (decoder-only LLMs)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")

# Masked language modeling (BERT-style)
model = AutoModelForMaskedLM.from_pretrained("answerdotai/ModernBERT-base")

# Sequence-to-sequence (T5/BART-style encoder-decoders)
model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")
```

### Common AutoModel Classes

**NLP:**
- `AutoModelForSequenceClassification`: text classification, sentiment
- `AutoModelForTokenClassification`: NER, POS tagging
- `AutoModelForQuestionAnswering`: extractive QA heads (the QA *pipeline* was removed in v5, the model class was not)
- `AutoModelForCausalLM`: text generation (Qwen, Llama, Gemma, SmolLM, …)
- `AutoModelForMaskedLM`: masked language modeling (BERT, ModernBERT)
- `AutoModelForSeq2SeqLM`: encoder-decoder translation/summarization (T5, BART)

**Vision:**
- `AutoModelForImageClassification`, `AutoModelForObjectDetection`, `AutoModelForImageSegmentation`

**Audio:**
- `AutoModelForAudioClassification`, `AutoModelForSpeechSeq2Seq` (Whisper), `AutoModelForCTC`

**Multimodal:**
- `AutoModelForImageTextToText`: vision-language chat models (replaces the removed `AutoModelForVision2Seq`)
- `AutoProcessor`: loads the matching tokenizer + image/audio processor bundle

## Loading Parameters

### Basic Parameters

**pretrained_model_name_or_path**: Hub ID or local directory
```python
model = AutoModel.from_pretrained("google-bert/bert-base-uncased")  # from the Hub
model = AutoModel.from_pretrained("./local/model/path")             # from disk
```

**revision**: pin a branch, tag, or commit for reproducibility
```python
model = AutoModel.from_pretrained("org/model-id", revision="a1b2c3d")
```

**num_labels**: output size of a new classification head
```python
model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-uncased", num_labels=3)
```

**cache_dir**: custom cache location for this call (globally, set `HF_HOME` or `HF_HUB_CACHE`)
```python
model = AutoModel.from_pretrained("org/model-id", cache_dir="./my_cache")
```

**token**: Hub token for gated/private repos (the old `use_auth_token=` was removed)

### Device Management

**device_map**: automatic placement for large models (requires `accelerate`)
```python
# Spread across available GPUs, then CPU
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", device_map="auto")

# Fill devices in order
model = AutoModelForCausalLM.from_pretrained("org/model-id", device_map="sequential")

# Custom map (module names depend on the architecture — inspect print(model) first)
device_map = {
    "model.embed_tokens": 0,
    "model.layers.0": 0,
    "model.layers.1": 1,
    "lm_head": "cpu",
}
model = AutoModelForCausalLM.from_pretrained("org/model-id", device_map=device_map)
```

Manual placement:
```python
import torch
model = AutoModel.from_pretrained("org/model-id")
model.to("cuda" if torch.cuda.is_available() else "cpu")
```

### Precision Control

**dtype** (the old `torch_dtype=` keyword still works but warns). In v5 the default is `dtype="auto"`, i.e. the dtype stored in the checkpoint (bf16 for most recent LLMs) rather than float32.
```python
import torch

model = AutoModel.from_pretrained("org/model-id", dtype=torch.bfloat16)  # better range than fp16
model = AutoModel.from_pretrained("org/model-id", dtype=torch.float16)
model = AutoModel.from_pretrained("org/model-id", dtype=torch.float32)  # force full precision (e.g. CPU numerics)
```

### Attention Implementation

**attn_implementation**: PyTorch SDPA is used by default where the architecture supports it, falling back to eager.
```python
# Flash Attention 2 (CUDA; requires the flash-attn package and fp16/bf16 weights)
model = AutoModel.from_pretrained("org/model-id", attn_implementation="flash_attention_2", dtype=torch.bfloat16)

# Eager — required to return attention weights (SDPA does not support output_attentions=True)
model = AutoModel.from_pretrained("org/model-id", attn_implementation="eager")
```

### Memory Optimization

Low-memory loading is the only loading path in v5, so the old `low_cpu_mem_usage=True` flag is silently ignored.

**Quantization** (bitsandbytes, CUDA): pass a `BitsAndBytesConfig` via `quantization_config`. The direct `load_in_8bit=` / `load_in_4bit=` keyword arguments were removed in v5.

8-bit:
```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    "org/model-id",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map="auto",
)
```

4-bit (NF4 + double quantization):
```python
import torch
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "org/model-id",
    quantization_config=quantization_config,
    device_map="auto",
)
```

Pre-quantized checkpoints (GPTQ, AWQ, FP8, …) load with plain `from_pretrained`; their config carries the quantization settings.

## Model Configuration

### Loading with Custom Config

```python
from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("google-bert/bert-base-uncased")
config.hidden_dropout_prob = 0.2
config.attention_probs_dropout_prob = 0.2

model = AutoModel.from_pretrained("google-bert/bert-base-uncased", config=config)
```

Or override attributes inline: `AutoModel.from_pretrained("org/model-id", hidden_dropout_prob=0.2)`.

v5 config notes: RoPE settings live in `config.rope_parameters` (not `config.rope_theta`); vision-language configs are nested (`config.text_config.vocab_size`); generation settings live in `model.generation_config`, not the model config.

### Initializing from Config Only

```python
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("Qwen/Qwen3-0.6B")
model = AutoModelForCausalLM.from_config(config)  # random weights — for pre-training from scratch
```

## Model Modes

`from_pretrained` returns the model in **evaluation mode** (`model.training == False`, dropout off). Switch explicitly when writing your own loop:

```python
model = AutoModel.from_pretrained("org/model-id")
print(model.training)  # False

model.train()  # enable dropout for fine-tuning in a custom loop
model.eval()   # back to deterministic inference
```

`Trainer` toggles modes for you. For inference, also wrap forward passes in `torch.inference_mode()` (or `torch.no_grad()`) to skip gradient tracking. Models built with `from_config` start in training mode, like any fresh `nn.Module`.

## Saving Models

### Save Locally

```python
model.save_pretrained("./my_model")
tokenizer.save_pretrained("./my_model")  # keep tokenizer/processor with the weights
```

This writes `config.json`, `generation_config.json` (generative models), and `model.safetensors` (sharded above the 50 GB default `max_shard_size`). v5 always saves safetensors; the `safe_serialization` argument was removed.

### Save to the Hugging Face Hub

```python
model.push_to_hub("username/model-name")
model.push_to_hub("username/model-name", commit_message="Update model", private=True)
```

In v5, `push_to_hub` arguments other than `repo_id` are keyword-only.

## Model Inspection

```python
total_params = model.num_parameters()
trainable_params = model.num_parameters(only_trainable=True)
print(f"Total: {total_params:,}  Trainable: {trainable_params:,}")

memory_mb = model.get_memory_footprint() / 1024**2
print(f"Memory: {memory_mb:.2f} MB")

print(model)          # module tree
print(model.config)   # architecture hyperparameters
```

## Forward Pass

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased-finetuned-sst-2-english")
model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased-finetuned-sst-2-english")

inputs = tokenizer("Sample text", return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits
predictions = logits.argmax(dim=-1)
labels = [model.config.id2label[i] for i in predictions.tolist()]
```

## Export

### ONNX

- **Optimum**: `optimum-onnx` (0.1.0, the current release as of 2026-09) still pins `transformers<4.58`, so it cannot share an environment with transformers v5. Use a separate v4 environment if you need `ORTModelFor…` classes.
- **In-library (v5)**: recent v5 releases ship `transformers.exporters` with an `OnnxExporter` built on `torch.export` + `torch.onnx` (needs `onnx` and `onnxscript`):

```python
from transformers.exporters.exporter_onnx import OnnxConfig, OnnxExporter

inputs = tokenizer("Sample text", return_tensors="pt")
exporter = OnnxExporter()
exporter.export(model, inputs, config=OnnxConfig(output_path="model.onnx"))
```

This module is new; check the installed version's docstrings before relying on it in a pipeline. `torch.onnx.export(..., dynamo=True)` is the lower-level fallback.

## Best Practices

1. **Use Auto classes** for architecture detection.
2. **Pin `revision=`** for any result you will publish.
3. **Set `dtype` deliberately**: bf16/fp16 on GPU; float32 when you need CPU numerical parity.
4. **Use `device_map="auto"`** for models larger than one device.
5. **Consider quantization** for memory-constrained inference (validate accuracy afterwards).
6. **Keep tokenizer/processor with the weights** when saving.
7. **Cache location**: set `HF_HOME` (the `TRANSFORMERS_CACHE` variable was removed in v5).

## Common Issues

**CUDA out of memory:**
```python
import torch
from transformers import AutoModel, BitsAndBytesConfig

model = AutoModel.from_pretrained("org/model-id", dtype=torch.bfloat16)          # lower precision
model = AutoModel.from_pretrained(                                               # or quantize
    "org/model-id",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map="auto",
)
model = AutoModel.from_pretrained("org/model-id", device_map="cpu")              # or stay on CPU
```

**Unexpected dtype after upgrading to v5:** the default changed from float32 to `"auto"`; pass `dtype=torch.float32` to restore v4 behaviour.

**Model not found / 401:**
```python
# Verify the model ID on https://huggingface.co and accept the license on gated repos
from huggingface_hub import login
login()
```

# Pipeline API Reference (transformers v5)

## Overview

Pipelines are the simplest way to run pretrained models for inference. They wrap tokenization/pre-processing, the forward pass, and post-processing behind one call. v5 pipelines are PyTorch-only and the task list was trimmed (see "Removed in v5" below).

## Basic Usage

```python
from transformers import pipeline

# Auto-selects the task's default checkpoint (and warns) — fine for a demo
pipe = pipeline("text-classification")
result = pipe("This is great!")

# Pin the checkpoint for anything you will report or rerun
pipe = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
```

`transformers.pipelines.get_supported_tasks()` lists the tasks your installed version supports.

## Supported Tasks

### Natural Language Processing

**text-generation** (plain prompt or chat messages):
```python
generator = pipeline("text-generation", model="Qwen/Qwen3-0.6B")

# Chat input: returns the conversation with the assistant turn appended
messages = [{"role": "user", "content": "Give three uses of bootstrapping in statistics."}]
out = generator(messages, max_new_tokens=200,
                tokenizer_encode_kwargs={"enable_thinking": False})  # Qwen3-only switch
print(out[0]["generated_text"][-1]["content"])

# Plain-text continuation
generator("Once upon a time", max_new_tokens=50, do_sample=True, num_return_sequences=2)
```
When no model is given, v5 defaults to `HuggingFaceTB/SmolLM3-3B` for this task.

**Summarization, translation, question answering** — no dedicated pipeline in v5; prompt a chat model:
```python
summarizer = pipeline("text-generation", model="Qwen/Qwen3-0.6B")
prompt = [{"role": "user", "content": f"Summarize in 3 bullet points:\n\n{article}"}]
summary = summarizer(prompt, max_new_tokens=200,
                     tokenizer_encode_kwargs={"enable_thinking": False})[0]["generated_text"][-1]["content"]
```
Swap the instruction for "Translate to French: …" or "Answer using only this context: …". Larger instruct models (e.g. `Qwen/Qwen3-4B-Instruct-2507`) give noticeably better summaries than sub-1B models.

**text-classification** (alias `sentiment-analysis`):
```python
classifier = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
result = classifier("I love this product!")  # [{'label': 'POSITIVE', 'score': ...}]
```

**token-classification** (alias `ner`):
```python
ner = pipeline("token-classification", model="dslim/bert-base-NER", aggregation_strategy="simple")
entities = ner("Hugging Face is based in New York City")
```

**fill-mask**:
```python
unmasker = pipeline("fill-mask", model="google-bert/bert-base-uncased")
result = unmasker("Paris is the [MASK] of France.")
```

**zero-shot-classification**:
```python
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
result = classifier(
    "This is a course about Python programming",
    candidate_labels=["education", "politics", "business"],
)
```

**feature-extraction** (token embeddings; pool them yourself):
```python
extractor = pipeline("feature-extraction", model="google-bert/bert-base-uncased")
embeddings = extractor("Some text", return_tensors=True)  # shape (1, seq_len, hidden)
```
For sentence embeddings as a measurement instrument in social-science text analysis, see `alterlab-text-as-data` (sentence-transformers).

**table-question-answering** and **document-question-answering** (`impira/layoutlm-document-qa`, needs `pytesseract` for OCR) are still available.

### Computer Vision

**image-classification**:
```python
from PIL import Image

classifier = pipeline("image-classification", model="google/vit-base-patch16-224")
result = classifier("path/to/image.jpg")         # path, URL, PIL image, or list of them
result = classifier(Image.open("image.jpg"))
```

**object-detection**:
```python
detector = pipeline("object-detection", model="facebook/detr-resnet-50")
results = detector("image.jpg")  # [{'score', 'label', 'box': {xmin, ymin, xmax, ymax}}, ...]
```

**image-segmentation**:
```python
segmenter = pipeline("image-segmentation", model="facebook/detr-resnet-50-panoptic")
segments = segmenter("image.jpg")
```

**depth-estimation**:
```python
depth = pipeline("depth-estimation", model="Intel/dpt-large")
result = depth("image.jpg")
```

**zero-shot-image-classification**:
```python
classifier = pipeline("zero-shot-image-classification", model="openai/clip-vit-base-patch32")
result = classifier("image.jpg", candidate_labels=["cat", "dog", "bird"])
```

Also available: `zero-shot-object-detection`, `image-feature-extraction`, `mask-generation` (SAM), `video-classification`, `keypoint-matching`.

### Audio

**automatic-speech-recognition**:
```python
asr = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3-turbo")
text = asr("audio.mp3", return_timestamps=True)  # timestamps needed for audio > 30 s with Whisper
```
`openai/whisper-base` is a lighter CPU option.

**audio-classification**:
```python
classifier = pipeline("audio-classification", model="MIT/ast-finetuned-audioset-10-10-0.4593")
result = classifier("audio.wav")
```

**text-to-audio** (alias `text-to-speech`):
```python
tts = pipeline("text-to-audio", model="suno/bark-small")
speech = tts("Hello, this is a test")  # {'audio': np.ndarray, 'sampling_rate': int}
```
Some TTS models need extra inputs (e.g. SpeechT5 requires `forward_params={"speaker_embeddings": ...}`); check the model card.

### Multimodal

**image-text-to-text** (captioning, visual question answering, chart/figure reading with a VLM):
```python
vlm = pipeline("image-text-to-text", model="Qwen/Qwen3-VL-2B-Instruct")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "https://example.com/figure.png"},
        {"type": "text", "text": "Describe the trend shown in this figure."},
    ],
}]
out = vlm(text=messages, max_new_tokens=200)
print(out[0]["generated_text"][-1]["content"])
```
In v5 images must be embedded in the chat `content`; passing `images=` alongside a chat is no longer accepted.

`any-to-any` covers omni models (e.g. text + audio + image in, text out).

### Removed in v5

| Removed task | Use instead |
|--------------|-------------|
| `summarization`, `translation_xx_to_yy`, `text2text-generation`, `question-answering` | `text-generation` with a chat model and an instruction prompt |
| `image-to-text`, `visual-question-answering` | `image-text-to-text` with a VLM |
| `image-to-image` | 🤗 Diffusers |

## Pipeline Parameters

**model**: Hub ID or local path
```python
pipe = pipeline("task", model="org/model-id", revision="main")  # pin revision= for reproducibility
```

**device**: GPU index, device string, or -1 for CPU
```python
pipe = pipeline("task", model="org/model-id", device=0)        # first CUDA GPU
pipe = pipeline("task", model="org/model-id", device="mps")    # Apple silicon
```

**device_map**: automatic placement for large models (requires `accelerate`)
```python
pipe = pipeline("task", model="org/large-model", device_map="auto")
```

**dtype**: precision (defaults to `"auto"` = checkpoint dtype in v5)
```python
import torch
pipe = pipeline("task", model="org/model-id", dtype=torch.bfloat16)
```
`torch_dtype=` still works but logs a deprecation warning — use `dtype=`.

**batch_size**: process several inputs per forward pass
```python
pipe = pipeline("task", model="org/model-id", batch_size=8)
results = pipe(["text1", "text2", "text3"])
```

There is no `framework=` argument any more: v5 is PyTorch-only.

## Batch Processing

```python
classifier = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
results = classifier(["Great product!", "Terrible experience", "Just okay"])
```

For large datasets, stream from a `datasets.Dataset` with `KeyDataset` so the pipeline can batch and prefetch:

```python
from datasets import load_dataset
from transformers.pipelines.pt_utils import KeyDataset

dataset = load_dataset("stanfordnlp/imdb", split="test")
pipe = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english",
                device=0, batch_size=32)

for output in pipe(KeyDataset(dataset, "text"), truncation=True):
    print(output)
```

## Performance Optimization

- **GPU**: pass `device=0` (or `device_map="auto"` for models that do not fit on one GPU).
- **Precision**: `dtype=torch.bfloat16` (Ampere+ GPUs) or `torch.float16` roughly halves memory versus float32.
- **Batching**: helps on GPU with similar-length inputs; usually not on CPU, and it adds latency for real-time use.

```python
pipe = pipeline("task", model="org/model-id", batch_size=32, device=0)
results = pipe(list_of_texts)
```

### Streaming Output

```python
from transformers import TextStreamer

generator = pipeline("text-generation", model="Qwen/Qwen3-0.6B")
streamer = TextStreamer(generator.tokenizer, skip_prompt=True)
# Pass the streamer at call time, not to the pipeline constructor
generator("The future of open science is", max_new_tokens=100, streamer=streamer)
```

## Custom Pipeline Configuration

Pass pre-loaded components:

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("org/model-id")
model = AutoModelForSequenceClassification.from_pretrained("org/model-id")
pipe = pipeline("text-classification", model=model, tokenizer=tokenizer)
```

Subclass a pipeline to customize a stage:

```python
from transformers import TextClassificationPipeline

class CustomPipeline(TextClassificationPipeline):
    def postprocess(self, model_outputs, **kwargs):
        # Custom post-processing
        return super().postprocess(model_outputs, **kwargs)

pipe = pipeline("text-classification", model="org/model-id", pipeline_class=CustomPipeline)
```

## Input Formats

- **Text tasks**: a string or a list of strings; `text-generation` also accepts chat message lists.
- **Image tasks**: URLs, file paths, PIL images, or lists of them.
- **Audio tasks**: file paths, NumPy arrays, or `{"raw": array, "sampling_rate": sr}` dicts.

## Error Handling

```python
import torch

try:
    result = pipe(input_data)
except torch.cuda.OutOfMemoryError:
    # Reduce batch_size, lower precision, or fall back to CPU
    pipe = pipeline("task", model="org/model-id", device=-1)
except OSError as e:
    # Raised for unknown/misspelled model IDs or gated repos without a token
    print(f"Check the model ID and your HF_TOKEN: {e}")
```

## Best Practices

1. **Use pipelines for prototyping** and straightforward batch inference.
2. **Pin model IDs (and revisions)**: task defaults change between releases.
3. **Enable GPU and reduced precision** when available.
4. **Batch for throughput** on GPU; skip batching for latency-sensitive use.
5. **Prefer a chat model** for summarization, translation, and QA in v5 — and validate its output on a labelled sample before using it as a research instrument.
6. **Cache models locally**: set `HF_HOME` (not the removed `TRANSFORMERS_CACHE`) to control the cache location.

# Tokenizers

## Overview

Tokenizers convert text into numerical representations (tokens) that models can process. They handle special tokens, padding, truncation, and attention masks. In transformers v5 each model has a single tokenizer class backed by the Rust 🤗 `tokenizers` library (`TokenizersBackend`), with SentencePiece/Python backends only where required — the v4 "slow vs fast" split is gone.

## Loading Tokenizers

### AutoTokenizer

Automatically load the correct tokenizer for a model:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
```

Load from local path:
```python
tokenizer = AutoTokenizer.from_pretrained("./local/tokenizer/path")
```

## Basic Tokenization

### Encode Text

```python
# Simple encoding
text = "Hello, how are you?"
tokens = tokenizer.encode(text)
print(tokens)  # [101, 7592, 1010, 2129, 2024, 2017, 1029, 102]

# With text tokenization
tokens = tokenizer.tokenize(text)
print(tokens)  # ['hello', ',', 'how', 'are', 'you', '?']
```

### Decode Tokens

```python
token_ids = [101, 7592, 1010, 2129, 2024, 2017, 1029, 102]
text = tokenizer.decode(token_ids)
print(text)  # "hello, how are you?"

# Skip special tokens
text = tokenizer.decode(token_ids, skip_special_tokens=True)
print(text)  # "hello, how are you?"

# v5: decode() also accepts a batch (list of lists or a 2-D tensor) and returns a list of strings;
# batch_decode() still works as a backward-compatible alias
texts = tokenizer.decode([[7592, 1010], [2129, 2024, 2017]], skip_special_tokens=True)
```

## The `__call__` Method

Primary tokenization interface:

```python
# Single text
inputs = tokenizer("Hello, how are you?")

# Returns dictionary with input_ids, attention_mask
print(inputs)
# {
#   'input_ids': [101, 7592, 1010, 2129, 2024, 2017, 1029, 102],
#   'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]
# }
```

Multiple texts:
```python
texts = ["Hello", "How are you?"]
inputs = tokenizer(texts, padding=True, truncation=True)
```

## Key Parameters

### Return Tensors

**return_tensors**: Output format — `"pt"`, `"np"`, or `"mlx"` (TensorFlow `"tf"` was removed in v5)
```python
# PyTorch tensors
inputs = tokenizer("text", return_tensors="pt")

# NumPy arrays
inputs = tokenizer("text", return_tensors="np")
```

### Padding

**padding**: Pad sequences to same length
```python
# Pad to longest sequence in batch
inputs = tokenizer(texts, padding=True)

# Pad to specific length
inputs = tokenizer(texts, padding="max_length", max_length=128)

# No padding
inputs = tokenizer(texts, padding=False)
```

**pad_to_multiple_of**: Pad to multiple of specified value
```python
inputs = tokenizer(texts, padding=True, pad_to_multiple_of=8)
```

### Truncation

**truncation**: Limit sequence length
```python
# Truncate to max_length
inputs = tokenizer(text, truncation=True, max_length=512)

# Truncate first sequence in pairs
inputs = tokenizer(text1, text2, truncation="only_first")

# Truncate second sequence
inputs = tokenizer(text1, text2, truncation="only_second")

# Truncate longest first (default for pairs)
inputs = tokenizer(text1, text2, truncation="longest_first", max_length=512)
```

### Max Length

**max_length**: Maximum sequence length
```python
inputs = tokenizer(text, max_length=512, truncation=True)
```

### Additional Outputs

**return_attention_mask**: Include attention mask (default True)
```python
inputs = tokenizer(text, return_attention_mask=True)
```

**return_token_type_ids**: Segment IDs for sentence pairs
```python
inputs = tokenizer(text1, text2, return_token_type_ids=True)
```

**return_offsets_mapping**: Character position mapping (Fast tokenizers only)
```python
inputs = tokenizer(text, return_offsets_mapping=True)
```

**return_length**: Include sequence lengths
```python
inputs = tokenizer(texts, padding=True, return_length=True)
```

## Special Tokens

### Predefined Special Tokens

Access special tokens:
```python
print(tokenizer.cls_token)      # [CLS] or <s>
print(tokenizer.sep_token)      # [SEP] or </s>
print(tokenizer.pad_token)      # [PAD]
print(tokenizer.unk_token)      # [UNK]
print(tokenizer.mask_token)     # [MASK]
print(tokenizer.eos_token)      # End of sequence
print(tokenizer.bos_token)      # Beginning of sequence

# Get IDs
print(tokenizer.cls_token_id)
print(tokenizer.sep_token_id)
```

### Add Special Tokens

Manual control:
```python
# Automatically add special tokens (default True)
inputs = tokenizer(text, add_special_tokens=True)

# Skip special tokens
inputs = tokenizer(text, add_special_tokens=False)
```

### Custom Special Tokens

```python
# v5 key is "extra_special_tokens" ("additional_special_tokens" is still accepted and converted)
special_tokens_dict = {
    "extra_special_tokens": ["<CUSTOM>", "<SPECIAL>"]
}

num_added = tokenizer.add_special_tokens(special_tokens_dict)
print(f"Added {num_added} tokens")
print(tokenizer.extra_special_tokens)  # v5 replacement for additional_special_tokens

# Resize model embeddings after adding tokens
model.resize_token_embeddings(len(tokenizer))
```

## Sentence Pairs

Tokenize text pairs:

```python
text1 = "What is the capital of France?"
text2 = "Paris is the capital of France."

# Automatically handles separation
inputs = tokenizer(text1, text2, padding=True, truncation=True)

# Results in: [CLS] text1 [SEP] text2 [SEP]
```

## Batch Encoding

Process multiple texts:

```python
texts = ["First text", "Second text", "Third text"]

# Basic batch encoding
batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

# Access individual encodings
for i in range(len(texts)):
    input_ids = batch["input_ids"][i]
    attention_mask = batch["attention_mask"][i]
```

## Tokenizer Backends

v5 consolidates the old slow/fast pair into one class per model. `AutoTokenizer` picks the backend automatically from the files and dependencies available (preferring the Rust `tokenizers` backend) and **ignores `use_fast=`**:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
print(type(tokenizer).__name__)  # BertTokenizer — backed by TokenizersBackend
print(tokenizer.is_fast)         # True for the tokenizers backend
```

`PreTrainedTokenizerFast` remains importable as an alias of `TokenizersBackend`; `PreTrainedTokenizer` is the Python backend (`PythonBackend`).

### Rust-Backend Features

**Offset mapping** (character positions):
```python
inputs = tokenizer("Hello world", return_offsets_mapping=True)
print(inputs["offset_mapping"])
# [(0, 0), (0, 5), (6, 11), (0, 0)]  # [CLS], "Hello", "world", [SEP]
```

**Token to word mapping**:
```python
encoding = tokenizer("Hello world")
word_ids = encoding.word_ids()
print(word_ids)  # [None, 0, 1, None]  # [CLS]=None, "Hello"=0, "world"=1, [SEP]=None
```

## Saving Tokenizers

Save locally:
```python
tokenizer.save_pretrained("./my_tokenizer")
```

Push to Hub:
```python
tokenizer.push_to_hub("username/my-tokenizer")
```

## Advanced Usage

### Vocabulary

Access vocabulary:
```python
vocab = tokenizer.get_vocab()
vocab_size = len(vocab)

# Get token for ID
token = tokenizer.convert_ids_to_tokens(100)

# Get ID for token
token_id = tokenizer.convert_tokens_to_ids("hello")
```

### Encoding Details

Get detailed encoding information:

```python
encoding = tokenizer("Hello world", return_tensors="pt")

# Original methods still available
tokens = encoding.tokens()
word_ids = encoding.word_ids()
sequence_ids = encoding.sequence_ids()
```

### Custom Preprocessing

`AutoTokenizer` is a factory (`from_pretrained` returns a concrete model-specific tokenizer class), so you cannot subclass it directly. Wrap the loaded tokenizer instead:

```python
from transformers import AutoTokenizer

class CustomTokenizer:
    def __init__(self, model_id):
        self.tokenizer = AutoTokenizer.from_pretrained(model_id)

    def __call__(self, text, **kwargs):
        # Custom preprocessing, then delegate
        if isinstance(text, str):
            text = text.lower().strip()
        else:
            text = [t.lower().strip() for t in text]
        return self.tokenizer(text, **kwargs)

tokenizer = CustomTokenizer("bert-base-uncased")
```

## Chat Templates

For conversational models:

```python
messages = [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there!"},
    {"role": "user", "content": "How are you?"}
]

# Render to a string (inspect the prompt format)
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(text)

# Tokenize directly — v5 returns a BatchEncoding (input_ids + attention_mask), not a bare tensor
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
```

`add_generation_prompt=True` appends the assistant header so the model answers rather than continuing the user turn. Pass `return_dict=False` only if you really need the bare `input_ids`.

## Common Patterns

### Pattern 1: Simple Text Classification

```python
texts = ["I love this!", "I hate this!"]
labels = [1, 0]

inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt"
)

# Use with model
outputs = model(**inputs, labels=torch.tensor(labels))
```

### Pattern 2: Question Answering

```python
question = "What is the capital?"
context = "Paris is the capital of France."

inputs = tokenizer(
    question,
    context,
    padding=True,
    truncation=True,
    max_length=384,
    return_tensors="pt"
)
```

### Pattern 3: Text Generation

```python
prompt = "Once upon a time"

inputs = tokenizer(prompt, return_tensors="pt")

# Generate — unpack so the attention mask is passed too
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    pad_token_id=tokenizer.eos_token_id
)

# Decode
text = tokenizer.decode(outputs[0], skip_special_tokens=True)
```

### Pattern 4: Dataset Tokenization

```python
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
        max_length=512
    )

# Apply to dataset
tokenized_dataset = dataset.map(tokenize_function, batched=True)
```

## Best Practices

1. **Always specify return_tensors**: For model input
2. **Use padding and truncation**: For batch processing
3. **Set max_length explicitly**: Prevent memory issues
4. **Batch generation with decoder-only models**: load with `padding_side="left"`
5. **Handle pad_token**: Set to eos_token if None for generation
6. **Add special tokens**: Leave enabled (default) unless specific reason
7. **Resize embeddings**: After adding custom tokens
8. **Decode with skip_special_tokens**: For cleaner output
9. **Use batched processing**: For efficiency with datasets
10. **Save tokenizer with model**: Ensure compatibility

## Common Issues

**Padding token not set:**
```python
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
```

**Sequence too long:**
```python
# Enable truncation
inputs = tokenizer(text, truncation=True, max_length=512)
```

**Mismatched vocabulary:**
```python
# Always load tokenizer and model from same checkpoint
tokenizer = AutoTokenizer.from_pretrained("model-id")
model = AutoModel.from_pretrained("model-id")
```

**Attention mask issues:**
```python
# Ensure attention_mask is passed
outputs = model(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"]
)
```

# Training and Fine-Tuning

## Overview

Fine-tune pre-trained models on custom datasets using the Trainer API. The Trainer handles training loops, gradient accumulation, mixed precision, logging, and checkpointing. This page reflects transformers v5 (`TrainingArguments` lost `logging_dir` and `warmup_ratio`; `Trainer(tokenizer=...)` became `processing_class=`; `report_to` defaults to `"none"`).

## Basic Fine-Tuning Workflow

### Step 1: Load and Preprocess Data

```python
from datasets import load_dataset

# Load dataset (canonical Hub datasets now live under an org namespace)
dataset = load_dataset("Yelp/yelp_review_full")
train_dataset = dataset["train"]
eval_dataset = dataset["test"]

# Tokenize
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
        max_length=512
    )

train_dataset = train_dataset.map(tokenize_function, batched=True)
eval_dataset = eval_dataset.map(tokenize_function, batched=True)
```

### Step 2: Load Model

```python
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "google-bert/bert-base-uncased",
    num_labels=5  # Number of classes
)
```

### Step 3: Define Metrics

```python
import evaluate
import numpy as np

metric = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)
```

### Step 4: Configure Training

```python
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./results",
    eval_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    weight_decay=0.01,
    logging_steps=10,
    report_to="tensorboard",  # v5 default is "none"
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
)
```

### Step 5: Create Trainer and Train

```python
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    processing_class=tokenizer,  # v5 name for the old `tokenizer=` argument
    compute_metrics=compute_metrics,
)

# Start training
trainer.train()

# Evaluate
results = trainer.evaluate()
print(results)
```

### Step 6: Save Model

```python
trainer.save_model("./fine_tuned_model")  # also saves processing_class (the tokenizer)

# Or push to the Hub: the target repo comes from TrainingArguments(hub_model_id=...)
# (default: the output_dir name under your account). The first positional argument
# of Trainer.push_to_hub is the commit message, not the repo id.
trainer.push_to_hub(commit_message="Fine-tuned on Yelp reviews")
```

## TrainingArguments Parameters

### Essential Parameters

**output_dir**: Directory for checkpoints and logs
```python
output_dir="./results"
```

**num_train_epochs**: Number of training epochs
```python
num_train_epochs=3
```

**per_device_train_batch_size**: Batch size per GPU/CPU
```python
per_device_train_batch_size=8
```

**learning_rate**: Optimizer learning rate
```python
learning_rate=2e-5  # Common for BERT-style models
learning_rate=5e-5  # Common for smaller models
```

**weight_decay**: L2 regularization
```python
weight_decay=0.01
```

### Evaluation and Saving

**eval_strategy**: When to evaluate ("no", "steps", "epoch")
```python
eval_strategy="epoch"  # Evaluate after each epoch
eval_strategy="steps"  # Evaluate every eval_steps
```

**save_strategy**: When to save checkpoints
```python
save_strategy="epoch"
save_strategy="steps"
save_steps=500
```

**load_best_model_at_end**: Load best checkpoint after training
```python
load_best_model_at_end=True
metric_for_best_model="accuracy"  # Metric to compare
```

### Optimization

**gradient_accumulation_steps**: Accumulate gradients over multiple steps
```python
gradient_accumulation_steps=4  # Effective batch size = batch_size * 4
```

**fp16**: Enable mixed precision (NVIDIA GPUs)
```python
fp16=True
```

**bf16**: Enable bfloat16 (newer GPUs)
```python
bf16=True
```

**gradient_checkpointing**: Trade compute for memory
```python
gradient_checkpointing=True  # Slower but uses less memory
```

**optim**: Optimizer choice
```python
optim="adamw_torch_fused"  # Default with torch >= 2.8 ("adamw_torch" otherwise)
optim="adamw_8bit"         # 8-bit AdamW (alias of adamw_bnb_8bit; requires bitsandbytes)
optim="adafactor"          # Memory-efficient alternative
```

### Learning Rate Scheduling

**lr_scheduler_type**: Learning rate schedule
```python
lr_scheduler_type="linear"       # Linear decay
lr_scheduler_type="cosine"       # Cosine annealing
lr_scheduler_type="constant"     # No decay
lr_scheduler_type="constant_with_warmup"
```

**warmup_steps**: Warmup period — an int is a step count, a float in [0, 1) is a ratio of total steps (`warmup_ratio` was removed in v5)
```python
warmup_steps=500
# Or
warmup_steps=0.1  # 10% of total steps
```

### Logging

**TensorBoard log directory**: the `logging_dir` argument was removed in v5; set the environment variable instead
```bash
export TENSORBOARD_LOGGING_DIR=./logs
```

**logging_steps**: Log every N steps
```python
logging_steps=10
```

**report_to**: Logging integrations (v5 default: `"none"`, so nothing is logged unless you set it)
```python
report_to=["tensorboard"]
report_to=["wandb"]
report_to=["tensorboard", "wandb"]
```

### Distributed Training

**ddp_backend**: Distributed backend
```python
ddp_backend="nccl"  # For multi-GPU
```

**deepspeed**: DeepSpeed config file
```python
deepspeed="ds_config.json"
```

## Data Collators

Handle dynamic padding and special preprocessing:

### DataCollatorWithPadding

Pad sequences to longest in batch:
```python
from transformers import DataCollatorWithPadding

data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    data_collator=data_collator,
)
```

### DataCollatorForLanguageModeling

For masked language modeling:
```python
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=True,
    mlm_probability=0.15
)
```

### DataCollatorForSeq2Seq

For sequence-to-sequence tasks:
```python
from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(
    tokenizer=tokenizer,
    model=model,
    padding=True
)
```

## Custom Training

### Custom Trainer

Override methods for custom behavior:

```python
import torch
from transformers import Trainer

class CustomTrainer(Trainer):
    # Current signature includes num_items_in_batch (added so the Trainer can
    # normalize loss correctly under gradient accumulation). Accept it even if unused.
    def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):
        labels = inputs.pop("labels")
        outputs = model(**inputs)
        logits = outputs.logits

        # Custom loss computation (e.g. class-weighted cross-entropy)
        loss_fct = torch.nn.CrossEntropyLoss(weight=class_weights)
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))

        return (loss, outputs) if return_outputs else loss
```

### Custom Callbacks

Monitor and control training:

```python
from transformers import TrainerCallback

class CustomCallback(TrainerCallback):
    def on_epoch_end(self, args, state, control, **kwargs):
        print(f"Epoch {state.epoch} completed")
        # Custom logic here
        return control

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    callbacks=[CustomCallback],
)
```

## Advanced Training Techniques

### Parameter-Efficient Fine-Tuning (PEFT)

Use LoRA for efficient fine-tuning:

```python
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["query", "value"],
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_CLS"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Shows reduced parameter count

# Train normally with Trainer
trainer = Trainer(model=model, args=training_args, ...)
trainer.train()
```

### Gradient Checkpointing

Reduce memory at cost of speed:

```python
model.gradient_checkpointing_enable()

training_args = TrainingArguments(
    gradient_checkpointing=True,
    ...
)
```

### Mixed Precision Training

```python
training_args = TrainingArguments(
    fp16=True,  # For NVIDIA GPUs with Tensor Cores
    # or
    bf16=True,  # For newer GPUs (A100, H100)
    ...
)
```

### DeepSpeed Integration

For very large models:

```python
# ds_config.json
{
  "train_batch_size": 16,
  "gradient_accumulation_steps": 1,
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": 2e-5
    }
  },
  "fp16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 2
  }
}
```

```python
training_args = TrainingArguments(
    deepspeed="ds_config.json",
    ...
)
```

## Training Tips

### Hyperparameter Tuning

Common starting points:
- **Learning rate**: 2e-5 to 5e-5 for BERT-like models, 1e-4 to 1e-3 for smaller models
- **Batch size**: 8-32 depending on GPU memory
- **Epochs**: 2-4 for fine-tuning, more for domain adaptation
- **Warmup**: 10% of total steps (`warmup_steps=0.1`)

Use Optuna for hyperparameter search:

```python
def model_init():
    return AutoModelForSequenceClassification.from_pretrained(
        "google-bert/bert-base-uncased",
        num_labels=5
    )

def optuna_hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-5, 5e-5, log=True),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [8, 16, 32]),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 2, 5),
    }

trainer = Trainer(model_init=model_init, args=training_args, ...)
best_trial = trainer.hyperparameter_search(
    direction="maximize",
    backend="optuna",
    hp_space=optuna_hp_space,
    n_trials=10,
)
```

### Monitoring Training

Use TensorBoard (with `report_to="tensorboard"`; logs go under `output_dir/runs` unless `TENSORBOARD_LOGGING_DIR` is set):
```bash
tensorboard --logdir ./results/runs
```

Or Weights & Biases:
```python
import wandb
wandb.init(project="my-project")

training_args = TrainingArguments(
    report_to=["wandb"],
    ...
)
```

### Resume Training

Resume from checkpoint:
```python
trainer.train(resume_from_checkpoint="./results/checkpoint-1000")
```

## Common Issues

**CUDA out of memory:**
- Reduce batch size
- Enable gradient checkpointing
- Use gradient accumulation
- Use 8-bit optimizers

**Overfitting:**
- Increase weight_decay
- Add dropout
- Use early stopping
- Reduce model size or training epochs

**Slow training:**
- Increase batch size
- Enable mixed precision (fp16/bf16)
- Use multiple GPUs
- Optimize data loading

## Best Practices

1. **Start small**: Test on small dataset subset first
2. **Use evaluation**: Monitor validation metrics
3. **Save checkpoints**: Enable save_strategy
4. **Log extensively**: Use TensorBoard or W&B
5. **Try different learning rates**: Start with 2e-5
6. **Use warmup**: Helps training stability
7. **Enable mixed precision**: Faster training
8. **Consider PEFT**: For large models with limited resources