alterlab-stable-baselines3
Trains single-agent reinforcement learning agents with Stable-Baselines3 — PPO, SAC, DQN, TD3, DDPG, and A2C behind a scikit-learn-like API. Use for standard single-agent RL experiments, quick prototyping, well-documented algorithm implementations on Gymnasium environments, or ad
Install
npx skills add https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/tree/main/skills/data-science/alterlab-stable-baselines3
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install alterlab-ieu-alterlab-academic-skills@llmmart
git clone https://github.com/AlterLab-IEU/AlterLab-Academic-Skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole alterlab-ieu/alterlab-academic-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Stable Baselines3
Overview
Stable Baselines3 (SB3) is a PyTorch-based library providing reliable implementations of reinforcement learning algorithms. This skill provides comprehensive guidance for training RL agents, creating custom environments, implementing callbacks, and optimizing training workflows using SB3's unified API.
When to Use This Skill
Use this skill when the user wants to:
- Train and evaluate a single-agent RL policy (PPO, A2C, SAC, TD3, DDPG, DQN, HER) on a Gymnasium environment.
- Wrap a research simulator as a custom
gymnasium.Envand validate it withcheck_env. - Add evaluation, checkpointing, early stopping, or custom logging through callbacks.
- Speed up training with vectorized environments, or save, load, and report trained agents reproducibly.
Does NOT Trigger
| Scenario | Use Instead |
|---|---|
| Millions of steps per second, native multi-agent environments, or PufferEnv/Atari-scale throughput | alterlab-pufferlib |
| Supervised deep-learning training loops (LightningModule, multi-GPU strategies) | alterlab-pytorch-lightning |
| Black-box or multi-objective optimization of parameters without a sequential decision process | alterlab-pymoo |
| Simulating queues or resource contention without learning a policy | alterlab-simpy |
Hand-off rule. When the request matches a row above, open your answer by naming that skill by its exact id (for example alterlab-pufferlib) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work.
Core Capabilities
1. Training RL Agents
Basic Training Pattern:
import gymnasium as gym
from stable_baselines3 import PPO
# Create environment
env = gym.make("CartPole-v1")
# Initialize agent
model = PPO("MlpPolicy", env, verbose=1)
# Train the agent
model.learn(total_timesteps=10000)
# Save the model
model.save("ppo_cartpole")
# Load the model (without prior instantiation)
model = PPO.load("ppo_cartpole", env=env)
Important Notes:
total_timestepsis a lower bound; actual training may exceed this due to batch collection- Use
model.load()as a static method, not on an existing instance - The replay buffer is NOT saved with the model to save space
Algorithm Selection:
Use references/algorithms.md for detailed algorithm characteristics and selection guidance. Quick reference:
- PPO/A2C: General-purpose, supports all action space types, good for multiprocessing
- SAC/TD3: Continuous control, off-policy, sample-efficient
- DQN: Discrete actions, off-policy
- HER: Goal-conditioned tasks
See scripts/train_rl_agent.py for a complete training template with best practices.
2. Custom Environments
Requirements:
Custom environments must inherit from gymnasium.Env and implement:
__init__(): Define action_space and observation_spacereset(seed, options): Return initial observation and info dictstep(action): Return observation, reward, terminated, truncated, inforender(): Visualization (optional)close(): Cleanup resources
Key Constraints:
- Image observations must be
np.uint8in range [0, 255] - Use channel-first format when possible (channels, height, width)
- SB3 normalizes images automatically by dividing by 255
- Set
normalize_images=Falsein policy_kwargs if pre-normalized - SB3 does NOT support
DiscreteorMultiDiscretespaces withstart!=0
Validation:
from stable_baselines3.common.env_checker import check_env
check_env(env, warn=True)
See scripts/custom_env_template.py for a complete custom environment template and references/custom_environments.md for comprehensive guidance.
3. Vectorized Environments
Purpose: Vectorized environments run multiple environment instances in parallel, accelerating training and enabling certain wrappers (frame-stacking, normalization).
Types:
- DummyVecEnv: Sequential execution on current process (for lightweight environments)
- SubprocVecEnv: Parallel execution across processes (for compute-heavy environments)
Quick Setup:
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
# Create 4 parallel environments
env = make_vec_env("CartPole-v1", n_envs=4, vec_env_cls=SubprocVecEnv)
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=25000)
With SubprocVecEnv, wrap the call site in if __name__ == "__main__": (required on macOS/Windows spawn).
Off-Policy Optimization:
When using multiple environments with off-policy algorithms (SAC, TD3, DQN), set gradient_steps=-1 to perform one gradient update per environment step, balancing wall-clock time and sample efficiency.
API Differences:
reset()returns only observations (info available invec_env.reset_infos)step()returns 4-tuple:(obs, rewards, dones, infos)not 5-tuple- Environments auto-reset after episodes
- Terminal observations available via
infos[env_idx]["terminal_observation"]
See references/vectorized_envs.md for detailed information on wrappers and advanced usage.
4. Callbacks for Monitoring and Control
Purpose: Callbacks enable monitoring metrics, saving checkpoints, implementing early stopping, and custom training logic without modifying core algorithms.
Common Callbacks:
- EvalCallback: Evaluate periodically and save best model
- CheckpointCallback: Save model checkpoints at intervals
- StopTrainingOnRewardThreshold: Stop when target reward reached
- ProgressBarCallback: Display training progress with timing
Custom Callback Structure:
from stable_baselines3.common.callbacks import BaseCallback
class CustomCallback(BaseCallback):
def _on_training_start(self):
# Called before first rollout
pass
def _on_step(self):
# Called after each environment step
# Return False to stop training
return True
def _on_rollout_end(self):
# Called at end of rollout
pass
Available Attributes:
self.model: The RL algorithm instanceself.num_timesteps: Total environment stepsself.training_env: The training environment
Chaining Callbacks:
from stable_baselines3.common.callbacks import CallbackList
callback = CallbackList([eval_callback, checkpoint_callback, custom_callback])
model.learn(total_timesteps=10000, callback=callback)
See references/callbacks.md for comprehensive callback documentation.
5. Model Persistence and Inspection
Saving and Loading:
# Save model
model.save("model_name")
# Save normalization statistics (if using VecNormalize)
vec_env.save("vec_normalize.pkl")
# Load model
model = PPO.load("model_name", env=env)
# Load normalization statistics
vec_env = VecNormalize.load("vec_normalize.pkl", vec_env)
Parameter Access:
# Get parameters
params = model.get_parameters()
# Set parameters
model.set_parameters(params)
# Access PyTorch state dict
state_dict = model.policy.state_dict()
6. Evaluation and Recording
Evaluation:
from stable_baselines3.common.evaluation import evaluate_policy
mean_reward, std_reward = evaluate_policy(
model,
env,
n_eval_episodes=10,
deterministic=True
)
Video Recording (the wrapped env must be created with render_mode="rgb_array", and moviepy must be importable, e.g. add --with "gymnasium[other]" to the uv run command):
from stable_baselines3.common.vec_env import VecVideoRecorder
# Wrap environment with video recorder
env = VecVideoRecorder(
env,
"videos/",
record_video_trigger=lambda x: x % 2000 == 0,
video_length=200
)
See scripts/evaluate_agent.py for a complete evaluation and recording template.
7. Advanced Features
Learning Rate Schedules:
def linear_schedule(initial_value):
def func(progress_remaining):
# progress_remaining goes from 1 to 0
return progress_remaining * initial_value
return func
model = PPO("MlpPolicy", env, learning_rate=linear_schedule(0.001))
# Built-in equivalent (SB3 >= 2.7; replaces the deprecated get_linear_fn/constant_fn/get_schedule_fn)
from stable_baselines3.common.utils import LinearSchedule
model = PPO("MlpPolicy", env, learning_rate=LinearSchedule(1e-3, 0.0, 1.0))
N-step Returns (off-policy, SB3 >= 2.7): SAC, TD3, DDPG, and DQN accept n_steps= to bootstrap from n-step returns (backed by NStepReplayBuffer).
Multi-Input Policies (Dict Observations):
model = PPO("MultiInputPolicy", env, verbose=1)
Use when observations are dictionaries (e.g., combining images with sensor data).
Hindsight Experience Replay:
from stable_baselines3 import SAC, HerReplayBuffer
model = SAC(
"MultiInputPolicy",
env,
replay_buffer_class=HerReplayBuffer,
replay_buffer_kwargs=dict(
n_sampled_goal=4,
goal_selection_strategy="future",
),
)
TensorBoard Integration:
model = PPO("MlpPolicy", env, tensorboard_log="./tensorboard/")
model.learn(total_timesteps=10000)
Workflow Guidance
Starting a New RL Project:
- Define the problem: Identify observation space, action space, and reward structure
- Choose algorithm: Use
references/algorithms.mdfor selection guidance - Create/adapt environment: Use
scripts/custom_env_template.pyif needed - Validate environment: Always run
check_env()before training - Set up training: Use
scripts/train_rl_agent.pyas starting template - Add monitoring: Implement callbacks for evaluation and checkpointing
- Optimize performance: Consider vectorized environments for speed
- Evaluate and iterate: Use
scripts/evaluate_agent.pyfor assessment
Common Issues:
- Memory errors: Reduce
buffer_sizefor off-policy algorithms or use fewer parallel environments - Slow training: Consider SubprocVecEnv for parallel environments
- Unstable training: Try different algorithms, tune hyperparameters, or check reward scaling
- Import errors: Run through uv instead of installing:
uv run --python ">=3.12" --with "stable-baselines3[extra]>=2.0" python train.py
Resources
scripts/
train_rl_agent.py: Complete training script template with best practicesevaluate_agent.py: Agent evaluation and video recording templatecustom_env_template.py: Custom Gym environment template
references/
algorithms.md: Detailed algorithm comparison and selection guidecustom_environments.md: Comprehensive custom environment creation guidecallbacks.md: Complete callback system referencevectorized_envs.md: Vectorized environment usage and wrappers
Installation
Requires Python 3.10+ and PyTorch >= 2.8 (SB3 2.9 raised the floor from 2.3). The [extra] option pulls in TensorBoard, OpenCV, ale-py (Atari), pygame-ce, tqdm/rich (progress bar), and — since 2.9 — pandas and matplotlib, which are no longer core dependencies (needed for results_plotter/load_results). Atari IDs such as "PongNoFrameskip-v4" or "ALE/Pong-v5" are registered when ale_py is imported (import ale_py; gym.register_envs(ale_py)).
# No install step: uv resolves SB3 (and PyTorch) per call
uv run --python ">=3.12" --with "stable-baselines3>=2.0" python train.py
# With extra dependencies (Tensorboard, Atari, etc.)
uv run --python ">=3.12" --with "stable-baselines3[extra]>=2.0" python train.py
# SB3 already importable (`python -c "import stable_baselines3"`)? run with plain `python`.
# CPU-only machine: add --index https://download.pytorch.org/whl/cpu so uv fetches CPU torch, not multi-GB CUDA wheels
Part of the AlterLab Academic Skills suite.
Files (alterlab-academic-skills)
-
evals
-
fixtures
-
inventory_env.py 2.5 KB
Loading…
-
-
evals.json 6.1 KB
Loading…
-
-
references
-
algorithms.md 10.1 KB
Loading… -
callbacks.md 15.3 KB
Loading… -
custom_environments.md 13.4 KB
Loading… -
vectorized_envs.md 14.8 KB
Loading…
-
-
scripts
-
custom_env_template.py 9.1 KB
Loading… -
evaluate_agent.py 7.7 KB
Loading… -
train_rl_agent.py 5 KB
Loading…
-
-
SKILL.md 12.8 KB
--- name: alterlab-stable-baselines3 description: Trains single-agent reinforcement learning agents with Stable-Baselines3 — PPO, SAC, DQN, TD3, DDPG, and A2C behind a scikit-learn-like API. Use for standard single-agent RL experiments, quick prototyping, well-documented algorithm implementations on Gymnasium environments, or adding callbacks and evaluation. For high-throughput parallel training, multi-agent systems, or custom vectorized environments prefer alterlab-pufferlib. Part of the AlterLab Academic Skills suite. license: MIT allowed-tools: Read Write Edit Bash(python:*) Bash(uv:*) compatibility: No API key required. Runs locally via `uv run --python ">=3.12" --with "stable-baselines3[extra]>=2.0" python script.py` (no install step; the interpreter pin matters because an older default Python resolves older releases); requires stable-baselines3 >= 2.0 (current 2.9.0 as of 2026-09, which needs Python >= 3.10, PyTorch >= 2.8, and gymnasium >= 0.29.1,<2.0; current gymnasium 1.3). metadata: skill-author: AlterLab version: "1.0.1" last_updated: "2026-09-23" --- # Stable Baselines3 ## Overview Stable Baselines3 (SB3) is a PyTorch-based library providing reliable implementations of reinforcement learning algorithms. This skill provides comprehensive guidance for training RL agents, creating custom environments, implementing callbacks, and optimizing training workflows using SB3's unified API. ## When to Use This Skill Use this skill when the user wants to: - Train and evaluate a single-agent RL policy (PPO, A2C, SAC, TD3, DDPG, DQN, HER) on a Gymnasium environment. - Wrap a research simulator as a custom `gymnasium.Env` and validate it with `check_env`. - Add evaluation, checkpointing, early stopping, or custom logging through callbacks. - Speed up training with vectorized environments, or save, load, and report trained agents reproducibly. ### Does NOT Trigger | Scenario | Use Instead | |----------|-------------| | Millions of steps per second, native multi-agent environments, or PufferEnv/Atari-scale throughput | `alterlab-pufferlib` | | Supervised deep-learning training loops (LightningModule, multi-GPU strategies) | `alterlab-pytorch-lightning` | | Black-box or multi-objective optimization of parameters without a sequential decision process | `alterlab-pymoo` | | Simulating queues or resource contention without learning a policy | `alterlab-simpy` | **Hand-off rule.** When the request matches a row above, open your answer by naming that skill by its exact id (for example `alterlab-pufferlib`) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work. ## Core Capabilities ### 1. Training RL Agents **Basic Training Pattern:** ```python import gymnasium as gym from stable_baselines3 import PPO # Create environment env = gym.make("CartPole-v1") # Initialize agent model = PPO("MlpPolicy", env, verbose=1) # Train the agent model.learn(total_timesteps=10000) # Save the model model.save("ppo_cartpole") # Load the model (without prior instantiation) model = PPO.load("ppo_cartpole", env=env) ``` **Important Notes:** - `total_timesteps` is a lower bound; actual training may exceed this due to batch collection - Use `model.load()` as a static method, not on an existing instance - The replay buffer is NOT saved with the model to save space **Algorithm Selection:** Use `references/algorithms.md` for detailed algorithm characteristics and selection guidance. Quick reference: - **PPO/A2C**: General-purpose, supports all action space types, good for multiprocessing - **SAC/TD3**: Continuous control, off-policy, sample-efficient - **DQN**: Discrete actions, off-policy - **HER**: Goal-conditioned tasks See `scripts/train_rl_agent.py` for a complete training template with best practices. ### 2. Custom Environments **Requirements:** Custom environments must inherit from `gymnasium.Env` and implement: - `__init__()`: Define action_space and observation_space - `reset(seed, options)`: Return initial observation and info dict - `step(action)`: Return observation, reward, terminated, truncated, info - `render()`: Visualization (optional) - `close()`: Cleanup resources **Key Constraints:** - Image observations must be `np.uint8` in range [0, 255] - Use channel-first format when possible (channels, height, width) - SB3 normalizes images automatically by dividing by 255 - Set `normalize_images=False` in policy_kwargs if pre-normalized - SB3 does NOT support `Discrete` or `MultiDiscrete` spaces with `start!=0` **Validation:** ```python from stable_baselines3.common.env_checker import check_env check_env(env, warn=True) ``` See `scripts/custom_env_template.py` for a complete custom environment template and `references/custom_environments.md` for comprehensive guidance. ### 3. Vectorized Environments **Purpose:** Vectorized environments run multiple environment instances in parallel, accelerating training and enabling certain wrappers (frame-stacking, normalization). **Types:** - **DummyVecEnv**: Sequential execution on current process (for lightweight environments) - **SubprocVecEnv**: Parallel execution across processes (for compute-heavy environments) **Quick Setup:** ```python from stable_baselines3.common.env_util import make_vec_env from stable_baselines3.common.vec_env import SubprocVecEnv # Create 4 parallel environments env = make_vec_env("CartPole-v1", n_envs=4, vec_env_cls=SubprocVecEnv) model = PPO("MlpPolicy", env, verbose=1) model.learn(total_timesteps=25000) ``` With `SubprocVecEnv`, wrap the call site in `if __name__ == "__main__":` (required on macOS/Windows spawn). **Off-Policy Optimization:** When using multiple environments with off-policy algorithms (SAC, TD3, DQN), set `gradient_steps=-1` to perform one gradient update per environment step, balancing wall-clock time and sample efficiency. **API Differences:** - `reset()` returns only observations (info available in `vec_env.reset_infos`) - `step()` returns 4-tuple: `(obs, rewards, dones, infos)` not 5-tuple - Environments auto-reset after episodes - Terminal observations available via `infos[env_idx]["terminal_observation"]` See `references/vectorized_envs.md` for detailed information on wrappers and advanced usage. ### 4. Callbacks for Monitoring and Control **Purpose:** Callbacks enable monitoring metrics, saving checkpoints, implementing early stopping, and custom training logic without modifying core algorithms. **Common Callbacks:** - **EvalCallback**: Evaluate periodically and save best model - **CheckpointCallback**: Save model checkpoints at intervals - **StopTrainingOnRewardThreshold**: Stop when target reward reached - **ProgressBarCallback**: Display training progress with timing **Custom Callback Structure:** ```python from stable_baselines3.common.callbacks import BaseCallback class CustomCallback(BaseCallback): def _on_training_start(self): # Called before first rollout pass def _on_step(self): # Called after each environment step # Return False to stop training return True def _on_rollout_end(self): # Called at end of rollout pass ``` **Available Attributes:** - `self.model`: The RL algorithm instance - `self.num_timesteps`: Total environment steps - `self.training_env`: The training environment **Chaining Callbacks:** ```python from stable_baselines3.common.callbacks import CallbackList callback = CallbackList([eval_callback, checkpoint_callback, custom_callback]) model.learn(total_timesteps=10000, callback=callback) ``` See `references/callbacks.md` for comprehensive callback documentation. ### 5. Model Persistence and Inspection **Saving and Loading:** ```python # Save model model.save("model_name") # Save normalization statistics (if using VecNormalize) vec_env.save("vec_normalize.pkl") # Load model model = PPO.load("model_name", env=env) # Load normalization statistics vec_env = VecNormalize.load("vec_normalize.pkl", vec_env) ``` **Parameter Access:** ```python # Get parameters params = model.get_parameters() # Set parameters model.set_parameters(params) # Access PyTorch state dict state_dict = model.policy.state_dict() ``` ### 6. Evaluation and Recording **Evaluation:** ```python from stable_baselines3.common.evaluation import evaluate_policy mean_reward, std_reward = evaluate_policy( model, env, n_eval_episodes=10, deterministic=True ) ``` **Video Recording** (the wrapped env must be created with `render_mode="rgb_array"`, and `moviepy` must be importable, e.g. add `--with "gymnasium[other]"` to the uv run command): ```python from stable_baselines3.common.vec_env import VecVideoRecorder # Wrap environment with video recorder env = VecVideoRecorder( env, "videos/", record_video_trigger=lambda x: x % 2000 == 0, video_length=200 ) ``` See `scripts/evaluate_agent.py` for a complete evaluation and recording template. ### 7. Advanced Features **Learning Rate Schedules:** ```python def linear_schedule(initial_value): def func(progress_remaining): # progress_remaining goes from 1 to 0 return progress_remaining * initial_value return func model = PPO("MlpPolicy", env, learning_rate=linear_schedule(0.001)) # Built-in equivalent (SB3 >= 2.7; replaces the deprecated get_linear_fn/constant_fn/get_schedule_fn) from stable_baselines3.common.utils import LinearSchedule model = PPO("MlpPolicy", env, learning_rate=LinearSchedule(1e-3, 0.0, 1.0)) ``` **N-step Returns (off-policy, SB3 >= 2.7):** SAC, TD3, DDPG, and DQN accept `n_steps=` to bootstrap from n-step returns (backed by `NStepReplayBuffer`). **Multi-Input Policies (Dict Observations):** ```python model = PPO("MultiInputPolicy", env, verbose=1) ``` Use when observations are dictionaries (e.g., combining images with sensor data). **Hindsight Experience Replay:** ```python from stable_baselines3 import SAC, HerReplayBuffer model = SAC( "MultiInputPolicy", env, replay_buffer_class=HerReplayBuffer, replay_buffer_kwargs=dict( n_sampled_goal=4, goal_selection_strategy="future", ), ) ``` **TensorBoard Integration:** ```python model = PPO("MlpPolicy", env, tensorboard_log="./tensorboard/") model.learn(total_timesteps=10000) ``` ## Workflow Guidance **Starting a New RL Project:** 1. **Define the problem**: Identify observation space, action space, and reward structure 2. **Choose algorithm**: Use `references/algorithms.md` for selection guidance 3. **Create/adapt environment**: Use `scripts/custom_env_template.py` if needed 4. **Validate environment**: Always run `check_env()` before training 5. **Set up training**: Use `scripts/train_rl_agent.py` as starting template 6. **Add monitoring**: Implement callbacks for evaluation and checkpointing 7. **Optimize performance**: Consider vectorized environments for speed 8. **Evaluate and iterate**: Use `scripts/evaluate_agent.py` for assessment **Common Issues:** - **Memory errors**: Reduce `buffer_size` for off-policy algorithms or use fewer parallel environments - **Slow training**: Consider SubprocVecEnv for parallel environments - **Unstable training**: Try different algorithms, tune hyperparameters, or check reward scaling - **Import errors**: Run through uv instead of installing: `uv run --python ">=3.12" --with "stable-baselines3[extra]>=2.0" python train.py` ## Resources ### scripts/ - `train_rl_agent.py`: Complete training script template with best practices - `evaluate_agent.py`: Agent evaluation and video recording template - `custom_env_template.py`: Custom Gym environment template ### references/ - `algorithms.md`: Detailed algorithm comparison and selection guide - `custom_environments.md`: Comprehensive custom environment creation guide - `callbacks.md`: Complete callback system reference - `vectorized_envs.md`: Vectorized environment usage and wrappers ## Installation Requires Python 3.10+ and PyTorch >= 2.8 (SB3 2.9 raised the floor from 2.3). The `[extra]` option pulls in TensorBoard, OpenCV, `ale-py` (Atari), `pygame-ce`, tqdm/rich (progress bar), and — since 2.9 — pandas and matplotlib, which are no longer core dependencies (needed for `results_plotter`/`load_results`). Atari IDs such as `"PongNoFrameskip-v4"` or `"ALE/Pong-v5"` are registered when `ale_py` is imported (`import ale_py; gym.register_envs(ale_py)`). ```bash # No install step: uv resolves SB3 (and PyTorch) per call uv run --python ">=3.12" --with "stable-baselines3>=2.0" python train.py # With extra dependencies (Tensorboard, Atari, etc.) uv run --python ">=3.12" --with "stable-baselines3[extra]>=2.0" python train.py # SB3 already importable (`python -c "import stable_baselines3"`)? run with plain `python`. # CPU-only machine: add --index https://download.pytorch.org/whl/cpu so uv fetches CPU torch, not multi-GB CUDA wheels ``` Part of the AlterLab Academic Skills suite.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.