alterlab-vaex
Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-c
Install
npx skills add https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/tree/main/skills/data-science/alterlab-vaex
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install alterlab-ieu-alterlab-academic-skills@llmmart
git clone https://github.com/AlterLab-IEU/AlterLab-Academic-Skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole alterlab-ieu/alterlab-academic-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Vaex
Overview
Vaex is a Python library for lazy, out-of-core DataFrames: it memory-maps HDF5/Arrow files, evaluates expressions lazily, and computes statistics on N-dimensional grids fast enough for interactive work on billions of rows on a single machine.
Maintenance Status (read first)
Vaex is effectively in minimal-maintenance mode. In the project's "Is Vaex active?" discussion (vaexio/vaex #2363, June 2024), one core developer stated he is no longer involved and the other committed only to "do my best to at least get vaex releases out". Releases have slowed to roughly one per year (vaex-core 4.18 in Oct 2024, 4.19 in Sep 2025), and 4.19 still supports only Python 3.9–3.12 with pandas<3; the 2-D matplotlib plots break on matplotlib ≥ 3.9. Practical consequences:
- Existing Vaex code: keep it running in a pinned Python 3.12 environment (below), and plan a migration.
- New projects: default to Polars (
alterlab-polars) —pl.scan_parquet(...)pluscollect(engine="streaming")covers most larger-than-RAM aggregation work — or Dask (alterlab-dask) for clusters. Recommend Vaex for new work only when its specific strengths (memory-mapped HDF5, fast binned statistics/visualization of huge tables) matter and the user accepts the environment constraints. - Data already in large HDF5 files, one machine, no cluster: this skill's job. Polars has no HDF5 reader, so open the file with
vaex.open(memory-mapped) and run the aggregations here; mention a one-time Parquet export only as the migration path.
When to Use This Skill
Use this skill when the user wants to:
- Run, debug, or extend an existing Vaex pipeline (
vaex.open, virtual columns, selections,vaex.ml). - Open memory-mapped HDF5/Arrow files that exceed RAM and compute fast aggregations or binned statistics.
- Batch several aggregations into one pass with
delay=True/df.execute(). - Convert large CSV to HDF5/Arrow/Parquet, or port a Vaex workflow to Polars.
Does NOT Trigger
| Scenario | Use Instead |
|---|---|
| New larger-than-RAM analytics project with no existing Vaex code whose data is Parquet/CSV/Arrow (not HDF5), or Python 3.13+ / pandas 3 environments | alterlab-polars |
| Data fits in memory and the goal is the fastest in-memory groupby/joins | alterlab-polars |
| Multi-node cluster execution, task graphs, or a distributed scheduler dashboard | alterlab-dask |
| Chunked N-D scientific arrays (not tables) on disk or cloud object stores | alterlab-zarr |
Hand-off rule. When the request matches a row above, open your answer by naming that skill by its exact id (for example alterlab-polars) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work.
Installation
# Dedicated environment — vaex 4.19 has no wheels for Python 3.13+ and requires pandas < 3
uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex "vaex==4.19.*"
uv pip install --python .venv-vaex "matplotlib<3.9" # only if you need df.viz.heatmap
uv run --python .venv-vaex python analysis.py
# One-off alternative with no venv: uv run --python 3.12 --with "vaex==4.19.*" --with "matplotlib<3.9" python analysis.py
Core Capabilities
Vaex provides six capability areas, each documented in references/:
1. DataFrames and Data Loading
Open HDF5/Arrow/Parquet/CSV files, convert from pandas/NumPy/Arrow, and inspect structure — references/core_dataframes.md.
2. Data Processing and Manipulation
Filtering, named selections, virtual columns and expressions, groupby/aggregations, strings, datetimes, and missing data — references/data_processing.md.
3. Performance and Optimization
Lazy evaluation, delay=True batching, materialization, caching, and async execution — references/performance.md.
4. Data Visualization
Binned histograms and heatmaps of huge tables via the df.viz accessor, selections, and widgets — references/visualization.md.
5. Machine Learning Integration
vaex.ml scalers, encoders, PCA, k-means, and scikit-learn/XGBoost/LightGBM/CatBoost wrappers — references/machine_learning.md. The booster and scikit-learn wrappers materialize the training features in memory; only the transformers and prediction columns stay lazy.
6. I/O Operations
Format recommendations, export strategies, Arrow interop, chunked CSV conversion, and remote data — references/io_operations.md.
Quick Start Pattern
import vaex
# 1. Open or create a DataFrame
df = vaex.open('large_file.hdf5') # or .arrow, .parquet; CSV is read (and optionally converted)
# OR
df = vaex.from_pandas(pandas_df)
# 2. Explore
print(df) # first/last rows and column info
df.describe() # statistical summary
# 3. Virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y
# 4. Filter
df_filtered = df[df.age > 25]
# 5. Statistics (lazy, computed in passes over the data)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})
# 6. Visualize (df.plot1d / df.plot are deprecated aliases of these)
df.viz.histogram(df.x, limits=[0, 100])
df.viz.heatmap(df.x, df.y, limits='99.7%') # needs matplotlib < 3.9
# 7. Export
df.export_hdf5('output.hdf5')
Best Practices
- Use HDF5 or Apache Arrow for memory-mapped access to large datasets; Parquet is the most portable choice if a Polars migration is likely.
- Prefer virtual columns to materializing data.
- Batch operations with
delay=Truewhen computing several statistics. - Convert CSV once to HDF5/Arrow/Parquet rather than re-reading it.
- Check memory with
df.byte_size()and inspect data withdf.describe()before optimizing. - Pin the environment (Python 3.12,
vaex==4.19.*) and record it with results so analyses stay reproducible.
Common Patterns
Pattern: Converting Large CSV to HDF5
import vaex
# convert=True streams the CSV in chunks to an HDF5 file next to it
df = vaex.from_csv('large_file.csv', convert=True, chunk_size=5_000_000)
# Future loads are memory-mapped and near-instant
df = vaex.open('large_file.csv.hdf5')
Pattern: Efficient Aggregations
# delay=True batches several aggregations into ONE pass over the data.
# Each delayed call returns a promise; df.execute() runs them together.
mean_x = df.mean(df.x, delay=True)
std_y = df.std(df.y, delay=True)
sum_z = df.sum(df.z, delay=True)
df.execute()
print(mean_x.get(), std_y.get(), sum_z.get())
Pattern: Virtual Columns for Feature Engineering
# No memory overhead — computed on the fly
df['age_squared'] = df.age ** 2
df['full_name'] = df.first_name + ' ' + df.last_name
df['is_adult'] = df.age >= 18
Pattern: Migrating a Vaex Query to Polars
# Vaex: df = vaex.open('big.parquet'); df['z'] = df.x ** 2 + df.y
# df[df.x > 0.5].groupby('category').agg({'z': 'mean'})
import polars as pl
result = (
pl.scan_parquet('big.parquet') # lazy, like vaex.open
.with_columns((pl.col('x') ** 2 + pl.col('y')).alias('z')) # virtual column
.filter(pl.col('x') > 0.5) # filter
.group_by('category').agg(pl.col('z').mean()) # groupby/agg
.collect(engine='streaming') # out-of-core execution
)
Polars cannot read Vaex's HDF5 layout directly — export once with df.export_parquet(...) or df.export_arrow(...). Several delay=True statistics become one select(...) in a single lazy query. See alterlab-polars for the rest of the Polars API.
Resources
references/core_dataframes.md— DataFrame creation, loading, and basic structurereferences/data_processing.md— filtering, expressions, aggregations, and transformationsreferences/performance.md— optimization strategies and lazy evaluationreferences/visualization.md— plotting and interactive visualizationsreferences/machine_learning.md— ML pipelines and model integrationreferences/io_operations.md— file formats and data import/export
Part of the AlterLab Academic Skills suite.
Files (alterlab-academic-skills)
-
evals
-
evals.json 6.1 KB
Loading…
-
-
references
-
core_dataframes.md 7.8 KB
Loading… -
data_processing.md 13 KB
Loading… -
io_operations.md 15.4 KB
Loading… -
machine_learning.md 17.5 KB
Loading… -
performance.md 14.1 KB
Loading… -
visualization.md 13.9 KB
Loading…
-
-
SKILL.md 9.4 KB
--- name: alterlab-vaex description: Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-core 4.19 from Sep 2025 supports only Python 3.9–3.12 and pandas < 3), so use it to run, fix, or migrate existing Vaex code; for new larger-than-RAM work prefer Polars lazy/streaming queries (alterlab-polars) or Dask (alterlab-dask). Use when a project already depends on Vaex, when opening memory-mapped HDF5/Arrow files, batching aggregations with delay=True, or porting a Vaex pipeline to Polars. Part of the AlterLab Academic Skills suite. license: MIT allowed-tools: Read Write Edit Bash(python:*) Bash(uv:*) compatibility: No API key required. Runs locally via `uv run --python 3.12 --with "vaex==4.19.*" python script.py` (no install step) or in a dedicated Python 3.12 (or older) environment; requires vaex 4.19.x, which pins pandas < 3 and has no wheels for Python 3.13+. 2-D plots (`df.viz.heatmap`) also need matplotlib < 3.9. metadata: skill-author: AlterLab version: "1.1.0" last_updated: "2026-09-23" --- # Vaex ## Overview Vaex is a Python library for lazy, out-of-core DataFrames: it memory-maps HDF5/Arrow files, evaluates expressions lazily, and computes statistics on N-dimensional grids fast enough for interactive work on billions of rows on a single machine. ## Maintenance Status (read first) Vaex is effectively in minimal-maintenance mode. In the project's "Is Vaex active?" discussion (vaexio/vaex #2363, June 2024), one core developer stated he is no longer involved and the other committed only to "do my best to at least get vaex releases out". Releases have slowed to roughly one per year (vaex-core 4.18 in Oct 2024, 4.19 in Sep 2025), and 4.19 still supports only Python 3.9–3.12 with `pandas<3`; the 2-D matplotlib plots break on matplotlib ≥ 3.9. Practical consequences: - **Existing Vaex code**: keep it running in a pinned Python 3.12 environment (below), and plan a migration. - **New projects**: default to Polars (`alterlab-polars`) — `pl.scan_parquet(...)` plus `collect(engine="streaming")` covers most larger-than-RAM aggregation work — or Dask (`alterlab-dask`) for clusters. Recommend Vaex for new work only when its specific strengths (memory-mapped HDF5, fast binned statistics/visualization of huge tables) matter and the user accepts the environment constraints. - **Data already in large HDF5 files, one machine, no cluster**: this skill's job. Polars has no HDF5 reader, so open the file with `vaex.open` (memory-mapped) and run the aggregations here; mention a one-time Parquet export only as the migration path. ## When to Use This Skill Use this skill when the user wants to: - Run, debug, or extend an existing Vaex pipeline (`vaex.open`, virtual columns, selections, `vaex.ml`). - Open memory-mapped HDF5/Arrow files that exceed RAM and compute fast aggregations or binned statistics. - Batch several aggregations into one pass with `delay=True` / `df.execute()`. - Convert large CSV to HDF5/Arrow/Parquet, or port a Vaex workflow to Polars. ### Does NOT Trigger | Scenario | Use Instead | |----------|-------------| | New larger-than-RAM analytics project with no existing Vaex code whose data is Parquet/CSV/Arrow (not HDF5), or Python 3.13+ / pandas 3 environments | `alterlab-polars` | | Data fits in memory and the goal is the fastest in-memory groupby/joins | `alterlab-polars` | | Multi-node cluster execution, task graphs, or a distributed scheduler dashboard | `alterlab-dask` | | Chunked N-D scientific arrays (not tables) on disk or cloud object stores | `alterlab-zarr` | **Hand-off rule.** When the request matches a row above, open your answer by naming that skill by its exact id (for example `alterlab-polars`) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work. ## Installation ```bash # Dedicated environment — vaex 4.19 has no wheels for Python 3.13+ and requires pandas < 3 uv venv --python 3.12 .venv-vaex uv pip install --python .venv-vaex "vaex==4.19.*" uv pip install --python .venv-vaex "matplotlib<3.9" # only if you need df.viz.heatmap uv run --python .venv-vaex python analysis.py # One-off alternative with no venv: uv run --python 3.12 --with "vaex==4.19.*" --with "matplotlib<3.9" python analysis.py ``` ## Core Capabilities Vaex provides six capability areas, each documented in `references/`: ### 1. DataFrames and Data Loading Open HDF5/Arrow/Parquet/CSV files, convert from pandas/NumPy/Arrow, and inspect structure — `references/core_dataframes.md`. ### 2. Data Processing and Manipulation Filtering, named selections, virtual columns and expressions, groupby/aggregations, strings, datetimes, and missing data — `references/data_processing.md`. ### 3. Performance and Optimization Lazy evaluation, `delay=True` batching, materialization, caching, and async execution — `references/performance.md`. ### 4. Data Visualization Binned histograms and heatmaps of huge tables via the `df.viz` accessor, selections, and widgets — `references/visualization.md`. ### 5. Machine Learning Integration `vaex.ml` scalers, encoders, PCA, k-means, and scikit-learn/XGBoost/LightGBM/CatBoost wrappers — `references/machine_learning.md`. The booster and scikit-learn wrappers materialize the training features in memory; only the transformers and prediction columns stay lazy. ### 6. I/O Operations Format recommendations, export strategies, Arrow interop, chunked CSV conversion, and remote data — `references/io_operations.md`. ## Quick Start Pattern ```python import vaex # 1. Open or create a DataFrame df = vaex.open('large_file.hdf5') # or .arrow, .parquet; CSV is read (and optionally converted) # OR df = vaex.from_pandas(pandas_df) # 2. Explore print(df) # first/last rows and column info df.describe() # statistical summary # 3. Virtual columns (no memory overhead) df['new_column'] = df.x ** 2 + df.y # 4. Filter df_filtered = df[df.age > 25] # 5. Statistics (lazy, computed in passes over the data) mean_val = df.x.mean() stats = df.groupby('category').agg({'value': 'sum'}) # 6. Visualize (df.plot1d / df.plot are deprecated aliases of these) df.viz.histogram(df.x, limits=[0, 100]) df.viz.heatmap(df.x, df.y, limits='99.7%') # needs matplotlib < 3.9 # 7. Export df.export_hdf5('output.hdf5') ``` ## Best Practices 1. **Use HDF5 or Apache Arrow** for memory-mapped access to large datasets; Parquet is the most portable choice if a Polars migration is likely. 2. **Prefer virtual columns** to materializing data. 3. **Batch operations** with `delay=True` when computing several statistics. 4. **Convert CSV once** to HDF5/Arrow/Parquet rather than re-reading it. 5. **Check memory with `df.byte_size()`** and inspect data with `df.describe()` before optimizing. 6. **Pin the environment** (Python 3.12, `vaex==4.19.*`) and record it with results so analyses stay reproducible. ## Common Patterns ### Pattern: Converting Large CSV to HDF5 ```python import vaex # convert=True streams the CSV in chunks to an HDF5 file next to it df = vaex.from_csv('large_file.csv', convert=True, chunk_size=5_000_000) # Future loads are memory-mapped and near-instant df = vaex.open('large_file.csv.hdf5') ``` ### Pattern: Efficient Aggregations ```python # delay=True batches several aggregations into ONE pass over the data. # Each delayed call returns a promise; df.execute() runs them together. mean_x = df.mean(df.x, delay=True) std_y = df.std(df.y, delay=True) sum_z = df.sum(df.z, delay=True) df.execute() print(mean_x.get(), std_y.get(), sum_z.get()) ``` ### Pattern: Virtual Columns for Feature Engineering ```python # No memory overhead — computed on the fly df['age_squared'] = df.age ** 2 df['full_name'] = df.first_name + ' ' + df.last_name df['is_adult'] = df.age >= 18 ``` ### Pattern: Migrating a Vaex Query to Polars ```python # Vaex: df = vaex.open('big.parquet'); df['z'] = df.x ** 2 + df.y # df[df.x > 0.5].groupby('category').agg({'z': 'mean'}) import polars as pl result = ( pl.scan_parquet('big.parquet') # lazy, like vaex.open .with_columns((pl.col('x') ** 2 + pl.col('y')).alias('z')) # virtual column .filter(pl.col('x') > 0.5) # filter .group_by('category').agg(pl.col('z').mean()) # groupby/agg .collect(engine='streaming') # out-of-core execution ) ``` Polars cannot read Vaex's HDF5 layout directly — export once with `df.export_parquet(...)` or `df.export_arrow(...)`. Several `delay=True` statistics become one `select(...)` in a single lazy query. See `alterlab-polars` for the rest of the Polars API. ## Resources - `references/core_dataframes.md` — DataFrame creation, loading, and basic structure - `references/data_processing.md` — filtering, expressions, aggregations, and transformations - `references/performance.md` — optimization strategies and lazy evaluation - `references/visualization.md` — plotting and interactive visualizations - `references/machine_learning.md` — ML pipelines and model integration - `references/io_operations.md` — file formats and data import/export Part of the AlterLab Academic Skills suite.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.