Claude Skill

alterlab-vaex

Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-c

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download AlterLab-IEU-AlterLab-Academic-Skills-skills_data-science_alterlab-vaex-3ab6d03.zip · 34 KB

Install

skills CLI npx skills add https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/tree/main/skills/data-science/alterlab-vaex
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install alterlab-ieu-alterlab-academic-skills@llmmart
Git git clone https://github.com/AlterLab-IEU/AlterLab-Academic-Skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole alterlab-ieu/alterlab-academic-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Vaex

Overview

Vaex is a Python library for lazy, out-of-core DataFrames: it memory-maps HDF5/Arrow files, evaluates expressions lazily, and computes statistics on N-dimensional grids fast enough for interactive work on billions of rows on a single machine.

Maintenance Status (read first)

Vaex is effectively in minimal-maintenance mode. In the project's "Is Vaex active?" discussion (vaexio/vaex #2363, June 2024), one core developer stated he is no longer involved and the other committed only to "do my best to at least get vaex releases out". Releases have slowed to roughly one per year (vaex-core 4.18 in Oct 2024, 4.19 in Sep 2025), and 4.19 still supports only Python 3.9–3.12 with pandas<3; the 2-D matplotlib plots break on matplotlib ≥ 3.9. Practical consequences:

  • Existing Vaex code: keep it running in a pinned Python 3.12 environment (below), and plan a migration.
  • New projects: default to Polars (alterlab-polars) — pl.scan_parquet(...) plus collect(engine="streaming") covers most larger-than-RAM aggregation work — or Dask (alterlab-dask) for clusters. Recommend Vaex for new work only when its specific strengths (memory-mapped HDF5, fast binned statistics/visualization of huge tables) matter and the user accepts the environment constraints.
  • Data already in large HDF5 files, one machine, no cluster: this skill's job. Polars has no HDF5 reader, so open the file with vaex.open (memory-mapped) and run the aggregations here; mention a one-time Parquet export only as the migration path.

When to Use This Skill

Use this skill when the user wants to:

  • Run, debug, or extend an existing Vaex pipeline (vaex.open, virtual columns, selections, vaex.ml).
  • Open memory-mapped HDF5/Arrow files that exceed RAM and compute fast aggregations or binned statistics.
  • Batch several aggregations into one pass with delay=True / df.execute().
  • Convert large CSV to HDF5/Arrow/Parquet, or port a Vaex workflow to Polars.

Does NOT Trigger

Scenario Use Instead
New larger-than-RAM analytics project with no existing Vaex code whose data is Parquet/CSV/Arrow (not HDF5), or Python 3.13+ / pandas 3 environments alterlab-polars
Data fits in memory and the goal is the fastest in-memory groupby/joins alterlab-polars
Multi-node cluster execution, task graphs, or a distributed scheduler dashboard alterlab-dask
Chunked N-D scientific arrays (not tables) on disk or cloud object stores alterlab-zarr

Hand-off rule. When the request matches a row above, open your answer by naming that skill by its exact id (for example alterlab-polars) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work.

Installation

# Dedicated environment — vaex 4.19 has no wheels for Python 3.13+ and requires pandas < 3
uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex "vaex==4.19.*"
uv pip install --python .venv-vaex "matplotlib<3.9"   # only if you need df.viz.heatmap
uv run --python .venv-vaex python analysis.py
# One-off alternative with no venv: uv run --python 3.12 --with "vaex==4.19.*" --with "matplotlib<3.9" python analysis.py

Core Capabilities

Vaex provides six capability areas, each documented in references/:

1. DataFrames and Data Loading

Open HDF5/Arrow/Parquet/CSV files, convert from pandas/NumPy/Arrow, and inspect structure — references/core_dataframes.md.

2. Data Processing and Manipulation

Filtering, named selections, virtual columns and expressions, groupby/aggregations, strings, datetimes, and missing data — references/data_processing.md.

3. Performance and Optimization

Lazy evaluation, delay=True batching, materialization, caching, and async execution — references/performance.md.

4. Data Visualization

Binned histograms and heatmaps of huge tables via the df.viz accessor, selections, and widgets — references/visualization.md.

5. Machine Learning Integration

vaex.ml scalers, encoders, PCA, k-means, and scikit-learn/XGBoost/LightGBM/CatBoost wrappers — references/machine_learning.md. The booster and scikit-learn wrappers materialize the training features in memory; only the transformers and prediction columns stay lazy.

6. I/O Operations

Format recommendations, export strategies, Arrow interop, chunked CSV conversion, and remote data — references/io_operations.md.

Quick Start Pattern

import vaex

# 1. Open or create a DataFrame
df = vaex.open('large_file.hdf5')  # or .arrow, .parquet; CSV is read (and optionally converted)
# OR
df = vaex.from_pandas(pandas_df)

# 2. Explore
print(df)          # first/last rows and column info
df.describe()      # statistical summary

# 3. Virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y

# 4. Filter
df_filtered = df[df.age > 25]

# 5. Statistics (lazy, computed in passes over the data)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})

# 6. Visualize (df.plot1d / df.plot are deprecated aliases of these)
df.viz.histogram(df.x, limits=[0, 100])
df.viz.heatmap(df.x, df.y, limits='99.7%')   # needs matplotlib < 3.9

# 7. Export
df.export_hdf5('output.hdf5')

Best Practices

  1. Use HDF5 or Apache Arrow for memory-mapped access to large datasets; Parquet is the most portable choice if a Polars migration is likely.
  2. Prefer virtual columns to materializing data.
  3. Batch operations with delay=True when computing several statistics.
  4. Convert CSV once to HDF5/Arrow/Parquet rather than re-reading it.
  5. Check memory with df.byte_size() and inspect data with df.describe() before optimizing.
  6. Pin the environment (Python 3.12, vaex==4.19.*) and record it with results so analyses stay reproducible.

Common Patterns

Pattern: Converting Large CSV to HDF5

import vaex

# convert=True streams the CSV in chunks to an HDF5 file next to it
df = vaex.from_csv('large_file.csv', convert=True, chunk_size=5_000_000)

# Future loads are memory-mapped and near-instant
df = vaex.open('large_file.csv.hdf5')

Pattern: Efficient Aggregations

# delay=True batches several aggregations into ONE pass over the data.
# Each delayed call returns a promise; df.execute() runs them together.
mean_x = df.mean(df.x, delay=True)
std_y = df.std(df.y, delay=True)
sum_z = df.sum(df.z, delay=True)

df.execute()
print(mean_x.get(), std_y.get(), sum_z.get())

Pattern: Virtual Columns for Feature Engineering

# No memory overhead — computed on the fly
df['age_squared'] = df.age ** 2
df['full_name'] = df.first_name + ' ' + df.last_name
df['is_adult'] = df.age >= 18

Pattern: Migrating a Vaex Query to Polars

# Vaex: df = vaex.open('big.parquet'); df['z'] = df.x ** 2 + df.y
#       df[df.x > 0.5].groupby('category').agg({'z': 'mean'})
import polars as pl

result = (
    pl.scan_parquet('big.parquet')                      # lazy, like vaex.open
    .with_columns((pl.col('x') ** 2 + pl.col('y')).alias('z'))   # virtual column
    .filter(pl.col('x') > 0.5)                          # filter
    .group_by('category').agg(pl.col('z').mean())       # groupby/agg
    .collect(engine='streaming')                        # out-of-core execution
)

Polars cannot read Vaex's HDF5 layout directly — export once with df.export_parquet(...) or df.export_arrow(...). Several delay=True statistics become one select(...) in a single lazy query. See alterlab-polars for the rest of the Polars API.

Resources

  • references/core_dataframes.md — DataFrame creation, loading, and basic structure
  • references/data_processing.md — filtering, expressions, aggregations, and transformations
  • references/performance.md — optimization strategies and lazy evaluation
  • references/visualization.md — plotting and interactive visualizations
  • references/machine_learning.md — ML pipelines and model integration
  • references/io_operations.md — file formats and data import/export

Part of the AlterLab Academic Skills suite.

Files (alterlab-academic-skills)
  • evals
    • evals.json 6.1 KB
      Loading…
  • references
    • core_dataframes.md 7.8 KB
      Loading…
    • data_processing.md 13 KB
      Loading…
    • io_operations.md 15.4 KB
      Loading…
    • machine_learning.md 17.5 KB
      Loading…
    • performance.md 14.1 KB
      Loading…
    • visualization.md 13.9 KB
      Loading…
  • SKILL.md 9.4 KB
    ---
    name: alterlab-vaex
    description: Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-core 4.19 from Sep 2025 supports only Python 3.9–3.12 and pandas < 3), so use it to run, fix, or migrate existing Vaex code; for new larger-than-RAM work prefer Polars lazy/streaming queries (alterlab-polars) or Dask (alterlab-dask). Use when a project already depends on Vaex, when opening memory-mapped HDF5/Arrow files, batching aggregations with delay=True, or porting a Vaex pipeline to Polars. Part of the AlterLab Academic Skills suite.
    license: MIT
    allowed-tools: Read Write Edit Bash(python:*) Bash(uv:*)
    compatibility: No API key required. Runs locally via `uv run --python 3.12 --with "vaex==4.19.*" python script.py` (no install step) or in a dedicated Python 3.12 (or older) environment; requires vaex 4.19.x, which pins pandas < 3 and has no wheels for Python 3.13+. 2-D plots (`df.viz.heatmap`) also need matplotlib < 3.9.
    metadata:
        skill-author: AlterLab
        version: "1.1.0"
        last_updated: "2026-09-23"
    ---
    
    # Vaex
    
    ## Overview
    
    Vaex is a Python library for lazy, out-of-core DataFrames: it memory-maps HDF5/Arrow files, evaluates expressions lazily, and computes statistics on N-dimensional grids fast enough for interactive work on billions of rows on a single machine.
    
    ## Maintenance Status (read first)
    
    Vaex is effectively in minimal-maintenance mode. In the project's "Is Vaex active?" discussion (vaexio/vaex #2363, June 2024), one core developer stated he is no longer involved and the other committed only to "do my best to at least get vaex releases out". Releases have slowed to roughly one per year (vaex-core 4.18 in Oct 2024, 4.19 in Sep 2025), and 4.19 still supports only Python 3.9–3.12 with `pandas<3`; the 2-D matplotlib plots break on matplotlib ≥ 3.9. Practical consequences:
    
    - **Existing Vaex code**: keep it running in a pinned Python 3.12 environment (below), and plan a migration.
    - **New projects**: default to Polars (`alterlab-polars`) — `pl.scan_parquet(...)` plus `collect(engine="streaming")` covers most larger-than-RAM aggregation work — or Dask (`alterlab-dask`) for clusters. Recommend Vaex for new work only when its specific strengths (memory-mapped HDF5, fast binned statistics/visualization of huge tables) matter and the user accepts the environment constraints.
    - **Data already in large HDF5 files, one machine, no cluster**: this skill's job. Polars has no HDF5 reader, so open the file with `vaex.open` (memory-mapped) and run the aggregations here; mention a one-time Parquet export only as the migration path.
    
    ## When to Use This Skill
    
    Use this skill when the user wants to:
    - Run, debug, or extend an existing Vaex pipeline (`vaex.open`, virtual columns, selections, `vaex.ml`).
    - Open memory-mapped HDF5/Arrow files that exceed RAM and compute fast aggregations or binned statistics.
    - Batch several aggregations into one pass with `delay=True` / `df.execute()`.
    - Convert large CSV to HDF5/Arrow/Parquet, or port a Vaex workflow to Polars.
    
    ### Does NOT Trigger
    
    | Scenario | Use Instead |
    |----------|-------------|
    | New larger-than-RAM analytics project with no existing Vaex code whose data is Parquet/CSV/Arrow (not HDF5), or Python 3.13+ / pandas 3 environments | `alterlab-polars` |
    | Data fits in memory and the goal is the fastest in-memory groupby/joins | `alterlab-polars` |
    | Multi-node cluster execution, task graphs, or a distributed scheduler dashboard | `alterlab-dask` |
    | Chunked N-D scientific arrays (not tables) on disk or cloud object stores | `alterlab-zarr` |
    
    **Hand-off rule.** When the request matches a row above, open your answer by naming that skill by its exact id (for example `alterlab-polars`) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work.
    
    ## Installation
    
    ```bash
    # Dedicated environment — vaex 4.19 has no wheels for Python 3.13+ and requires pandas < 3
    uv venv --python 3.12 .venv-vaex
    uv pip install --python .venv-vaex "vaex==4.19.*"
    uv pip install --python .venv-vaex "matplotlib<3.9"   # only if you need df.viz.heatmap
    uv run --python .venv-vaex python analysis.py
    # One-off alternative with no venv: uv run --python 3.12 --with "vaex==4.19.*" --with "matplotlib<3.9" python analysis.py
    ```
    
    ## Core Capabilities
    
    Vaex provides six capability areas, each documented in `references/`:
    
    ### 1. DataFrames and Data Loading
    Open HDF5/Arrow/Parquet/CSV files, convert from pandas/NumPy/Arrow, and inspect structure — `references/core_dataframes.md`.
    
    ### 2. Data Processing and Manipulation
    Filtering, named selections, virtual columns and expressions, groupby/aggregations, strings, datetimes, and missing data — `references/data_processing.md`.
    
    ### 3. Performance and Optimization
    Lazy evaluation, `delay=True` batching, materialization, caching, and async execution — `references/performance.md`.
    
    ### 4. Data Visualization
    Binned histograms and heatmaps of huge tables via the `df.viz` accessor, selections, and widgets — `references/visualization.md`.
    
    ### 5. Machine Learning Integration
    `vaex.ml` scalers, encoders, PCA, k-means, and scikit-learn/XGBoost/LightGBM/CatBoost wrappers — `references/machine_learning.md`. The booster and scikit-learn wrappers materialize the training features in memory; only the transformers and prediction columns stay lazy.
    
    ### 6. I/O Operations
    Format recommendations, export strategies, Arrow interop, chunked CSV conversion, and remote data — `references/io_operations.md`.
    
    ## Quick Start Pattern
    
    ```python
    import vaex
    
    # 1. Open or create a DataFrame
    df = vaex.open('large_file.hdf5')  # or .arrow, .parquet; CSV is read (and optionally converted)
    # OR
    df = vaex.from_pandas(pandas_df)
    
    # 2. Explore
    print(df)          # first/last rows and column info
    df.describe()      # statistical summary
    
    # 3. Virtual columns (no memory overhead)
    df['new_column'] = df.x ** 2 + df.y
    
    # 4. Filter
    df_filtered = df[df.age > 25]
    
    # 5. Statistics (lazy, computed in passes over the data)
    mean_val = df.x.mean()
    stats = df.groupby('category').agg({'value': 'sum'})
    
    # 6. Visualize (df.plot1d / df.plot are deprecated aliases of these)
    df.viz.histogram(df.x, limits=[0, 100])
    df.viz.heatmap(df.x, df.y, limits='99.7%')   # needs matplotlib < 3.9
    
    # 7. Export
    df.export_hdf5('output.hdf5')
    ```
    
    ## Best Practices
    
    1. **Use HDF5 or Apache Arrow** for memory-mapped access to large datasets; Parquet is the most portable choice if a Polars migration is likely.
    2. **Prefer virtual columns** to materializing data.
    3. **Batch operations** with `delay=True` when computing several statistics.
    4. **Convert CSV once** to HDF5/Arrow/Parquet rather than re-reading it.
    5. **Check memory with `df.byte_size()`** and inspect data with `df.describe()` before optimizing.
    6. **Pin the environment** (Python 3.12, `vaex==4.19.*`) and record it with results so analyses stay reproducible.
    
    ## Common Patterns
    
    ### Pattern: Converting Large CSV to HDF5
    ```python
    import vaex
    
    # convert=True streams the CSV in chunks to an HDF5 file next to it
    df = vaex.from_csv('large_file.csv', convert=True, chunk_size=5_000_000)
    
    # Future loads are memory-mapped and near-instant
    df = vaex.open('large_file.csv.hdf5')
    ```
    
    ### Pattern: Efficient Aggregations
    ```python
    # delay=True batches several aggregations into ONE pass over the data.
    # Each delayed call returns a promise; df.execute() runs them together.
    mean_x = df.mean(df.x, delay=True)
    std_y = df.std(df.y, delay=True)
    sum_z = df.sum(df.z, delay=True)
    
    df.execute()
    print(mean_x.get(), std_y.get(), sum_z.get())
    ```
    
    ### Pattern: Virtual Columns for Feature Engineering
    ```python
    # No memory overhead — computed on the fly
    df['age_squared'] = df.age ** 2
    df['full_name'] = df.first_name + ' ' + df.last_name
    df['is_adult'] = df.age >= 18
    ```
    
    ### Pattern: Migrating a Vaex Query to Polars
    ```python
    # Vaex: df = vaex.open('big.parquet'); df['z'] = df.x ** 2 + df.y
    #       df[df.x > 0.5].groupby('category').agg({'z': 'mean'})
    import polars as pl
    
    result = (
        pl.scan_parquet('big.parquet')                      # lazy, like vaex.open
        .with_columns((pl.col('x') ** 2 + pl.col('y')).alias('z'))   # virtual column
        .filter(pl.col('x') > 0.5)                          # filter
        .group_by('category').agg(pl.col('z').mean())       # groupby/agg
        .collect(engine='streaming')                        # out-of-core execution
    )
    ```
    Polars cannot read Vaex's HDF5 layout directly — export once with `df.export_parquet(...)` or `df.export_arrow(...)`. Several `delay=True` statistics become one `select(...)` in a single lazy query. See `alterlab-polars` for the rest of the Polars API.
    
    ## Resources
    
    - `references/core_dataframes.md` — DataFrame creation, loading, and basic structure
    - `references/data_processing.md` — filtering, expressions, aggregations, and transformations
    - `references/performance.md` — optimization strategies and lazy evaluation
    - `references/visualization.md` — plotting and interactive visualizations
    - `references/machine_learning.md` — ML pipelines and model integration
    - `references/io_operations.md` — file formats and data import/export
    
    Part of the AlterLab Academic Skills suite.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.