{
  "skill": "alterlab-vaex",
  "evals": [
    {
      "id": "out-of-core-aggregations-single-machine",
      "prompt": "I have a single 300 GB HDF5 file of tabular sensor readings on one workstation with 48 GB of RAM. I want fast groupby aggregations and overall statistics without loading it all into memory, and no cluster to set up.",
      "expected_output": "Invokes alterlab-vaex for out-of-core single-machine analytics: vaex.open on the HDF5 file (memory-mapped, instant), df.describe()/df.x.mean() and df.groupby(...).agg(...) computed lazily over the full dataset without loading into RAM. Leverages lazy evaluation and stays on one machine with no cluster. Notes that Vaex is in minimal-maintenance mode and needs a pinned Python <= 3.12 / pandas < 3 environment.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "vaex.open" },
        { "type": "behavior", "value": "Uses vaex.open and lazy out-of-core aggregations on a single machine, without loading the dataset into RAM or invoking a cluster." }
      ]
    },
    {
      "id": "virtual-columns-feature-engineering",
      "prompt": "My billion-row Vaex DataFrame is already near my memory limit. I need to add several engineered columns (age squared, a full-name concatenation, an is_adult flag) without blowing up memory by materializing copies.",
      "expected_output": "Invokes alterlab-vaex using virtual columns: assigns df['age_squared'] = df.age ** 2, df['full_name'] = df.first_name + ' ' + df.last_name, df['is_adult'] = df.age >= 18 as lazily-evaluated expressions computed on the fly with no memory overhead, rather than materializing new arrays.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "virtual column" },
        { "type": "behavior", "value": "Adds engineered columns as Vaex virtual columns (lazy expressions) instead of materializing data, to avoid memory overhead." }
      ]
    },
    {
      "id": "csv-to-hdf5-conversion-bigdata-viz",
      "prompt": "I keep re-reading a huge multi-gigabyte CSV and it's painfully slow, and I also want a density heatmap of two columns across the whole billion-row dataset. What's the efficient workflow?",
      "expected_output": "Invokes alterlab-vaex for I/O and big-data visualization: convert the slow CSV once via vaex.from_csv (e.g. convert=True) or export_hdf5 so future vaex.open loads are near-instant, and produce a 2D density heatmap with df.viz.heatmap(df.x, df.y, limits='99.7%') over the full dataset (df.plot is a deprecated alias; 2-D plots need matplotlib < 3.9). Recommends HDF5/Arrow over CSV for repeated access.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "export_hdf5" },
        { "type": "behavior", "value": "Recommends converting CSV to HDF5 (or Arrow) for fast repeated access and uses Vaex's built-in heatmap/2D plotting for the big-data visualization." }
      ]
    },
    {
      "id": "batched-delayed-aggregations",
      "prompt": "I want to compute the mean of one column, the std of another, and the sum of a third over my huge Vaex DataFrame, but I'd rather not pass over the data three separate times. Can these run in a single scan?",
      "expected_output": "Invokes alterlab-vaex for batched lazy aggregations: requests each statistic with delay=True (df.mean(df.x, delay=True), df.std(df.y, delay=True), df.sum(df.z, delay=True)), calls df.execute() once to run them in a single pass, then collects each result with .get() rather than scanning the data three times.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "delay=True" },
        { "type": "behavior", "value": "Batches the multiple aggregations with delay=True and triggers a single pass with df.execute(), retrieving each result via .get(), so they share one scan over the data." }
      ]
    },
    {
      "id": "near-miss-dask",
      "prompt": "My dataset is 10 TB across hundreds of Parquet files and I need to distribute the processing across a multi-node cluster with a monitoring dashboard and custom parallel task dependencies.",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-dask. The user needs distributed multi-machine cluster execution with a scheduling dashboard and custom task-graph dependencies, which is Dask's distributed model, whereas Vaex is single-machine out-of-core.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-dask" }
      ]
    },
    {
      "id": "near-miss-polars",
      "prompt": "I have a 4 GB Parquet file that fits easily in my 48 GB RAM and I just want the fastest in-memory groupby and joins with a modern lazy query API. No out-of-core needed.",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-polars. The data fits in memory and the user explicitly wants maximum in-memory speed with a modern DataFrame/lazy-query API and no out-of-core processing, which is polars' niche rather than Vaex's out-of-core focus.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-polars" }
      ]
    },
    {
      "id": "near-miss-polars-new-out-of-core-project",
      "prompt": "I'm starting a new analysis project on Python 3.13 with pandas 3. I have about 200 GB of Parquet files and a 32 GB laptop, and I need out-of-core filtering and groupby aggregations. Which DataFrame library should I build this on?",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-polars. This is a new larger-than-RAM project on Python 3.13 with pandas 3, where Vaex cannot be installed (vaex-core 4.19 supports only Python <= 3.12 and pandas < 3) and is in minimal-maintenance mode; Polars lazy scans (pl.scan_parquet) collected with the streaming engine are the recommended out-of-core route.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-polars" }
      ]
    }
  ]
}

# Core DataFrames and Data Loading

This reference covers Vaex DataFrame basics, loading data from various sources, and understanding the DataFrame structure.

## DataFrame Fundamentals

A Vaex DataFrame is the central data structure for working with large tabular datasets. Unlike pandas, Vaex DataFrames:
- Use **lazy evaluation** - operations are not executed until needed
- Work **out-of-core** - data doesn't need to fit in RAM
- Support **virtual columns** - computed columns with no memory overhead
- Enable **billion-row-per-second** processing through optimized C++ backend

## Opening Existing Files

### Primary Method: `vaex.open()`

The most common way to load data:

```python
import vaex

# Works with multiple formats
df = vaex.open('data.hdf5')     # HDF5 (recommended)
df = vaex.open('data.arrow')    # Apache Arrow (recommended)
df = vaex.open('data.parquet')  # Parquet
df = vaex.open('data.csv')      # CSV (slower for large files)
df = vaex.open('data.fits')     # FITS (astronomy)

# Can open multiple files as one DataFrame
df = vaex.open('data_*.hdf5')   # Wildcards supported
```

**Key characteristics:**
- **Instant for HDF5/Arrow** - Memory-maps files, no loading time
- **Handles large CSVs** - Automatically chunks large CSV files
- **Returns immediately** - Lazy evaluation means no computation until needed

### Format-Specific Loaders

```python
# CSV with options
df = vaex.from_csv(
    'large_file.csv',
    chunk_size=5_000_000,      # Process in chunks
    convert=True,               # Convert to HDF5 automatically
    copy_index=False            # Don't copy pandas index if present
)

# Apache Arrow
df = vaex.open('data.arrow')    # Native support, very fast

# HDF5 (optimal format)
df = vaex.open('data.hdf5')     # Instant loading via memory mapping
```

## Creating DataFrames from Other Sources

### From Pandas

```python
import pandas as pd
import vaex

# Convert pandas DataFrame
pdf = pd.read_csv('data.csv')
df = vaex.from_pandas(pdf, copy_index=False)

# Warning: This loads entire pandas DataFrame into memory
# For large data, prefer vaex.from_csv() directly
```

### From NumPy Arrays

```python
import numpy as np
import vaex

# Single array
x = np.random.rand(1_000_000)
df = vaex.from_arrays(x=x)

# Multiple arrays
x = np.random.rand(1_000_000)
y = np.random.rand(1_000_000)
df = vaex.from_arrays(x=x, y=y)
```

### From Dictionaries

```python
import vaex

# Dictionary of lists/arrays
data = {
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'salary': [50000, 60000, 70000]
}
df = vaex.from_dict(data)
```

### From Arrow Tables

```python
import pyarrow as pa
import vaex

# From Arrow Table
arrow_table = pa.table({
    'x': [1, 2, 3],
    'y': [4, 5, 6]
})
df = vaex.from_arrow_table(arrow_table)
```

## Example Datasets

Vaex provides built-in example datasets for testing:

```python
import vaex

# NYC taxi dataset (~1GB, 11 million rows)
df = vaex.example()

# Smaller datasets
df = vaex.datasets.titanic()
df = vaex.datasets.iris()
```

## Inspecting DataFrames

### Basic Information

```python
# Display first and last rows
print(df)

# Shape (rows, columns)
print(df.shape)  # Returns (row_count, column_count)
print(len(df))   # Row count

# Column names
print(df.columns)
print(df.column_names)

# Data types
print(df.dtypes)

# Memory usage (for materialized columns)
df.byte_size()
```

### Statistical Summary

```python
# Quick statistics for all numeric columns
df.describe()

# Single column statistics
df.x.mean()
df.x.std()
df.x.min()
df.x.max()
df.x.sum()
df.x.count()

# Quantiles
df.percentile_approx(df.x, 50)            # Median (approximate)
df.percentile_approx(df.x, [25, 50, 75])  # Multiple percentiles
```

### Viewing Data

```python
# First/last rows (returns pandas DataFrame)
df.head(10)
df.tail(10)

# Random sample
df.sample(n=100)

# Convert to pandas (careful with large data!)
pdf = df.to_pandas_df()

# Convert specific columns only
pdf = df[['x', 'y']].to_pandas_df()
```

## DataFrame Structure

### Columns

```python
# Access columns as expressions
x_column = df.x
y_column = df['y']

# Column operations return expressions (lazy)
sum_column = df.x + df.y    # Not computed yet

# List all columns
print(df.get_column_names())

# Check column types
print(df.dtypes)

# Virtual vs materialized columns
print(df.get_column_names(virtual=False))  # Materialized only
print(df.get_column_names(virtual=True))   # All columns
```

### Rows

```python
# Row count
row_count = len(df)
row_count = df.count()

# Single row (a list of values in column order)
row = df[0]
print(dict(zip(df.get_column_names(), row)))

# Note: Iterating over rows is NOT recommended in Vaex
# Use vectorized operations instead
```

## Working with Expressions

Expressions are Vaex's way of representing computations that haven't been executed yet:

```python
# Create expressions (no computation)
expr = df.x ** 2 + df.y

# Expressions can be used in many contexts
mean_of_expr = expr.mean()          # Still lazy
df['new_col'] = expr                # Virtual column
filtered = df[expr > 10]            # Selection

# Force evaluation
result = expr.values  # Returns NumPy array (use carefully!)
```

## DataFrame Operations

### Copying

```python
# Shallow copy (shares data)
df_copy = df.copy()

# Deep copy (independent data)
df_deep = df.copy(deep=True)
```

### Trimming/Slicing

```python
# Select row range
df_subset = df[1000:2000]      # Rows 1000-2000
df_subset = df[:1000]          # First 1000 rows
df_subset = df[-1000:]         # Last 1000 rows

# Note: This creates a view, not a copy (efficient)
```

### Concatenating

```python
# Vertical concatenation (combine rows)
df_combined = vaex.concat([df1, df2, df3])

# Horizontal concatenation (combine columns)
# Use join or simply assign columns
df['new_col'] = other_df.some_column
```

## Best Practices

1. **Prefer HDF5 or Arrow formats** - Instant loading, optimal performance
2. **Convert large CSVs to HDF5** - One-time conversion for repeated use
3. **Avoid `.to_pandas_df()` on large data** - Defeats Vaex's purpose
4. **Use expressions instead of `.values`** - Keep operations lazy
5. **Check data types** - Ensure numeric columns aren't string type
6. **Use virtual columns** - Zero memory overhead for derived data

## Common Patterns

### Pattern: One-time CSV to HDF5 Conversion

```python
# Initial conversion (do once)
df = vaex.from_csv('large_data.csv', convert='large_data.hdf5')

# Future loads (instant)
df = vaex.open('large_data.hdf5')
```

### Pattern: Inspecting Large Datasets

```python
import vaex

df = vaex.open('large_file.hdf5')

# Quick overview
print(df)                    # First/last rows
print(df.shape)             # Dimensions
print(df.describe())        # Statistics

# Sample for detailed inspection
sample = df.sample(1000).to_pandas_df()
print(sample.head())
```

### Pattern: Loading Multiple Files

```python
# Load multiple files as one DataFrame
df = vaex.open('data_part*.hdf5')

# Or explicitly concatenate
df1 = vaex.open('data_2020.hdf5')
df2 = vaex.open('data_2021.hdf5')
df_all = vaex.concat([df1, df2])
```

## Common Issues and Solutions

### Issue: CSV Loading is Slow

```python
# Solution: Convert to HDF5 first
df = vaex.from_csv('large.csv', convert='large.hdf5')
# Future loads: df = vaex.open('large.hdf5')
```

### Issue: Column Shows as String Type

```python
# Check type
print(df.dtypes)

# Convert to numeric (creates virtual column)
df['age_numeric'] = df.age.astype('int64')
```

### Issue: Out of Memory on Small Operations

```python
# Likely using .values or .to_pandas_df()
# Solution: Use lazy operations

# Bad (loads into memory)
array = df.x.values

# Good (stays lazy)
mean = df.x.mean()
filtered = df[df.x > 10]
```

## Related Resources

- For data manipulation and filtering: See `data_processing.md`
- For performance optimization: See `performance.md`
- For file format details: See `io_operations.md`

# Data Processing and Manipulation

This reference covers filtering, selections, virtual columns, expressions, aggregations, groupby operations, and data transformations in Vaex.

## Filtering and Selections

Vaex uses boolean expressions to filter data efficiently without copying:

### Basic Filtering

```python
# Simple filter
df_filtered = df[df.age > 25]

# Multiple conditions
df_filtered = df[(df.age > 25) & (df.salary > 50000)]
df_filtered = df[(df.category == 'A') | (df.category == 'B')]

# Negation
df_filtered = df[~(df.age < 18)]
```

### Selection Objects

Vaex can maintain multiple named selections simultaneously:

```python
# Create named selection
df.select(df.age > 30, name='adults')
df.select(df.salary > 100000, name='high_earners')

# Use selection in operations
mean_age_adults = df.mean(df.age, selection='adults')
count_high_earners = df.count(selection='high_earners')

# Combine selections
df.select((df.age > 30) & (df.salary > 100000), name='adult_high_earners')

# Check whether a named selection is set
print(df.has_selection('adults'))

# Clear a named selection
df.select_nothing(name='adults')
```

### Advanced Filtering

```python
# String matching
df_filtered = df[df.name.str.contains('John')]
df_filtered = df[df.name.str.startswith('A')]
df_filtered = df[df.email.str.endswith('@gmail.com')]

# Null/missing value filtering
df_filtered = df[df.age.isna()]      # Keep missing
df_filtered = df[df.age.notna()]     # Remove missing

# Value membership
df_filtered = df[df.category.isin(['A', 'B', 'C'])]

# Range filtering (expressions have no .between(); combine comparisons)
df_filtered = df[(df.age >= 25) & (df.age <= 65)]
```

## Virtual Columns and Expressions

Virtual columns are computed on-the-fly with zero memory overhead:

### Creating Virtual Columns

```python
# Arithmetic operations
df['total'] = df.price * df.quantity
df['price_squared'] = df.price ** 2

# Mathematical functions
df['log_price'] = df.price.log()
df['sqrt_value'] = df.value.sqrt()
df['abs_diff'] = (df.x - df.y).abs()

# Conditional logic
df['is_adult'] = df.age >= 18
df['category'] = (df.score > 80).where('A', 'B')  # If-then-else
```

### Expression Methods

```python
# Mathematical
df.x.abs()          # Absolute value
df.x.sqrt()         # Square root
df.x.log()          # Natural log
df.x.log10()        # Base-10 log
df.x.exp()          # Exponential

# Trigonometric
df.angle.sin()
df.angle.cos()
df.angle.tan()
df.angle.arcsin()

# Rounding
df.x.round(2)       # Round to 2 decimals
df.x // 1           # Round down (expressions have no .floor())
-(-df.x // 1)       # Round up (no .ceil())

# Type conversion
df.x.astype('int64')
df.x.astype('float32')
df.x.astype('str')
```

### Conditional Expressions

```python
# where() method: condition.where(true_value, false_value)
df['status'] = (df.age >= 18).where('adult', 'minor')

# Multiple conditions with nested where
df['grade'] = (df.score >= 90).where('A',
              (df.score >= 80).where('B',
              (df.score >= 70).where('C', 'F')))

# Binning with digitize + map (0: <18, 1: 18-64, 2: >=65)
df['age_group'] = df.age.digitize([18, 65]).map({0: 'minor', 1: 'adult', 2: 'senior'})
```

## String Operations

Access string methods via the `.str` accessor:

### Basic String Methods

```python
# Case conversion
df['upper_name'] = df.name.str.upper()
df['lower_name'] = df.name.str.lower()
df['title_name'] = df.name.str.title()

# Trimming
df['trimmed'] = df.text.str.strip()
df['ltrimmed'] = df.text.str.lstrip()
df['rtrimmed'] = df.text.str.rstrip()

# Searching
df['has_john'] = df.name.str.contains('John')
df['starts_with_a'] = df.name.str.startswith('A')
df['ends_with_com'] = df.email.str.endswith('.com')

# Slicing
df['first_char'] = df.name.str.slice(0, 1)
df['last_three'] = df.name.str.slice(-3, None)

# Length
df['name_length'] = df.name.str.len()
```

### Advanced String Operations

```python
# Replacing
df['clean_text'] = df.text.str.replace('bad', 'good')

# First token. str.split() returns a list column, and indexing it with [0]
# raises NotImplementedError in vaex 4.19, so strip everything after the first space.
df['first_name'] = df.full_name.str.replace(r'\s.*$', '', regex=True)

# Concatenation
df['full_name'] = df.first_name + ' ' + df.last_name

# Padding
df['padded'] = df.code.str.pad(10, '0', 'left')  # Zero-padding
```

## DateTime Operations

Access datetime methods via the `.dt` accessor:

### DateTime Properties

```python
# Parsing strings to datetime
df['date_parsed'] = df.date_string.astype('datetime64')

# Extracting components
df['year'] = df.timestamp.dt.year
df['month'] = df.timestamp.dt.month
df['day'] = df.timestamp.dt.day
df['hour'] = df.timestamp.dt.hour
df['minute'] = df.timestamp.dt.minute
df['second'] = df.timestamp.dt.second

# Day of week
df['weekday'] = df.timestamp.dt.dayofweek  # 0=Monday
df['day_name'] = df.timestamp.dt.day_name  # 'Monday', 'Tuesday', ...

# Date arithmetic
df['tomorrow'] = df.date + pd.Timedelta(days=1)
df['next_week'] = df.date + pd.Timedelta(weeks=1)
```

## Aggregations

Vaex performs aggregations efficiently across billions of rows:

### Basic Aggregations

```python
# Single column
mean_age = df.age.mean()
std_age = df.age.std()
min_age = df.age.min()
max_age = df.age.max()
sum_sales = df.sales.sum()
count_rows = df.count()

# With selections
mean_adult_age = df.age.mean(selection='adults')

# Multiple at once with delay (single pass): build promises, then df.execute()
mean = df.mean(df.age, delay=True)
std = df.std(df.age, delay=True)
df.execute()
mean_val, std_val = mean.get(), std.get()
```

### Available Aggregation Functions

```python
# Central tendency
df.x.mean()
df.median_approx(df.x)  # Approximate median (fast; a DataFrame method)

# Dispersion
df.x.std()           # Standard deviation
df.x.var()           # Variance
df.x.min()
df.x.max()
df.x.minmax()        # Both min and max

# Count
df.count()           # Total rows
df.x.count()         # Non-missing values

# Sum
df.x.sum()

# Percentiles (approximate, binned; expressions have no .quantile())
df.percentile_approx(df.x, 50)          # Median
df.percentile_approx(df.x, [25, 75])    # Quartiles

# Correlation
df.correlation(df.x, df.y)
df.covar(df.x, df.y)

# Higher moments
df.x.kurtosis()
df.x.skew()

# Unique values
df.x.nunique()       # Count unique
df.x.unique()        # Get unique values (returns array)
```

## GroupBy Operations

Group data and compute aggregations per group:

### Basic GroupBy

```python
# Single column groupby
grouped = df.groupby('category')

# Aggregation
result = grouped.agg({'sales': 'sum'})
result = grouped.agg({'sales': 'sum', 'quantity': 'mean'})

# Multiple aggregations on same column
result = grouped.agg({
    'sales': ['sum', 'mean', 'std'],
    'quantity': 'sum'
})
```

### Advanced GroupBy

```python
# Multiple grouping columns
result = df.groupby(['category', 'region']).agg({
    'sales': 'sum',
    'quantity': 'mean'
})

# Custom aggregation functions
result = df.groupby('category').agg({
    'sales': lambda x: x.max() - x.min()
})

# Available aggregation functions
# 'sum', 'mean', 'std', 'min', 'max', 'count', 'first', 'last'
```

### GroupBy with Binning

```python
# Bin a continuous variable, then group on the bin index.
# digitize() returns the bin each value falls into (a virtual column).
df['sales_bin'] = df.amount.digitize([0, 25, 50, 75, 100])
result = df.groupby('sales_bin').agg({'sales': 'sum'})

# Datetime binning: group on an extracted datetime component
result = df.groupby(df.timestamp.dt.year).agg({
    'sales': 'sum'
})
```

## Binning and Discretization

Create bins from continuous variables:

### Simple Binning

```python
# Create bins
df['age_bin'] = df.age.digitize([18, 30, 50, 65, 100])

# Labeled bins
bins = [0, 18, 30, 50, 65, 100]
labels = ['child', 'young_adult', 'adult', 'middle_age', 'senior']
df['age_group'] = df.age.digitize(bins)
# Note: Apply labels using where() or mapping
```

### Statistical Binning

```python
import numpy as np

# Equal-width bins (compute the edges once, then bin)
edges = np.linspace(df.value.min(), df.value.max(), 11)  # 10 bins -> 11 edges
df['value_bin'] = df.value.digitize(edges)

# Quantile-based bins
quantiles = df.percentile_approx(df.value, [25, 50, 75])
df['value_quartile'] = df.value.digitize(quantiles)
```

## Multi-dimensional Aggregations

Compute statistics on grids:

```python
# 2D histogram/heatmap data
counts = df.count(binby=[df.x, df.y], limits=[[0, 10], [0, 10]], shape=(100, 100))

# Mean on a grid
mean_z = df.mean(df.z, binby=[df.x, df.y], limits=[[0, 10], [0, 10]], shape=(50, 50))

# Multiple statistics on grid in a single pass
stats = df.mean(df.z, binby=[df.x, df.y], shape=(50, 50), delay=True)
counts = df.count(binby=[df.x, df.y], shape=(50, 50), delay=True)
df.execute()
mean_grid, count_grid = stats.get(), counts.get()
```

## Handling Missing Data

Work with missing, null, and NaN values:

### Detecting Missing Data

```python
# Check for missing
df['age_missing'] = df.age.isna()
df['age_present'] = df.age.notna()

# Count missing
missing_count = df.age.isna().sum()
missing_pct = df.age.isna().mean() * 100
```

### Handling Missing Data

```python
# Filter out missing
df_clean = df[df.age.notna()]

# Fill missing with a value (creates a virtual column)
df['age_filled'] = df.age.fillna(0)
df['age_filled'] = df.age.fillna(df.age.mean())

# Note: out-of-core, order-dependent fills (forward/backward) are not a native
# Vaex operation — compute the fill on a materialized/pandas slice if you need it.
```

### Missing Data Types in Vaex

Vaex distinguishes between:
- **NaN** - IEEE floating point Not-a-Number
- **NA** - Arrow null type
- **Missing** - General term for absent data

```python
# Check which missing type
df.is_masked('column_name')  # True if uses Arrow null (NA)
```

## Sorting

```python
# Sort by single column
df_sorted = df.sort('age')
df_sorted = df.sort('age', ascending=False)

# Sort by multiple columns
df_sorted = df.sort(['category', 'age'])

# Note: Sorting materializes a new column with indices
# For very large datasets, consider if sorting is necessary
```

## Joining DataFrames

Combine DataFrames based on keys:

```python
# Inner join
df_joined = df1.join(df2, on='key_column')

# Left join
df_joined = df1.join(df2, on='key_column', how='left')

# Join on different column names
df_joined = df1.join(
    df2,
    left_on='id',
    right_on='user_id',
    how='left'
)

# Multiple key columns
df_joined = df1.join(df2, on=['key1', 'key2'])
```

## Adding and Removing Columns

### Adding Columns

```python
# Virtual column (no memory)
df['new_col'] = df.x + df.y

# From external array (must match length)
import numpy as np
new_data = np.random.rand(len(df))
df['random'] = new_data

# Constant value (a bare scalar raises ValueError; vconstant is a lazy, zero-memory column)
df['constant'] = vaex.vconstant(42, len(df))
```

### Removing Columns

```python
# Drop single column
df = df.drop('column_name')

# Drop multiple columns
df = df.drop(['col1', 'col2', 'col3'])

# Select specific columns (drop others)
df = df[['col1', 'col2', 'col3']]
```

### Renaming Columns

```python
# Rename single column
df = df.rename('old_name', 'new_name')

# Rename multiple columns
df = df.rename({
    'old_name1': 'new_name1',
    'old_name2': 'new_name2'
})
```

## Common Patterns

### Pattern: Complex Feature Engineering

```python
# Multiple derived features
df['log_price'] = df.price.log()
df['price_per_unit'] = df.price / df.quantity
df['is_discount'] = df.discount > 0
df['price_category'] = (df.price > 100).where('expensive', 'affordable')
df['revenue'] = df.price * df.quantity * (1 - df.discount)
```

### Pattern: Text Cleaning

```python
# Clean and standardize text
df['email_clean'] = df.email.str.lower().str.strip()
df['has_valid_email'] = df.email_clean.str.contains('@')
df['domain'] = df.email_clean.str.replace(r'^.*@', '', regex=True)  # split('@')[1] is not implemented
```

### Pattern: Time-based Analysis

```python
# Extract temporal features
df['year'] = df.timestamp.dt.year
df['month'] = df.timestamp.dt.month
df['day_of_week'] = df.timestamp.dt.dayofweek
df['is_weekend'] = df.day_of_week >= 5
df['quarter'] = ((df.month - 1) // 3) + 1
```

### Pattern: Grouped Statistics

```python
# Compute statistics by group
monthly_sales = df.groupby(df.timestamp.dt.month).agg({
    'revenue': ['sum', 'mean', 'count'],
    'quantity': 'sum'
})

# Multiple grouping levels
category_region_sales = df.groupby(['category', 'region']).agg({
    'sales': 'sum',
    'profit': 'mean'
})
```

## Performance Tips

1. **Use virtual columns** - They're computed on-the-fly with no memory cost
2. **Batch operations with delay=True** - Compute multiple aggregations at once
3. **Avoid `.values` or `.to_pandas_df()`** - Keep operations lazy when possible
4. **Use selections** - Multiple named selections are more efficient than creating new DataFrames
5. **Leverage expressions** - They enable query optimization
6. **Minimize sorting** - Sorting is expensive on large datasets

## Related Resources

- For DataFrame creation: See `core_dataframes.md`
- For performance optimization: See `performance.md`
- For visualization: See `visualization.md`
- For ML pipelines: See `machine_learning.md`

# I/O Operations

This reference covers file input/output operations, format conversions, export strategies, and working with various data formats in Vaex.

## Overview

Vaex supports multiple file formats with varying performance characteristics. The choice of format significantly impacts loading speed, memory usage, and overall performance.

**Format recommendations:**
- **HDF5** - Best for most use cases (instant loading, memory-mapped)
- **Apache Arrow** - Best for interoperability (instant loading, columnar)
- **Parquet** - Good for distributed systems (compressed, columnar)
- **CSV** - Avoid for large datasets (slow loading, not memory-mapped)

## Reading Data

### HDF5 Files (Recommended)

```python
import vaex

# Open HDF5 file (instant, memory-mapped)
df = vaex.open('data.hdf5')

# Multiple files as one DataFrame
df = vaex.open('data_part*.hdf5')
df = vaex.open(['data_2020.hdf5', 'data_2021.hdf5', 'data_2022.hdf5'])
```

**Advantages:**
- Instant loading (memory-mapped, no data read into RAM)
- Optimal performance for Vaex operations
- Uncompressed by design: `export_hdf5` has no compression option, because contiguous arrays are what make memory-mapping possible
- Random access patterns

### Apache Arrow Files

```python
# Open Arrow file (instant, memory-mapped)
df = vaex.open('data.arrow')
df = vaex.open('data.feather')  # Feather is Arrow format

# Multiple Arrow files
df = vaex.open('data_*.arrow')
```

**Advantages:**
- Instant loading (memory-mapped)
- Language-agnostic format
- Excellent for data sharing
- Zero-copy integration with Arrow ecosystem

### Parquet Files

```python
# Open Parquet file
df = vaex.open('data.parquet')

# Multiple Parquet files
df = vaex.open('data_*.parquet')

# From cloud storage
df = vaex.open('s3://bucket/data.parquet')
df = vaex.open('gs://bucket/data.parquet')
```

**Advantages:**
- Compressed by default
- Columnar format
- Wide ecosystem support
- Good for distributed systems

**Considerations:**
- Slower than HDF5/Arrow for local files
- May require full file read for some operations

### CSV Files

```python
# Simple CSV
df = vaex.from_csv('data.csv')

# Large CSV with automatic chunking
df = vaex.from_csv('large_data.csv', chunk_size=5_000_000)

# CSV with conversion to HDF5
df = vaex.from_csv('large_data.csv', convert='large_data.hdf5')
# Creates HDF5 file for future fast loading

# CSV with options
df = vaex.from_csv(
    'data.csv',
    sep=',',
    header=0,
    names=['col1', 'col2', 'col3'],
    dtype={'col1': 'int64', 'col2': 'float64'},
    usecols=['col1', 'col2'],  # Only load specific columns
    nrows=100000  # Limit number of rows
)
```

**Recommendations:**
- **Always convert large CSVs to HDF5** for repeated use
- Use `convert` parameter to create HDF5 automatically
- CSV loading can take significant time for large files

### FITS Files (Astronomy)

```python
# Open FITS file
df = vaex.open('astronomical_data.fits')

# Multiple FITS files
df = vaex.open('survey_*.fits')
```

## Writing/Exporting Data

### Export to HDF5

```python
# Export to HDF5 (recommended for Vaex)
df.export_hdf5('output.hdf5')

# With progress bar
df.export_hdf5('output.hdf5', progress=True)

# Export subset of columns
df[['col1', 'col2', 'col3']].export_hdf5('subset.hdf5')

# export_hdf5 has no compression argument; for smaller files use
# export_parquet(..., compression=...) or export_feather(..., compression='zstd')
```

### Export to Arrow

```python
# Export to Arrow format
df.export_arrow('output.arrow')

# Export to Feather (Arrow format)
df.export_feather('output.feather')
```

### Export to Parquet

```python
# Export to Parquet
df.export_parquet('output.parquet')

# With compression
df.export_parquet('output.parquet', compression='snappy')
df.export_parquet('output.parquet', compression='gzip')
```

### Export to CSV

```python
# Export to CSV (not recommended for large data)
df.export_csv('output.csv')

# With options
df.export_csv(
    'output.csv',
    sep=',',
    header=True,
    index=False,
    chunk_size=1_000_000
)

# Export subset
df[df.age > 25].export_csv('filtered_output.csv')
```

## Format Conversion

### CSV to HDF5 (Most Common)

```python
import vaex

# Method 1: Automatic conversion during read
df = vaex.from_csv('large.csv', convert='large.hdf5')
# Creates large.hdf5, returns DataFrame pointing to it

# Method 2: Explicit conversion
df = vaex.from_csv('large.csv')
df.export_hdf5('large.hdf5')

# Future loads (instant)
df = vaex.open('large.hdf5')
```

### HDF5 to Arrow

```python
# Load HDF5
df = vaex.open('data.hdf5')

# Export to Arrow
df.export_arrow('data.arrow')
```

### Parquet to HDF5

```python
# Load Parquet
df = vaex.open('data.parquet')

# Export to HDF5
df.export_hdf5('data.hdf5')
```

### Multiple CSV Files to Single HDF5

```python
import vaex
import glob

# Find all CSV files
csv_files = glob.glob('data_*.csv')

# Load and concatenate
dfs = [vaex.from_csv(f) for f in csv_files]
df_combined = vaex.concat(dfs)

# Export as single HDF5
df_combined.export_hdf5('combined_data.hdf5')
```

## Incremental/Chunked I/O

### Processing Large CSV in Chunks

```python
import vaex

# Stream a large CSV through Vaex and write a single HDF5 in one call.
# chunk_size controls how much is read at a time; convert= writes the HDF5.
df = vaex.open('huge.csv', convert='processed.hdf5', chunk_size=1_000_000)

# Now memory-mapped and instant to reopen
df = vaex.open('processed.hdf5')

# Add derived columns lazily after conversion (no extra memory)
df['new_col'] = df.x + df.y
```

### Exporting Large DataFrames

```python
# export_* already streams the data; you don't need a manual chunk loop.
# Prefer HDF5/Arrow over CSV for large output.
df.export_hdf5('large_output.hdf5', progress=True)

# If you must write CSV, export_csv writes incrementally under the hood:
df.export_csv('large_output.csv', chunk_size=1_000_000)
```

## Pandas Integration

### From Pandas to Vaex

```python
import pandas as pd
import vaex

# Read with pandas
pdf = pd.read_csv('data.csv')

# Convert to Vaex
df = vaex.from_pandas(pdf, copy_index=False)

# For better performance: Use Vaex directly
df = vaex.from_csv('data.csv')  # Preferred
```

### From Vaex to Pandas

```python
# Full conversion (careful with large data!)
pdf = df.to_pandas_df()

# Convert subset
pdf = df[['col1', 'col2']].to_pandas_df()
pdf = df[:10000].to_pandas_df()  # First 10k rows
pdf = df[df.age > 25].to_pandas_df()  # Filtered

# Sample for exploration
pdf_sample = df.sample(n=10000).to_pandas_df()
```

## Arrow Integration

### From Arrow to Vaex

```python
import pyarrow as pa
import vaex

# From Arrow Table
arrow_table = pa.table({
    'a': [1, 2, 3],
    'b': [4, 5, 6]
})
df = vaex.from_arrow_table(arrow_table)

# From an Arrow IPC file (vaex.open('data.arrow') does this directly).
# df.export_arrow() writes the IPC *stream* format by default (as_stream=True),
# so such files need pa.ipc.open_stream; pa.ipc.open_file raises "Not an Arrow file".
arrow_table = pa.ipc.open_stream('data.arrow').read_all()
df = vaex.from_arrow_table(arrow_table)
```

### From Vaex to Arrow

```python
# Convert to Arrow Table
arrow_table = df.to_arrow_table()

# Write Arrow file
import pyarrow as pa
with pa.ipc.new_file('output.arrow', arrow_table.schema) as writer:
    writer.write_table(arrow_table)

# Or use Vaex export
df.export_arrow('output.arrow')
```

## Remote and Cloud Storage

### Reading from S3

```python
import vaex

# Read from S3 (requires s3fs)
df = vaex.open('s3://bucket-name/data.parquet')
df = vaex.open('s3://bucket-name/data.hdf5')

# With credentials
import s3fs
fs = s3fs.S3FileSystem(key='access_key', secret='secret_key')
df = vaex.open('s3://bucket-name/data.parquet', fs=fs)
```

### Reading from Google Cloud Storage

```python
# Read from GCS (requires gcsfs)
df = vaex.open('gs://bucket-name/data.parquet')

# With credentials
import gcsfs
fs = gcsfs.GCSFileSystem(token='path/to/credentials.json')
df = vaex.open('gs://bucket-name/data.parquet', fs=fs)
```

### Reading from Azure

```python
# Read from Azure Blob Storage (requires adlfs)
df = vaex.open('az://container-name/data.parquet')
```

### Writing to Cloud Storage

```python
# Export to S3
df.export_parquet('s3://bucket-name/output.parquet')
df.export_hdf5('s3://bucket-name/output.hdf5')

# Export to GCS
df.export_parquet('gs://bucket-name/output.parquet')
```

## Database Integration

### Reading from SQL Databases

```python
import vaex
import pandas as pd
from sqlalchemy import create_engine

# Read with pandas, convert to Vaex
engine = create_engine('postgresql://user:password@host:port/database')
pdf = pd.read_sql('SELECT * FROM table', engine)
df = vaex.from_pandas(pdf)

# For large tables: Read in chunks
chunks = []
for chunk in pd.read_sql('SELECT * FROM large_table', engine, chunksize=100000):
    chunks.append(vaex.from_pandas(chunk))
df = vaex.concat(chunks)

# Better: Export from database to CSV/Parquet, then load with Vaex
```

### Writing to SQL Databases

```python
# Convert to pandas, then write
pdf = df.to_pandas_df()
pdf.to_sql('table_name', engine, if_exists='replace', index=False)

# For large data: Write in chunks
chunk_size = 100000
for i in range(0, len(df), chunk_size):
    chunk = df[i:i+chunk_size].to_pandas_df()
    chunk.to_sql('table_name', engine,
                 if_exists='append' if i > 0 else 'replace',
                 index=False)
```

## Memory-Mapped Files

### Understanding Memory Mapping

```python
# HDF5 and Arrow files are memory-mapped by default
df = vaex.open('data.hdf5')  # No data loaded into RAM

# Data is read from disk on-demand
mean = df.x.mean()  # Streams through data, minimal memory

# Check whether a column is virtual (lazy) vs a real on-disk/in-memory column
print('column_name' in df.virtual_columns)  # True = virtual expression
```

### Forcing Data into Memory

```python
# If needed, load data into memory
df_in_memory = df.copy()
for col in df.get_column_names():
    df_in_memory[col] = df[col].values  # Materializes in memory
```

## File Compression

### HDF5 (always uncompressed)

`export_hdf5` has no `compression` argument: vaex writes contiguous, uncompressed arrays so the
file can be memory-mapped. When disk size matters more than open time, use Parquet (below) or
compressed Feather:

```python
df.export_feather('data.feather', compression='zstd')  # default codec is 'lz4'
```

### Parquet Compression

```python
# Parquet is compressed by default
df.export_parquet('data.parquet', compression='snappy')  # Fast
df.export_parquet('data.parquet', compression='gzip')    # Better compression
df.export_parquet('data.parquet', compression='brotli')  # Best compression
```

## Vaex Server (Remote Data)

### Starting Vaex Server

```bash
# Start server
vaex-server data.hdf5 --host 0.0.0.0 --port 9000
```

### Connecting to Remote Server

```python
import vaex

# Connect to remote Vaex server
df = vaex.open('ws://hostname:9000/data')

# Operations work transparently
mean = df.x.mean()  # Computed on server
```

## State Files

### Saving DataFrame State

```python
# Save state (includes virtual columns, selections, etc.)
df.state_write('state.json')

# Includes:
# - Virtual column definitions
# - Active selections
# - Variables
# - Transformations (scalers, encoders, models)
```

### Loading DataFrame State

```python
# Load data
df = vaex.open('data.hdf5')

# Apply saved state
df.state_load('state.json')

# All virtual columns, selections, and transformations restored
```

## Best Practices

### 1. Choose the Right Format

```python
# For local work: HDF5
df.export_hdf5('data.hdf5')

# For sharing/interoperability: Arrow
df.export_arrow('data.arrow')

# For distributed systems: Parquet
df.export_parquet('data.parquet')

# Avoid CSV for large data
```

### 2. Convert CSV Once

```python
# One-time conversion
df = vaex.from_csv('large.csv', convert='large.hdf5')

# All future loads
df = vaex.open('large.hdf5')  # Instant!
```

### 3. Materialize Before Export

```python
# If DataFrame has many virtual columns
df_materialized = df.materialize()
df_materialized.export_hdf5('output.hdf5')

# Faster exports and future loads
```

### 4. Use Compression Wisely

```python
# For archival or infrequently accessed data: compressed Parquet
df.export_parquet('archived.parquet', compression='zstd')

# For active work (fastest open): uncompressed, memory-mapped HDF5
df.export_hdf5('working.hdf5')
```

### 5. Checkpoint Long Pipelines

```python
# After expensive preprocessing
df_preprocessed = preprocess(df)
df_preprocessed.export_hdf5('checkpoint_preprocessed.hdf5')

# After feature engineering
df_features = engineer_features(df_preprocessed)
df_features.export_hdf5('checkpoint_features.hdf5')

# Enables resuming from checkpoints
```

## Performance Comparisons

### Format Loading Speed

```python
import time
import vaex

# CSV (slowest)
start = time.time()
df_csv = vaex.from_csv('data.csv')
csv_time = time.time() - start

# HDF5 (instant)
start = time.time()
df_hdf5 = vaex.open('data.hdf5')
hdf5_time = time.time() - start

# Arrow (instant)
start = time.time()
df_arrow = vaex.open('data.arrow')
arrow_time = time.time() - start

print(f"CSV: {csv_time:.2f}s")
print(f"HDF5: {hdf5_time:.4f}s")
print(f"Arrow: {arrow_time:.4f}s")
```

## Common Patterns

### Pattern: Production Data Pipeline

```python
import vaex

# Read from source (CSV, database export, etc.)
df = vaex.from_csv('raw_data.csv')

# Process
df['cleaned'] = clean(df.raw_column)
df['feature'] = engineer_feature(df)

# Export for production use
df.export_hdf5('production_data.hdf5')
df.state_write('production_state.json')

# In production: Fast loading
df_prod = vaex.open('production_data.hdf5')
df_prod.state_load('production_state.json')
```

### Pattern: Archiving with Compression

```python
# Archive old data as compressed Parquet (vaex HDF5 exports are always uncompressed)
df_2020 = vaex.open('data_2020.hdf5')
df_2020.export_parquet('archive_2020.parquet', compression='zstd')

# Remove the uncompressed original once the archive is verified
import os
assert len(vaex.open('archive_2020.parquet')) == len(df_2020)
df_2020.close()
os.remove('data_2020.hdf5')
```

### Pattern: Multi-Source Data Loading

```python
import vaex

# Load from multiple sources
df_csv = vaex.from_csv('data.csv')
df_hdf5 = vaex.open('data.hdf5')
df_parquet = vaex.open('data.parquet')

# Concatenate
df_all = vaex.concat([df_csv, df_hdf5, df_parquet])

# Export unified format
df_all.export_hdf5('unified.hdf5')
```

## Troubleshooting

### Issue: CSV Loading Too Slow

```python
# Solution: Convert to HDF5
df = vaex.from_csv('large.csv', convert='large.hdf5')
# Future: df = vaex.open('large.hdf5')
```

### Issue: Out of Memory on Export

```python
# Solution: Export in chunks or materialize first
df_materialized = df.materialize()
df_materialized.export_hdf5('output.hdf5')
```

### Issue: Can't Read File from Cloud

```python
# Install required libraries
# pip install s3fs gcsfs adlfs

# Verify credentials
import s3fs
fs = s3fs.S3FileSystem()
fs.ls('s3://bucket-name/')
```

## Format Feature Matrix

| Feature | HDF5 | Arrow | Parquet | CSV |
|---------|------|-------|---------|-----|
| Load Speed | Instant | Instant | Fast | Slow |
| Memory-mapped | Yes | Yes | No | No |
| Compression | Optional | No | Yes | No |
| Columnar | Yes | Yes | Yes | No |
| Portability | Good | Excellent | Excellent | Excellent |
| File Size | Medium | Medium | Small | Large |
| Best For | Vaex workflows | Interop | Distributed | Exchange |

## Related Resources

- For DataFrame creation: See `core_dataframes.md`
- For performance optimization: See `performance.md`
- For data processing: See `data_processing.md`

# Machine Learning Integration

This reference covers Vaex's machine learning capabilities including transformers, encoders, feature engineering, model integration, and building ML pipelines on large datasets.

## Overview

Vaex provides a comprehensive ML framework (`vaex.ml`) that works seamlessly with large datasets. The framework includes:
- Transformers for feature scaling and engineering
- Encoders for categorical variables
- Dimensionality reduction (PCA)
- Clustering algorithms
- Integration with scikit-learn, XGBoost, LightGBM, CatBoost, and Keras
- State management for production deployment

**Key advantage:** All transformations create virtual columns, so preprocessing doesn't increase memory usage.

## Feature Scaling

### Standard Scaler

```python
import vaex
import vaex.ml

df = vaex.open('data.hdf5')

# Fit standard scaler
scaler = vaex.ml.StandardScaler(features=['age', 'income', 'score'])
scaler.fit(df)

# Transform (creates virtual columns)
df = scaler.transform(df)

# Scaled columns created as: 'standard_scaled_age', 'standard_scaled_income', etc.
print(df.column_names)
```

### MinMax Scaler

```python
# Scale to [0, 1] range
minmax_scaler = vaex.ml.MinMaxScaler(features=['age', 'income'])
minmax_scaler.fit(df)
df = minmax_scaler.transform(df)

# Custom range
minmax_scaler = vaex.ml.MinMaxScaler(
    features=['age'],
    feature_range=(-1, 1)
)
```

### MaxAbs Scaler

```python
# Scale by maximum absolute value
maxabs_scaler = vaex.ml.MaxAbsScaler(features=['values'])
maxabs_scaler.fit(df)
df = maxabs_scaler.transform(df)
```

### Robust Scaler

```python
# Scale using median and IQR (robust to outliers)
robust_scaler = vaex.ml.RobustScaler(features=['income', 'age'])
robust_scaler.fit(df)
df = robust_scaler.transform(df)
```

## Categorical Encoding

### Label Encoder

```python
# Encode categorical to integers
label_encoder = vaex.ml.LabelEncoder(features=['category', 'region'])
label_encoder.fit(df)
df = label_encoder.transform(df)

# Creates: 'label_encoded_category', 'label_encoded_region'
```

### One-Hot Encoder

```python
# Create binary columns for each category
onehot = vaex.ml.OneHotEncoder(features=['category'])
onehot.fit(df)
df = onehot.transform(df)

# Creates columns like: 'category_A', 'category_B', 'category_C'

# Control prefix
onehot = vaex.ml.OneHotEncoder(
    features=['category'],
    prefix='cat_'
)
```

### Frequency Encoder

```python
# Encode by category frequency
freq_encoder = vaex.ml.FrequencyEncoder(features=['category'])
freq_encoder.fit(df)
df = freq_encoder.transform(df)

# Each category replaced by its frequency in the dataset
```

### Target Encoder (Bayesian Mean Encoder)

```python
# Encode category by a smoothed target mean (for supervised learning).
# vaex.ml has no plain TargetEncoder; BayesianTargetEncoder shrinks each category
# mean toward the global mean with `weight`.
target_encoder = vaex.ml.BayesianTargetEncoder(
    features=['category'],
    target='target_variable',
    weight=100,
)
target_encoder.fit(df)
df = target_encoder.transform(df)

# Creates: 'mean_encoded_category'; unseen categories get the global mean
```

Fit target encoders on the training split only, then transform validation/test data, to avoid target leakage.

### Weight of Evidence Encoder

```python
# Encode for binary classification
woe_encoder = vaex.ml.WeightOfEvidenceEncoder(
    features=['category'],
    target='binary_target'
)
woe_encoder.fit(df)
df = woe_encoder.transform(df)
```

## Feature Engineering

### Binning/Discretization

```python
# Bin continuous variable into discrete bins
binner = vaex.ml.KBinsDiscretizer(
    features=['age'],
    n_bins=5,
    strategy='uniform'  # or 'quantile' / 'kmeans'
)
binner.fit(df)
df = binner.transform(df)  # creates 'binned_age'
```

### Cyclic Transformations

```python
# Transform cyclic features. `n` is a single int period, so use one transformer per period.
hour_cycle = vaex.ml.CycleTransformer(features=['hour'], n=24)
dow_cycle = vaex.ml.CycleTransformer(features=['day_of_week'], n=7)
df = hour_cycle.fit_transform(df)
df = dow_cycle.fit_transform(df)

# Creates x/y (cos/sin) components, e.g. 'hour_x', 'hour_y'
```

### PCA (Principal Component Analysis)

```python
# Dimensionality reduction
pca = vaex.ml.PCA(
    features=['feature1', 'feature2', 'feature3', 'feature4'],
    n_components=2
)
pca.fit(df)
df = pca.transform(df)

# Creates: 'PCA_0', 'PCA_1'

# Access explained variance
print(pca.explained_variance_ratio_)
```

### Random Projection

```python
# Fast dimensionality reduction
projector = vaex.ml.RandomProjections(
    features=['x1', 'x2', 'x3', 'x4', 'x5'],
    n_components=3
)
projector.fit(df)
df = projector.transform(df)  # creates 'random_projection_0', ...
```

## Clustering

### K-Means

```python
# Cluster data (KMeans lives in vaex.ml.cluster)
import vaex.ml.cluster

kmeans = vaex.ml.cluster.KMeans(
    features=['feature1', 'feature2', 'feature3'],
    n_clusters=5,
    max_iter=100
)
kmeans.fit(df)
df = kmeans.transform(df)

# Creates 'prediction_kmeans' column with cluster labels

# Access cluster centers (no trailing underscore)
print(kmeans.cluster_centers)
```

## Integration with External Libraries

### Scikit-Learn

```python
from sklearn.ensemble import RandomForestClassifier
import vaex.ml.sklearn  # `import vaex.ml` alone does not load the sklearn/xgboost/... submodules

# Prepare data. Use df['split'], not df.split: DataFrame.split is a method, so
# df.split == 'train' silently compares the method, not the column.
train_df = df[df['split'] == 'train']
test_df = df[df['split'] == 'test']

# Features and target
features = ['feature1', 'feature2', 'feature3']
target = 'target'

# Train scikit-learn model
model = RandomForestClassifier(n_estimators=100)

# Fit using Vaex data
sklearn_model = vaex.ml.sklearn.Predictor(
    features=features,
    target=target,
    model=model,
    prediction_name='rf_prediction'
)
sklearn_model.fit(train_df)

# Predict (creates virtual column)
test_df = sklearn_model.transform(test_df)

# Access predictions
predictions = test_df.rf_prediction.values
```

### XGBoost

```python
import xgboost as xgb
import vaex.ml.xgboost

# Configure parameters
params = {
    'max_depth': 6,
    'eta': 0.1,
    'objective': 'reg:squarederror',
    'eval_metric': 'rmse'
}

# params and num_boost_round are constructor arguments, not fit() arguments
booster = vaex.ml.xgboost.XGBoostModel(
    features=features,
    target=target,
    params=params,
    num_boost_round=100,
    prediction_name='xgb_pred'
)

# Train (loads the training features into memory)
booster.fit(train_df)

# Predict
test_df = booster.transform(test_df)
```

### LightGBM

```python
import lightgbm as lgb
import vaex.ml.lightgbm

# Parameters
params = {
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 31,
    'learning_rate': 0.05
}

# params and num_boost_round are constructor arguments
lgb_model = vaex.ml.lightgbm.LightGBMModel(
    features=features,
    target=target,
    params=params,
    num_boost_round=100,
    prediction_name='lgb_pred'
)

# Train
lgb_model.fit(train_df)

# Predict
test_df = lgb_model.transform(test_df)
```

### CatBoost

```python
from catboost import CatBoostClassifier
import vaex.ml.catboost

# Parameters
params = {
    'depth': 6,
    'learning_rate': 0.1,
    'loss_function': 'Logloss'
}

# params / num_boost_round / prediction_type are constructor arguments
catboost_model = vaex.ml.catboost.CatBoostModel(
    features=features,
    target=target,
    params=params,
    num_boost_round=100,
    prediction_type='Probability',  # or 'Class' / 'RawFormulaVal'
    prediction_name='catboost_pred'
)

# Train
catboost_model.fit(train_df)

# Predict
test_df = catboost_model.transform(test_df)
```

### Keras/TensorFlow

The wrapper lives in `vaex.ml.tensorflow` and only serves predictions: train the network with the
Keras API (its `KerasModel.fit` raises `NotImplementedError`), streaming batches from Vaex.

```python
from tensorflow import keras
import vaex.ml.tensorflow

# Stream training batches out of the Vaex DataFrame
gen_train = train_df.ml.tensorflow.to_keras_generator(features=features, target=target, batch_size=10_000)

nn_model = keras.Sequential([
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(1, activation='sigmoid')
])
nn_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
nn_model.fit(x=gen_train, epochs=10, steps_per_epoch=len(train_df) // 10_000)

# Wrap the fitted model to get lazy predictions + state serialization
keras_model = vaex.ml.tensorflow.KerasModel(
    features=features,
    model=nn_model,
    prediction_name='keras_pred'
)
test_df = keras_model.transform(test_df)
```

## Building ML Pipelines

### Sequential Pipeline

```python
import vaex.ml

# Create preprocessing pipeline
pipeline = []

# Step 1: Encode categorical
label_enc = vaex.ml.LabelEncoder(features=['category'])
pipeline.append(label_enc)

# Step 2: Scale features
scaler = vaex.ml.StandardScaler(features=['age', 'income'])
pipeline.append(scaler)

# Step 3: PCA
pca = vaex.ml.PCA(features=['age', 'income'], n_components=2)
pipeline.append(pca)

# Fit pipeline
for step in pipeline:
    step.fit(df)
    df = step.transform(df)

# Or use fit_transform
for step in pipeline:
    df = step.fit_transform(df)
```

### Complete ML Pipeline

```python
import vaex
import vaex.ml
import vaex.ml.sklearn
from sklearn.ensemble import RandomForestClassifier

# Load data
df = vaex.open('data.hdf5')

# Split data
train_df = df[df.year < 2020]
test_df = df[df.year >= 2020]

# Define pipeline
# 1. Categorical encoding
cat_encoder = vaex.ml.LabelEncoder(features=['category', 'region'])

# 2. Feature scaling
scaler = vaex.ml.StandardScaler(features=['age', 'income', 'score'])

# 3. Model
features = ['label_encoded_category', 'label_encoded_region',
            'standard_scaled_age', 'standard_scaled_income', 'standard_scaled_score']
model = vaex.ml.sklearn.Predictor(
    features=features,
    target='target',
    model=RandomForestClassifier(n_estimators=100),
    prediction_name='prediction'
)

# Fit pipeline
train_df = cat_encoder.fit_transform(train_df)
train_df = scaler.fit_transform(train_df)
model.fit(train_df)

# Apply to test
test_df = cat_encoder.transform(test_df)
test_df = scaler.transform(test_df)
test_df = model.transform(test_df)

# Evaluate
accuracy = (test_df.prediction == test_df.target).mean()
print(f"Accuracy: {accuracy:.4f}")
```

## State Management and Deployment

### Saving Pipeline State

```python
# After fitting all transformers and model
# Save the entire pipeline state
train_df.state_write('pipeline_state.json')

# In production: Load fresh data and apply transformations.
# The state also stores train_df's row filter (e.g. split == 'train'); set_filter=False
# keeps it from being applied to the new data.
prod_df = vaex.open('new_data.hdf5')
prod_df.state_load('pipeline_state.json', set_filter=False)

# All transformations and models are applied
predictions = prod_df.prediction.values
```

### Transferring State Between DataFrames

```python
# Fit on training data
train_df = cat_encoder.fit_transform(train_df)
train_df = scaler.fit_transform(train_df)
model.fit(train_df)

# Save state
train_df.state_write('model_state.json')

# Apply to test data. With the default set_filter=True, the training filter in the
# state replaces test_df's own filter and you would silently evaluate on training rows.
test_df.state_load('model_state.json', set_filter=False)

# Apply to validation data
val_df.state_load('model_state.json', set_filter=False)
```

### Exporting with Transformations

```python
# Export DataFrame with all virtual columns materialized
df_with_features = df.copy()
df_with_features = df_with_features.materialize()
df_with_features.export_hdf5('processed_data.hdf5')
```

## Model Evaluation

### Classification Metrics

```python
# Binary classification
from sklearn.metrics import accuracy_score, roc_auc_score, f1_score

y_true = test_df.target.values
y_pred = test_df.prediction.values
y_proba = test_df.prediction_proba.values if hasattr(test_df, 'prediction_proba') else None

accuracy = accuracy_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)
if y_proba is not None:
    auc = roc_auc_score(y_true, y_proba)

print(f"Accuracy: {accuracy:.4f}")
print(f"F1 Score: {f1:.4f}")
if y_proba is not None:
    print(f"AUC-ROC: {auc:.4f}")
```

### Regression Metrics

```python
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

y_true = test_df.target.values
y_pred = test_df.prediction.values

mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_true, y_pred)
r2 = r2_score(y_true, y_pred)

print(f"RMSE: {rmse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"R²: {r2:.4f}")
```

### Cross-Validation

```python
# Manual K-fold cross-validation
import numpy as np

# Create fold indices
df['fold'] = np.random.randint(0, 5, len(df))

results = []
for fold in range(5):
    train = df[df.fold != fold]
    val = df[df.fold == fold]

    # Fit pipeline
    train = encoder.fit_transform(train)
    train = scaler.fit_transform(train)
    model.fit(train)

    # Validate
    val = encoder.transform(val)
    val = scaler.transform(val)
    val = model.transform(val)

    accuracy = (val.prediction == val.target).mean()
    results.append(accuracy)

print(f"CV Accuracy: {np.mean(results):.4f} ± {np.std(results):.4f}")
```

## Feature Selection

### Correlation-Based

```python
# Compute correlations with target
correlations = {}
for feature in features:
    corr = df.correlation(df[feature], df.target)
    correlations[feature] = abs(corr)

# Sort by correlation
sorted_features = sorted(correlations.items(), key=lambda x: x[1], reverse=True)
top_features = [f[0] for f in sorted_features[:10]]

print("Top 10 features:", top_features)
```

### Variance-Based

```python
# Remove low-variance features
feature_variances = {}
for feature in features:
    var = df[feature].std() ** 2
    feature_variances[feature] = var

# Keep features with variance above threshold
threshold = 0.01
selected_features = [f for f, v in feature_variances.items() if v > threshold]
```

## Handling Imbalanced Data

### Class Weights

```python
# Compute class weights from per-class counts
counts = df['target'].value_counts()  # pandas Series indexed by class label
total = len(df)
weights = {
    label: total / (len(counts) * n)
    for label, n in counts.items()
}

# Use in model
model = RandomForestClassifier(class_weight=weights)
```

### Undersampling

```python
# Undersample majority class
minority_count = df[df.target == 1].count()

# Sample from majority class
majority_sampled = df[df.target == 0].sample(n=minority_count)
minority_all = df[df.target == 1]

# Combine
df_balanced = vaex.concat([majority_sampled, minority_all])
```

### Oversampling (SMOTE alternative)

```python
# Duplicate minority class samples
minority = df[df.target == 1]
majority = df[df.target == 0]

# Replicate minority
minority_oversampled = vaex.concat([minority] * 5)

# Combine
df_balanced = vaex.concat([majority, minority_oversampled])
```

## Common Patterns

### Pattern: End-to-End Classification Pipeline

```python
import vaex
import vaex.ml
import vaex.ml.sklearn
from sklearn.ensemble import RandomForestClassifier

# Load and split
df = vaex.open('data.hdf5')
train = df[df['split'] == 'train']  # df.split is a method, so index the column
test = df[df['split'] == 'test']

# Preprocessing
# Categorical encoding
cat_enc = vaex.ml.LabelEncoder(features=['cat1', 'cat2'])
train = cat_enc.fit_transform(train)

# Feature scaling
scaler = vaex.ml.StandardScaler(features=['num1', 'num2', 'num3'])
train = scaler.fit_transform(train)

# Model training
features = ['label_encoded_cat1', 'label_encoded_cat2',
            'standard_scaled_num1', 'standard_scaled_num2', 'standard_scaled_num3']
model = vaex.ml.sklearn.Predictor(
    features=features,
    target='target',
    model=RandomForestClassifier(n_estimators=100)
)
model.fit(train)
train = model.transform(train)  # adds the 'prediction' column, so the state carries the model

# Save state
train.state_write('production_pipeline.json')

# Apply to test (set_filter=False keeps test's own split filter)
test.state_load('production_pipeline.json', set_filter=False)

# Evaluate
accuracy = (test.prediction == test.target).mean()
print(f"Test Accuracy: {accuracy:.4f}")
```

### Pattern: Feature Engineering Pipeline

```python
# Create rich features
df['age_squared'] = df.age ** 2
df['income_log'] = df.income.log()
df['age_income_interaction'] = df.age * df.income

# Binning
df['age_bin'] = df.age.digitize([0, 18, 30, 50, 65, 100])

# Cyclic features
df['hour_sin'] = (2 * np.pi * df.hour / 24).sin()
df['hour_cos'] = (2 * np.pi * df.hour / 24).cos()

# Aggregate features
avg_by_category = df.groupby('category').agg({'income': 'mean'})
# Join back to create feature
df = df.join(avg_by_category, on='category', rsuffix='_category_mean')
```

## Best Practices

1. **Use virtual columns** - Transformers create virtual columns (no memory overhead)
2. **Save state files** - Enable easy deployment and reproduction
3. **Batch operations** - Use `delay=True` when computing multiple features
4. **Feature scaling** - Always scale features before PCA or distance-based algorithms
5. **Encode categories** - Use appropriate encoder (label, one-hot, target)
6. **Cross-validate** - Always validate on held-out data
7. **Monitor memory** - Check memory usage with `df.byte_size()`
8. **Export checkpoints** - Save intermediate results in long pipelines

## Related Resources

- For data preprocessing: See `data_processing.md`
- For performance optimization: See `performance.md`
- For DataFrame operations: See `core_dataframes.md`

# Performance and Optimization

This reference covers Vaex's performance features including lazy evaluation, caching, memory management, async operations, and optimization strategies for processing massive datasets.

## Understanding Lazy Evaluation

Lazy evaluation is the foundation of Vaex's performance:

### How Lazy Evaluation Works

```python
import vaex

df = vaex.open('large_file.hdf5')

# No computation happens here - just defines what to compute
df['total'] = df.price * df.quantity
df['log_price'] = df.price.log()
mean_expr = df.total.mean()

# Computation happens here (when result is needed)
result = mean_expr  # Now the mean is actually calculated
```

**Key concepts:**
- **Expressions** are lazy - they define computations without executing them
- **Materialization** happens when you access the result
- **Query optimization** happens automatically before execution

### When Does Evaluation Happen?

```python
# These trigger evaluation:
print(df.x.mean())                    # Accessing value
array = df.x.values                   # Getting NumPy array
pdf = df.to_pandas_df()              # Converting to pandas
df.export_hdf5('output.hdf5')       # Exporting

# These do NOT trigger evaluation:
df['new_col'] = df.x + df.y          # Creating virtual column
expr = df.x.mean()                    # Creating expression
df_filtered = df[df.x > 10]          # Creating filtered view
```

## Batching Operations with delay=True

Execute multiple operations together for better performance:

### Basic Delayed Execution

```python
# Without delay - each operation processes entire dataset
mean_x = df.x.mean()      # Pass 1 through data
std_x = df.x.std()        # Pass 2 through data
max_x = df.x.max()        # Pass 3 through data

# With delay - single pass through dataset.
# Each delayed call returns a promise; df.execute() runs all pending ops together.
mean_x = df.mean(df.x, delay=True)
std_x = df.std(df.x, delay=True)
max_x = df.max(df.x, delay=True)

df.execute()        # Single pass triggers all delayed operations
print(mean_x.get()) # mean
print(std_x.get())  # std
print(max_x.get())  # max
```

### Delayed Execution with Multiple Columns

```python
# Compute statistics for many columns efficiently.
# Build the promises first, run one pass, then collect with .get().
columns = ['sales', 'quantity', 'profit', 'cost']
promises = {
    column: (df.mean(df[column], delay=True), df.std(df[column], delay=True))
    for column in columns
}

df.execute()  # Single pass over the data

stats = {
    column: {'mean': mean_p.get(), 'std': std_p.get()}
    for column, (mean_p, std_p) in promises.items()
}
```

### When to Use delay=True

Use `delay=True` when:
- Computing multiple aggregations
- Computing statistics on many columns
- Building dashboards or reports
- Any scenario requiring multiple passes through data

```python
# Bad: 4 passes through dataset
mean1 = df.col1.mean()
mean2 = df.col2.mean()
mean3 = df.col3.mean()
mean4 = df.col4.mean()

# Good: 1 pass through dataset
promises = [
    df.mean(df.col1, delay=True),
    df.mean(df.col2, delay=True),
    df.mean(df.col3, delay=True),
    df.mean(df.col4, delay=True),
]
df.execute()
results = [p.get() for p in promises]
```

## Composing Delayed Computations

Vaex's delayed mechanism is promise-based (not Python `async`/`await`). Use the
`@vaex.delayed` decorator to chain post-processing onto delayed results; the whole
chain runs when `df.execute()` is called.

```python
import vaex

df = vaex.open('large_file.hdf5')

@vaex.delayed
def normalized_range(min_val, max_val):
    # Runs once the two delayed aggregations resolve
    return (max_val - min_val)

# Build the chain (nothing computed yet)
result = normalized_range(df.min(df.x, delay=True), df.max(df.x, delay=True))

df.execute()       # Single pass triggers the aggregations and the chained function
print(result.get())
```

### Retrieving a single delayed result

```python
# A delayed call returns a promise
promise = df.mean(df.x, delay=True)

# Do other work / build more promises...

df.execute()           # Trigger the pass over the data
result = promise.get() # Now the value is available
```

## Virtual Columns vs Materialized Columns

Understanding the difference is crucial for performance:

### Virtual Columns (Preferred)

```python
# Virtual column - computed on-the-fly, zero memory
df['total'] = df.price * df.quantity
df['log_sales'] = df.sales.log()
df['full_name'] = df.first_name + ' ' + df.last_name

# Check if a column is virtual
print('total' in df.virtual_columns)  # True = virtual (lazy expression)

# Benefits:
# - Zero memory overhead
# - Always up-to-date if source data changes
# - Fast to create
```

### Materialized Columns

```python
# Materialize an existing virtual column in place (computes once, stores in RAM).
# materialize(column=None) materializes all virtual columns when column is omitted.
df = df.materialize('total')

# When to materialize:
# - Column computed repeatedly (amortize cost)
# - Complex expression used in many operations
# - Need to export data
```

### Deciding: Virtual vs Materialized

```python
# Virtual is better when:
# - Column is simple (x + y, x * 2, etc.)
# - Column used infrequently
# - Memory is limited

# Materialize when:
# - Complex computation (multiple operations)
# - Used repeatedly in aggregations
# - Slows down other operations

# Example: Complex calculation used many times
df['complex'] = (df.x.log() * df.y.sqrt() + df.z ** 2).values  # Materialize
```

## Caching Strategies

Vaex automatically caches some operations, but you can optimize further:

### Automatic Caching

```python
# First call computes and caches
mean1 = df.x.mean()  # Computes

# Second call uses cache
mean2 = df.x.mean()  # From cache (instant)

# Cache invalidated if DataFrame changes
df['new_col'] = df.x + 1
mean3 = df.x.mean()  # Recomputes
```

### State Management

```python
# Save DataFrame state (includes virtual columns)
df.state_write('state.json')

# Load state later
df_new = vaex.open('data.hdf5')
df_new.state_load('state.json')  # Restores virtual columns, selections
```

### Checkpoint Pattern

```python
# Export intermediate results for complex pipelines
df['processed'] = complex_calculation(df)

# Save checkpoint
df.export_hdf5('checkpoint.hdf5')

# Resume from checkpoint
df = vaex.open('checkpoint.hdf5')
# Continue processing...
```

## Memory Management

Optimize memory usage for very large datasets:

### Memory-Mapped Files

```python
# HDF5 and Arrow are memory-mapped (optimal)
df = vaex.open('data.hdf5')  # No memory used until accessed

# File stays on disk, only accessed portions loaded to RAM
mean = df.x.mean()  # Streams through data, minimal memory
```

### Chunked Processing

```python
# Process large DataFrame in chunks
chunk_size = 1_000_000

for i1, i2, chunk in df.to_pandas_df(chunk_size=chunk_size):
    # Process chunk (careful: defeats Vaex's purpose)
    process_chunk(chunk)

# Better: Use Vaex operations directly (no chunking needed)
result = df.x.mean()  # Handles large data automatically
```

### Monitoring Memory Usage

```python
# Check DataFrame memory footprint
print(df.byte_size())  # Bytes used by materialized columns

# Check which columns are materialized vs virtual
for col in df.get_column_names():
    kind = 'virtual' if col in df.virtual_columns else 'materialized'
    print(f"{col}: {kind}")

# Time an operation
import time
start = time.time()
result = df.x.mean()
print(f"mean computed in {time.time() - start:.3f}s")
```

## Parallel Computation

Vaex automatically parallelizes operations:

### Multithreading

```python
# Vaex uses all CPU cores by default
import vaex

# Operations automatically parallelize across all cores
mean = df.x.mean()  # Uses all threads

# Cap the thread pool via the environment before importing vaex, e.g.:
#   VAEX_NUM_THREADS=8 python script.py
```

### Going Distributed

Vaex is single-machine out-of-core. If you genuinely need a multi-node cluster
(10s of TB, custom task graphs, a scheduler dashboard), reach for **Dask** instead
— see the `alterlab-dask` skill. Convert a column to a Dask array when you need to
hand data off:

```python
arr = df.x.to_dask_array()  # bridge a single column into the Dask ecosystem
```

## JIT Compilation

Vaex can JIT-compile arithmetic expressions for a speed boost.

### `.jit_numba()` / `.jit_pythran()` on expressions

```python
import vaex

df = vaex.open('large_file.hdf5')

# Build an expression as usual, then JIT-compile it.
# Vaex generates and compiles optimized code for the expression.
df['dist'] = (df.x ** 2 + df.y ** 2).jit_numba()   # requires numba
# Alternative backend: (df.x ** 2 + df.y ** 2).jit_pythran()

mean_dist = df.dist.mean()  # runs over the JIT-compiled expression
```

### Registering a custom Python function

```python
# Register a NumPy-vectorized function, then use it inside expressions.
@vaex.register_function()
def squared_sum(x, y):
    return x ** 2 + y ** 2  # x, y arrive as NumPy arrays

df['custom'] = df.func.squared_sum(df.x, df.y)
result = df.custom.mean()
```

## Optimization Strategies

### Strategy 1: Minimize Materializations

```python
# Bad: Creates many materialized columns
df['a'] = (df.x + df.y).values
df['b'] = (df.a * 2).values
df['c'] = (df.b + df.z).values

# Good: Keep virtual until final export
df['a'] = df.x + df.y
df['b'] = df.a * 2
df['c'] = df.b + df.z
# Only materialize if exporting:
# df.export_hdf5('output.hdf5')
```

### Strategy 2: Use Selections Instead of Filtering

```python
# Less efficient: Creates new DataFrames
df_high = df[df.value > 100]
df_low = df[df.value <= 100]
mean_high = df_high.value.mean()
mean_low = df_low.value.mean()

# More efficient: Use selections
df.select(df.value > 100, name='high')
df.select(df.value <= 100, name='low')
mean_high = df.value.mean(selection='high')
mean_low = df.value.mean(selection='low')
```

### Strategy 3: Batch Aggregations

```python
# Less efficient: Multiple passes
stats = {
    'mean': df.x.mean(),
    'std': df.x.std(),
    'min': df.x.min(),
    'max': df.x.max()
}

# More efficient: Single pass
promises = [
    df.mean(df.x, delay=True),
    df.std(df.x, delay=True),
    df.min(df.x, delay=True),
    df.max(df.x, delay=True),
]
df.execute()
stats = dict(zip(['mean', 'std', 'min', 'max'], [p.get() for p in promises]))
```

### Strategy 4: Choose Optimal File Formats

```python
# Slow: Large CSV
df = vaex.from_csv('huge.csv')  # Can take minutes

# Fast: HDF5 or Arrow
df = vaex.open('huge.hdf5')     # Instant
df = vaex.open('huge.arrow')    # Instant

# One-time conversion
df = vaex.from_csv('huge.csv', convert='huge.hdf5')
# Future loads: vaex.open('huge.hdf5')
```

### Strategy 5: Optimize Expressions

```python
# Less efficient: Repeated calculations
df['result'] = df.x.log() + df.x.log() * 2

# More efficient: Reuse calculations
df['log_x'] = df.x.log()
df['result'] = df.log_x + df.log_x * 2

# Even better: Combine operations
df['result'] = df.x.log() * 3  # Simplified math
```

## Performance Profiling

### Basic Profiling

```python
import time
import vaex

df = vaex.open('large_file.hdf5')

# Time operations
start = time.time()
result = df.x.mean()
elapsed = time.time() - start
print(f"Computed in {elapsed:.2f} seconds")
```

### Progress Reporting

```python
# Many Vaex operations accept progress=True to show a progress bar,
# which doubles as a rough timing/feedback tool for long passes.
result = df.groupby('category', agg={'value': 'sum'}, progress=True)
df.export_hdf5('out.hdf5', progress=True)
```

### Benchmarking Patterns

```python
# Compare strategies
def benchmark_operation(operation, name):
    start = time.time()
    result = operation()
    elapsed = time.time() - start
    print(f"{name}: {elapsed:.3f}s")
    return result

# Test different approaches
benchmark_operation(lambda: df.x.mean(), "Direct mean")
benchmark_operation(lambda: df[df.x > 0].x.mean(), "Filtered mean")
benchmark_operation(lambda: df.x.mean(selection='positive'), "Selection mean")
```

## Common Performance Issues and Solutions

### Issue: Slow Aggregations

```python
# Problem: Multiple separate aggregations
for col in df.column_names:
    print(f"{col}: {df[col].mean()}")

# Solution: Batch with delay=True
promises = [df.mean(df[col], delay=True) for col in df.column_names]
df.execute()
for col, p in zip(df.column_names, promises):
    print(f"{col}: {p.get()}")
```

### Issue: High Memory Usage

```python
# Problem: Materializing large virtual columns
df['large_col'] = (complex_expression).values

# Solution: Keep virtual, or materialize and export
df['large_col'] = complex_expression  # Virtual
# Or: df.export_hdf5('with_new_col.hdf5')
```

### Issue: Slow Exports

```python
# Problem: Exporting with many virtual columns
df.export_csv('output.csv')  # Slow if many virtual columns

# Solution: Export to HDF5 or Arrow (faster)
df.export_hdf5('output.hdf5')
df.export_arrow('output.arrow')

# Or materialize first for CSV
df_materialized = df.materialize()
df_materialized.export_csv('output.csv')
```

### Issue: Repeated Complex Calculations

```python
# Problem: Complex virtual column used repeatedly
df['complex'] = df.x.log() * df.y.sqrt() + df.z ** 3
result1 = df.groupby('cat1').agg({'complex': 'mean'})
result2 = df.groupby('cat2').agg({'complex': 'sum'})
result3 = df.complex.std()

# Solution: Materialize once
df['complex'] = (df.x.log() * df.y.sqrt() + df.z ** 3).values
# Or: df = df.materialize('complex')
```

## Performance Best Practices Summary

1. **Use HDF5 or Arrow formats** - Orders of magnitude faster than CSV
2. **Leverage lazy evaluation** - Don't force computation until necessary
3. **Batch operations with delay=True** - Minimize passes through data
4. **Keep columns virtual** - Materialize only when beneficial
5. **Use selections not filters** - More efficient for multiple segments
6. **Profile your code** - Identify bottlenecks before optimizing
7. **Avoid `.values` and `.to_pandas_df()`** - Keep operations in Vaex
8. **Parallelize naturally** - Vaex uses all cores automatically
9. **Export to efficient formats** - Checkpoint complex pipelines
10. **Optimize expressions** - Simplify math and reuse calculations

## Related Resources

- For DataFrame basics: See `core_dataframes.md`
- For data operations: See `data_processing.md`
- For file I/O optimization: See `io_operations.md`

# Data Visualization

This reference covers Vaex's visualization capabilities for creating plots, heatmaps, and interactive visualizations of large datasets.

## Overview

Vaex excels at visualizing datasets with billions of rows through efficient binning and aggregation. The visualization system works directly with large data without sampling, providing accurate representations of the entire dataset.

**Key features:**
- Visualize billion-row datasets interactively
- No sampling required - uses all data
- Automatic binning and aggregation
- Integration with matplotlib
- Interactive widgets for Jupyter

**API note (vaex 4.19):** plots live on the `df.viz` accessor — `df.viz.histogram(...)` and
`df.viz.heatmap(...)`. The older `df.plot1d(...)` / `df.plot(...)` / `df.scatter(...)` are
deprecated aliases (removal announced for 5.x) that forward the same arguments. `df.viz.heatmap`
calls `matplotlib.cm.get_cmap`, which matplotlib removed in 3.9, so 2-D plots need
`matplotlib<3.9` in the Vaex environment; alternatively compute the grid yourself with
`df.count(binby=[df.x, df.y], limits=..., shape=...)` and draw it with `plt.imshow`.

## Basic Plotting

### 1D Histograms

```python
import vaex
import matplotlib.pyplot as plt

df = vaex.open('data.hdf5')

# Simple histogram
df.viz.histogram(df.age)

# With customization
df.viz.histogram(df.age,
                 limits=[0, 100],
                 shape=50,              # Number of bins
                 figsize=(10, 6),
                 xlabel='Age',
                 ylabel='Count')

plt.show()
```

### 2D Density Plots (Heatmaps)

```python
# Basic 2D plot
df.viz.heatmap(df.x, df.y)

# With limits
df.viz.heatmap(df.x, df.y, limits=[[0, 10], [0, 10]])

# Auto limits using percentiles
df.viz.heatmap(df.x, df.y, limits='99.7%')  # 3-sigma limits

# Customize shape (resolution)
df.viz.heatmap(df.x, df.y, shape=(512, 512))

# Logarithmic color scale
df.viz.heatmap(df.x, df.y, f='log')
```

### Scatter Plots (Small Data)

```python
# For smaller datasets or samples
df_sample = df.sample(n=1000)

df_sample.viz.scatter(df_sample.x, df_sample.y,
                      alpha=0.5,
                      s=10)  # extra kwargs go to plt.scatter

plt.show()
```

## Advanced Visualization Options

### Color Scales and Normalization

```python
# Linear scale (default)
df.viz.heatmap(df.x, df.y, f='identity')

# Logarithmic scale
df.viz.heatmap(df.x, df.y, f='log')
df.viz.heatmap(df.x, df.y, f='log10')

# Square root scale
df.viz.heatmap(df.x, df.y, f='sqrt')

# Custom colormap
df.viz.heatmap(df.x, df.y, colormap='viridis')
df.viz.heatmap(df.x, df.y, colormap='plasma')
df.viz.heatmap(df.x, df.y, colormap='hot')
```

### Limits and Ranges

```python
# Manual limits
df.viz.heatmap(df.x, df.y, limits=[[xmin, xmax], [ymin, ymax]])

# Percentile-based limits
df.viz.heatmap(df.x, df.y, limits='99.7%')  # 3-sigma
df.viz.heatmap(df.x, df.y, limits='95%')
df.viz.heatmap(df.x, df.y, limits='minmax')  # Full range

# Mixed limits
df.viz.heatmap(df.x, df.y, limits=[[0, 100], 'minmax'])
```

### Resolution Control

```python
# Higher resolution (more bins)
df.viz.heatmap(df.x, df.y, shape=(1024, 1024))

# Lower resolution (faster)
df.viz.heatmap(df.x, df.y, shape=(128, 128))

# Different resolutions per axis
df.viz.heatmap(df.x, df.y, shape=(512, 256))
```

## Statistical Visualizations

### Visualizing Aggregations

```python
# Mean on a grid
df.viz.heatmap(df.x, df.y, what='mean(z)',
        limits=[[0, 10], [0, 10]],
        shape=(100, 100),
        colormap='viridis')

# Standard deviation
df.viz.heatmap(df.x, df.y, what='std(z)')

# Sum
df.viz.heatmap(df.x, df.y, what='sum(z)')

# Count (default)
df.viz.heatmap(df.x, df.y, what='count(*)')
```

### Multiple Statistics

```python
# Create figure with subplots
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Count
plt.sca(axes[0, 0])
df.viz.heatmap(df.x, df.y, what='count(*)', show=False)
axes[0, 0].set_title('Count')

# Mean
plt.sca(axes[0, 1])
df.viz.heatmap(df.x, df.y, what='mean(z)', show=False)
axes[0, 1].set_title('Mean of z')

# Std
plt.sca(axes[1, 0])
df.viz.heatmap(df.x, df.y, what='std(z)', show=False)
axes[1, 0].set_title('Std of z')

# Min
plt.sca(axes[1, 1])
df.viz.heatmap(df.x, df.y, what='min(z)', show=False)
axes[1, 1].set_title('Min of z')

plt.tight_layout()
plt.show()
```

## Working with Selections

Visualize different segments of data simultaneously:

```python
import vaex
import matplotlib.pyplot as plt

df = vaex.open('data.hdf5')

# Create selections
df.select(df.category == 'A', name='group_a')
df.select(df.category == 'B', name='group_b')

# Plot both selections
df.viz.histogram(df.value, selection='group_a', label='Group A')
df.viz.histogram(df.value, selection='group_b', label='Group B')
plt.legend()
plt.show()

# 2D plot with selection
df.viz.heatmap(df.x, df.y, selection='group_a')
```

### Overlay Multiple Selections

```python
# heatmap blends several selections in one call (None = the full dataset);
# it has no alpha= or ax= parameters.
df.viz.heatmap(df.x, df.y, selection=[None, 'group_a', 'group_b'])

plt.title('Overlaid Selections')
plt.show()
```

## Subplots and Layouts

### Creating Multiple Plots

```python
import matplotlib.pyplot as plt

# Create subplot grid
fig, axes = plt.subplots(2, 3, figsize=(15, 10))

# Plot different variables
variables = ['x', 'y', 'z', 'a', 'b', 'c']
for idx, var in enumerate(variables):
    row = idx // 3
    col = idx % 3
    plt.sca(axes[row, col])
    df.viz.histogram(df[var], show=False)
    axes[row, col].set_title(f'Distribution of {var}')

plt.tight_layout()
plt.show()
```

### Faceted Plots

```python
# Plot by category
categories = df.category.unique()

fig, axes = plt.subplots(1, len(categories), figsize=(15, 5))

for idx, cat in enumerate(categories):
    df_cat = df[df.category == cat]
    plt.sca(axes[idx])
    df_cat.viz.heatmap(df_cat.x, df_cat.y, show=False)
    axes[idx].set_title(f'Category {cat}')

plt.tight_layout()
plt.show()
```

## Interactive Widgets (Jupyter)

Create interactive visualizations in Jupyter notebooks:

### Selection Widget

```python
# Interactive selection
df.widget.selection_expression()
```

### Heatmap Widget

```python
# Interactive 2-D heatmap with selection (requires vaex-jupyter widgets in a notebook).
# df.plot_widget() is deprecated in favour of the df.widget accessor.
df.widget.heatmap(df.x, df.y)
```

## Customization

### Styling Plots

```python
import matplotlib.pyplot as plt

# Create plot with custom styling
fig, ax = plt.subplots(figsize=(12, 8))

plt.sca(ax)
df.viz.heatmap(df.x, df.y,
        limits='99%',
        shape=(256, 256),
        colormap='plasma',
        colorbar_label='Density',   # heatmap draws its own colorbar
        show=False)

# Customize axes
ax.set_xlabel('X Variable', fontsize=14, fontweight='bold')
ax.set_ylabel('Y Variable', fontsize=14, fontweight='bold')
ax.set_title('Custom Density Plot', fontsize=16, fontweight='bold')
ax.grid(alpha=0.3)


plt.tight_layout()
plt.show()
```

### Figure Size and DPI

```python
# High-resolution plot (heatmap has no dpi= argument; set it when saving)
df.viz.heatmap(df.x, df.y, figsize=(12, 10))
plt.savefig('density.png', dpi=300)
```

## Specialized Visualizations

### Hexbin Plots

```python
# Alternative to heatmap using hexagonal bins
plt.figure(figsize=(10, 8))
plt.hexbin(df.x.values[:100000], df.y.values[:100000],
           gridsize=50, cmap='viridis')
plt.colorbar(label='Count')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()
```

### Contour Plots

```python
import numpy as np

# Get 2D histogram data
counts = df.count(binby=[df.x, df.y],
                  limits=[[0, 10], [0, 10]],
                  shape=(100, 100))

# Create contour plot
x = np.linspace(0, 10, 100)
y = np.linspace(0, 10, 100)
plt.contourf(x, y, counts.T, levels=20, cmap='viridis')
plt.colorbar(label='Count')
plt.xlabel('X')
plt.ylabel('Y')
plt.title('Contour Plot')
plt.show()
```

### Vector Field Overlay

```python
# Compute mean vectors on grid
mean_vx = df.mean(df.vx, binby=[df.x, df.y],
                  limits=[[0, 10], [0, 10]],
                  shape=(20, 20))
mean_vy = df.mean(df.vy, binby=[df.x, df.y],
                  limits=[[0, 10], [0, 10]],
                  shape=(20, 20))

# Create grid
x = np.linspace(0, 10, 20)
y = np.linspace(0, 10, 20)
X, Y = np.meshgrid(x, y)

# Plot
fig, ax = plt.subplots(figsize=(10, 8))

# Base heatmap
plt.sca(ax)
df.viz.heatmap(df.x, df.y, show=False)

# Vector overlay
ax.quiver(X, Y, mean_vx.T, mean_vy.T, alpha=0.7, color='white')

plt.show()
```

## Performance Considerations

### Optimizing Large Visualizations

```python
# For very large datasets, reduce shape
df.viz.heatmap(df.x, df.y, shape=(256, 256))  # Fast

# For publication quality
df.viz.heatmap(df.x, df.y, shape=(1024, 1024))  # Higher quality

# Balance quality and performance
df.viz.heatmap(df.x, df.y, shape=(512, 512))  # Good balance
```

### Caching Visualization Data

```python
# Compute once, plot multiple times
counts = df.count(binby=[df.x, df.y],
                  limits=[[0, 10], [0, 10]],
                  shape=(512, 512))

# Use in different plots
plt.figure()
plt.imshow(counts.T, origin='lower', cmap='viridis')
plt.colorbar()
plt.show()

plt.figure()
plt.imshow(np.log10(counts.T + 1), origin='lower', cmap='plasma')
plt.colorbar()
plt.show()
```

## Export and Saving

### Saving Figures

```python
# Save as PNG
df.viz.heatmap(df.x, df.y)
plt.savefig('plot.png', dpi=300, bbox_inches='tight')

# Save as PDF (vector)
plt.savefig('plot.pdf', bbox_inches='tight')

# Save as SVG
plt.savefig('plot.svg', bbox_inches='tight')
```

### Batch Plotting

```python
# Generate multiple plots
variables = ['x', 'y', 'z']

for var in variables:
    plt.figure(figsize=(10, 6))
    df.viz.histogram(df[var])
    plt.title(f'Distribution of {var}')
    plt.savefig(f'plot_{var}.png', dpi=300, bbox_inches='tight')
    plt.close()
```

## Common Patterns

### Pattern: Exploratory Data Analysis

```python
import matplotlib.pyplot as plt

# Create comprehensive visualization
fig = plt.figure(figsize=(16, 12))

# 1D histograms
ax1 = plt.subplot(3, 3, 1)
plt.sca(ax1)
df.viz.histogram(df.x, show=False)
ax1.set_title('X Distribution')

ax2 = plt.subplot(3, 3, 2)
plt.sca(ax2)
df.viz.histogram(df.y, show=False)
ax2.set_title('Y Distribution')

ax3 = plt.subplot(3, 3, 3)
plt.sca(ax3)
df.viz.histogram(df.z, show=False)
ax3.set_title('Z Distribution')

# 2D plots
ax4 = plt.subplot(3, 3, 4)
plt.sca(ax4)
df.viz.heatmap(df.x, df.y, show=False)
ax4.set_title('X vs Y')

ax5 = plt.subplot(3, 3, 5)
plt.sca(ax5)
df.viz.heatmap(df.x, df.z, show=False)
ax5.set_title('X vs Z')

ax6 = plt.subplot(3, 3, 6)
plt.sca(ax6)
df.viz.heatmap(df.y, df.z, show=False)
ax6.set_title('Y vs Z')

# Statistics on grids
ax7 = plt.subplot(3, 3, 7)
plt.sca(ax7)
df.viz.heatmap(df.x, df.y, what='mean(z)', show=False)
ax7.set_title('Mean Z on X-Y grid')

plt.tight_layout()
plt.savefig('eda_summary.png', dpi=300, bbox_inches='tight')
plt.show()
```

### Pattern: Comparison Across Groups

```python
# Compare distributions by category
categories = df.category.unique()

fig, axes = plt.subplots(len(categories), 2,
                         figsize=(12, 4 * len(categories)))

for idx, cat in enumerate(categories):
    df.select(df.category == cat, name=f'cat_{cat}')

    # 1D histogram
    plt.sca(axes[idx, 0])
    df.viz.histogram(df.value, selection=f'cat_{cat}', show=False)
    axes[idx, 0].set_title(f'Category {cat} - Distribution')

    # 2D plot
    plt.sca(axes[idx, 1])
    df.viz.heatmap(df.x, df.y, selection=f'cat_{cat}', show=False)
    axes[idx, 1].set_title(f'Category {cat} - X vs Y')

plt.tight_layout()
plt.show()
```

### Pattern: Time Series Visualization

```python
# Aggregate by time bins
df['year'] = df.timestamp.dt.year
df['month'] = df.timestamp.dt.month

# Plot time series
monthly_sales = df.groupby(['year', 'month']).agg({'sales': 'sum'})

plt.figure(figsize=(14, 6))
plt.plot(range(len(monthly_sales)), monthly_sales['sales'])
plt.xlabel('Time Period')
plt.ylabel('Sales')
plt.title('Sales Over Time')
plt.grid(alpha=0.3)
plt.show()
```

## Integration with Other Libraries

### Plotly for Interactivity

```python
import plotly.graph_objects as go

# Get data from Vaex
counts = df.count(binby=[df.x, df.y], shape=(100, 100))

# Create plotly figure
fig = go.Figure(data=go.Heatmap(z=counts.T))
fig.update_layout(title='Interactive Heatmap')
fig.show()
```

### Seaborn Style

```python
import seaborn as sns
import matplotlib.pyplot as plt

# Use seaborn styling
sns.set_style('darkgrid')
sns.set_palette('husl')

df.viz.histogram(df.value)
plt.show()
```

## Best Practices

1. **Use appropriate shape** - Balance resolution and performance (256-512 for exploration, 1024+ for publication)
2. **Apply sensible limits** - Use percentile-based limits ('99%', '99.7%') to handle outliers
3. **Choose color scales wisely** - Log scale for wide-ranging counts, linear for uniform data
4. **Leverage selections** - Compare subsets without creating new DataFrames
5. **Cache aggregations** - Compute once if creating multiple similar plots
6. **Use vector formats for publication** - Save as PDF or SVG for scalable figures
7. **Avoid sampling** - Vaex visualizations use all data, no sampling needed

## Troubleshooting

### Issue: Empty or Sparse Plot

```python
# Problem: Limits don't match data range
df.viz.heatmap(df.x, df.y, limits=[[0, 10], [0, 10]])

# Solution: Use automatic limits
df.viz.heatmap(df.x, df.y, limits='minmax')
df.viz.heatmap(df.x, df.y, limits='99%')
```

### Issue: Plot Too Slow

```python
# Problem: Too high resolution
df.viz.heatmap(df.x, df.y, shape=(2048, 2048))

# Solution: Reduce shape
df.viz.heatmap(df.x, df.y, shape=(512, 512))
```

### Issue: Can't See Low-Density Regions

```python
# Problem: Linear scale overwhelmed by high-density areas
df.viz.heatmap(df.x, df.y, f='identity')

# Solution: Use logarithmic scale
df.viz.heatmap(df.x, df.y, f='log')
```

## Related Resources

- For data aggregation: See `data_processing.md`
- For performance optimization: See `performance.md`
- For DataFrame basics: See `core_dataframes.md`