{"slug":"alterlab-polars","title":"alterlab-polars","summary":"Fast in-memory DataFrame analytics with Polars — lazy evaluation, parallel execution, and an Apache Arrow backend for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory, for 1-100GB datasets, ETL pipelines, or a faster pandas replacement. For larg","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-23T18:57:04.149819Z","repo":{"url":"https://github.com/AlterLab-IEU/AlterLab-Academic-Skills","stars":68,"forks":13,"license":"MIT","updatedAt":"2026-09-23T13:42:59Z"},"bodyHtml":"
\n

name: alterlab-polars\ndescription: Fast in-memory DataFrame analytics with Polars — lazy evaluation, parallel execution, and an Apache Arrow backend for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory, for 1-100GB datasets, ETL pipelines, or a faster pandas replacement. For larger-than-RAM data prefer dask or vaex. Part of the AlterLab Academic Skills suite.\nlicense: MIT\nallowed-tools: Read Write Edit Bash(python:) Bash(uv:)\ncompatibility: No API key required. Runs locally via uv run python; requires polars >= 1.0 (current 1.44 as of 2026-09; 2.0 is in release candidate).\nmetadata:\nskill-author: AlterLab\nversion: \"1.0.1\"\nlast_updated: \"2026-09-23\"

\n

Polars

\n

Overview

\n

Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.

\n

When to Use This Skill

\n\n

Does NOT Trigger

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ScenarioUse Instead
Distributed execution across a cluster, or scaling existing pandas/NumPy code with a task scheduler and dashboardalterlab-dask
Billion-row out-of-core exploration with memory-mapped HDF5/Arrow files and big-data plotsalterlab-vaex
First look at an unfamiliar data file: structure, missingness, and quality reportalterlab-eda
\n

Quick Start

\n

Installation and Basic Usage

\n

Pin a recent 1.x (examples here use the Polars 1.x API):

\n
uv add 'polars>=1.0,<2'\n
\n

Polars 2.0 is in release candidate (2.0.0rc2, September 2026). Its upgrade guide lists changes\nthat alter results silently: LazyFrame.collect() defaults to the streaming engine (row order\nof some joins/unpivots can change), pl.concat(how=\"horizontal\") requires equal heights\n(how=\"horizontal_extend\" pads), explode() drops empty lists instead of emitting a null row,\nis_in refuses lossy casts, LazyFrame.profile() is removed, and headerless CSV columns are\nnamed from column_0. Check https://docs.pola.rs/releases/upgrade/ before moving to 2.x.

\n

Basic DataFrame creation and operations:

\n
import polars as pl\n\n# Create DataFrame\ndf = pl.DataFrame({\n    \"name\": [\"Alice\", \"Bob\", \"Charlie\"],\n    \"age\": [25, 30, 35],\n    \"city\": [\"NY\", \"LA\", \"SF\"]\n})\n\n# Select columns\ndf.select(\"name\", \"age\")\n\n# Filter rows\ndf.filter(pl.col(\"age\") > 25)\n\n# Add computed columns\ndf.with_columns(\n    age_plus_10=pl.col(\"age\") + 10\n)\n
\n

Core Concepts

\n

Expressions

\n

Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.

\n

Key principles:

\n\n

Example:

\n
# Expression-based computation\ndf.select(\n    pl.col(\"name\"),\n    (pl.col(\"age\") * 12).alias(\"age_in_months\")\n)\n
\n

Lazy vs Eager Evaluation

\n

Eager (DataFrame): Operations execute immediately

\n
df = pl.read_csv(\"file.csv\")  # Reads immediately\nresult = df.filter(pl.col(\"age\") > 25)  # Executes immediately\n
\n

Lazy (LazyFrame): Operations build a query plan, optimized before execution

\n
lf = pl.scan_csv(\"file.csv\")  # Doesn't read yet\nresult = lf.filter(pl.col(\"age\") > 25).select(\"name\", \"age\")\ndf = result.collect()  # Now executes optimized query\n
\n

When to use lazy:

\n\n

Benefits of lazy evaluation:

\n\n

For detailed concepts, load references/core_concepts.md.

\n

Common Operations

\n

Select

\n

Select and manipulate columns:

\n
# Select specific columns\ndf.select(\"name\", \"age\")\n\n# Select with expressions\ndf.select(\n    pl.col(\"name\"),\n    (pl.col(\"age\") * 2).alias(\"double_age\")\n)\n\n# Select all columns matching a pattern\ndf.select(pl.col(\"^.*_id$\"))\n
\n

Filter

\n

Filter rows by conditions:

\n
# Single condition\ndf.filter(pl.col(\"age\") > 25)\n\n# Multiple conditions (cleaner than using &)\ndf.filter(\n    pl.col(\"age\") > 25,\n    pl.col(\"city\") == \"NY\"\n)\n\n# Complex conditions\ndf.filter(\n    (pl.col(\"age\") > 25) | (pl.col(\"city\") == \"LA\")\n)\n
\n

With Columns

\n

Add or modify columns while preserving existing ones:

\n
# Add new columns\ndf.with_columns(\n    age_plus_10=pl.col(\"age\") + 10,\n    name_upper=pl.col(\"name\").str.to_uppercase()\n)\n\n# Parallel computation (all columns computed in parallel)\ndf.with_columns(\n    pl.col(\"value\") * 10,\n    pl.col(\"value\") * 100,\n)\n
\n

Group By and Aggregations

\n

Group data and compute aggregations:

\n
# Basic grouping\ndf.group_by(\"city\").agg(\n    pl.col(\"age\").mean().alias(\"avg_age\"),\n    pl.len().alias(\"count\")\n)\n\n# Multiple group keys\ndf.group_by(\"city\", \"department\").agg(\n    pl.col(\"salary\").sum()\n)\n\n# Conditional aggregations\ndf.group_by(\"city\").agg(\n    (pl.col(\"age\") > 30).sum().alias(\"over_30\")\n)\n
\n

For detailed operation patterns, load references/operations.md.

\n

Aggregations and Window Functions

\n

Aggregation Functions

\n

Common aggregations within group_by context:

\n\n

Window Functions with over()

\n

Apply aggregations while preserving row count:

\n
# Add group statistics to each row\ndf.with_columns(\n    avg_age_by_city=pl.col(\"age\").mean().over(\"city\"),\n    rank_in_city=pl.col(\"salary\").rank().over(\"city\")\n)\n\n# Multiple grouping columns\ndf.with_columns(\n    group_avg=pl.col(\"value\").mean().over(\"category\", \"region\")\n)\n
\n

Mapping strategies:

\n\n

Data I/O

\n

Supported Formats

\n

Polars supports reading and writing:

\n\n

Common I/O Operations

\n

CSV:

\n
# Eager\ndf = pl.read_csv(\"file.csv\")\ndf.write_csv(\"output.csv\")\n\n# Lazy (preferred for large files)\nlf = pl.scan_csv(\"file.csv\")\nresult = lf.filter(...).select(...).collect()\n
\n

Parquet (recommended for performance):

\n
df = pl.read_parquet(\"file.parquet\")\ndf.write_parquet(\"output.parquet\")\n
\n

JSON:

\n
df = pl.read_json(\"file.json\")\ndf.write_json(\"output.json\")\n
\n

For comprehensive I/O documentation, load references/io_guide.md.

\n

Transformations

\n

Joins

\n

Combine DataFrames:

\n
# Inner join\ndf1.join(df2, on=\"id\", how=\"inner\")\n\n# Left join\ndf1.join(df2, on=\"id\", how=\"left\")\n\n# Join on different column names\ndf1.join(df2, left_on=\"user_id\", right_on=\"id\")\n
\n

Concatenation

\n

Stack DataFrames:

\n
# Vertical (stack rows)\npl.concat([df1, df2], how=\"vertical\")\n\n# Horizontal (add columns)\npl.concat([df1, df2], how=\"horizontal\")\n\n# Diagonal (union with different schemas)\npl.concat([df1, df2], how=\"diagonal\")\n
\n

Pivot and Unpivot

\n

Reshape data:

\n
# Pivot (wide format): on= is the column whose values become new columns\ndf.pivot(\"product\", index=\"date\", values=\"sales\")\n\n# Unpivot (long format)\ndf.unpivot(index=\"id\", on=[\"col1\", \"col2\"])\n
\n

For detailed transformation examples, load references/transformations.md.

\n

Pandas Migration

\n

Polars offers significant performance improvements over pandas with a cleaner API. Key differences:

\n

Conceptual Differences

\n\n

Common Operation Mappings

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
OperationPandasPolars
Select columndf[\"col\"]df.select(\"col\")
Filterdf[df[\"col\"] > 10]df.filter(pl.col(\"col\") > 10)
Add columndf.assign(x=...)df.with_columns(x=...)
Group bydf.groupby(\"col\").agg(...)df.group_by(\"col\").agg(...)
Windowdf.groupby(\"col\").transform(...)df.with_columns(...).over(\"col\")
\n

Key Syntax Patterns

\n

Pandas sequential (slow):

\n
df.assign(\n    col_a=lambda df_: df_.value * 10,\n    col_b=lambda df_: df_.value * 100\n)\n
\n

Polars parallel (fast):

\n
df.with_columns(\n    col_a=pl.col(\"value\") * 10,\n    col_b=pl.col(\"value\") * 100,\n)\n
\n

For comprehensive migration guide, load references/pandas_migration.md.

\n

Best Practices

\n

Polars 1.x API notes (renamed since 0.x)

\n

These older names appear in stale tutorials and LLM training data; use the 1.x form:

\n\n

Performance Optimization

\n
    \n

  1. Use lazy evaluation for large datasets:

    \n
    lf = pl.scan_csv(\"large.csv\")  # Don't use read_csv\nresult = lf.filter(...).select(...).collect()\n
    \n
    2. \n

  2. Avoid Python functions in hot paths:

    \n\n
    3. \n

  3. Use the streaming engine to lower peak memory:

    \n
    lf.collect(engine=\"streaming\")  # `streaming=True` is deprecated\n
    \n
    4. \n

  4. Select only needed columns early:

    \n
    # Good: Select columns early\nlf.select(\"col1\", \"col2\").filter(...)\n\n# Bad: Filter on all columns first\nlf.filter(...).select(\"col1\", \"col2\")\n
    \n
    5. \n

  5. Use appropriate data types:

    \n\n
    6. \n
\n

Expression Patterns

\n

Conditional operations:

\n
pl.when(condition).then(value).otherwise(other_value)\n
\n

Column operations across multiple columns:

\n
df.select(pl.col(\"^.*_value$\") * 2)  # Regex pattern\n
\n

Null handling:

\n
pl.col(\"x\").fill_null(0)\npl.col(\"x\").is_null()\npl.col(\"x\").drop_nulls()\n
\n

For additional best practices and patterns, load references/best_practices.md.

\n

Resources

\n

This skill includes comprehensive reference documentation:

\n

references/

\n\n

Load these references as needed when users require detailed information about specific topics.

\n","files":[{"path":"evals/evals.json","sizeBytes":5559,"isText":true},{"path":"references/best_practices.md","sizeBytes":14479,"isText":true},{"path":"references/core_concepts.md","sizeBytes":8642,"isText":true},{"path":"references/io_guide.md","sizeBytes":12156,"isText":true},{"path":"references/operations.md","sizeBytes":12379,"isText":true},{"path":"references/pandas_migration.md","sizeBytes":12345,"isText":true},{"path":"references/transformations.md","sizeBytes":11391,"isText":true},{"path":"SKILL.md","sizeBytes":11878,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-23T18:58:46.616837Z","sha256":"A10824799D138AFE852D1363EA677FED415C0C2322E922B51B5F3EAA3BD09F46","sizeBytes":30943},"review":null,"source":{"repositoryUrl":"https://github.com/AlterLab-IEU/AlterLab-Academic-Skills","path":"skills/data-science/alterlab-polars","license":"MIT","commit":"e4836c08a20da195a11f30f203a8cf23ec30aa95","subtreeSha":"69DDAD482A3C7E532104783A0187319EE94B48D433285649B1353640F89A07E9","lastSyncedAt":"2026-09-23T18:56:52.297238Z"},"reviewedAt":"2026-09-23T19:02:10.537716Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/tree/main/skills/data-science/alterlab-polars"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install alterlab-ieu-alterlab-academic-skills@llmmart"},{"target":"git","command":"git clone https://github.com/AlterLab-IEU/AlterLab-Academic-Skills.git"}]}