Claude Skill

geo-data-engineering

Always invoke when geospatial data must be acquired, prepared, repaired, scaled, or moved through a repeatable pipeline. Covers open-data/OSM/STAC acquisition, spatial formats, CRS transforms, quality checks, and batch ETL architecture for growing or recurring joins. Invoke along

LLM Mart · 0 points · 0 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download muend-geoai-skills-skills_geo-data-engineering-096e5d4.zip · 5 KB

muend/geoai-skills 24 2 forks MIT Updated 21d ago

Part of muend/geoai-skills — 18 skills

Install

skills CLI npx skills add https://github.com/muend/geoai-skills/tree/main/skills/geo-data-engineering

Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install muend-geoai-skills@llmmart

Git git clone https://github.com/muend/geoai-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole muend/geoai-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Geospatial Data Engineering

Purpose: get spatial data into a clean, validated, analysis-ready state with a repeatable pipeline — the stage where most real-world GIS time is spent and most silent errors are born.

Format selection

Format	Use for	Avoid because
GeoParquet	Analysis interchange, big vector, columnar workflows	Not yet readable by some legacy desktop GIS
GeoPackage	Desktop GIS exchange, multi-layer projects	Slower than Parquet at scale; SQLite locking
FlatGeobuf	Streaming, HTTP range reads	Single layer
COG (Cloud-Optimized GeoTIFF)	All raster deliverables	— (make every GeoTIFF a COG)
Zarr/NetCDF	Multi-dimensional (time × band × y × x)	Overkill for single rasters
Shapefile	Only when a legacy tool demands it	10-char columns, 2 GB cap, encoding chaos, multi-file fragility
CSV + WKT/lon-lat	Simple point exchange	No CRS metadata — document it explicitly

Acquisition playbook

OpenStreetMap: small areas → osmnx; large extracts → Geofabrik PBF + pyrosm/osmium. Respect tag heterogeneity: always inspect tag value distributions before filtering.
Buildings/places at scale: Overture Maps (GeoParquet on S3/Azure, query with DuckDB spatial — often the fastest path).
Satellite/raster: STAC APIs via pystac-client + odc-stac — see remote-sensing-analysis; planetary archives → google-earth-engine.
Boundaries: authoritative national source first; Natural Earth / GADM / geoBoundaries for global work — record which, versions differ materially.
Record every acquisition: source URL, query parameters, retrieval date, license. Put it in a DATA_SOURCES.md next to the data.

CRS engineering

Store in EPSG:4326 or source CRS; analyze in a projected CRS suited to the extent: local UTM zone (gdf.estimate_utm_crs()), national grid, or equal-area (EPSG:6933/Mollweide) for cross-region area stats.
Datum shifts matter at sub-meter precision: transformations between datums need the right transformation grid (pyproj.network.set_network_enabled(True) when accuracy matters).
Never strip or overwrite a CRS to "fix" misaligned layers — diagnose which layer is wrong with a known landmark instead.

Cleaning pipeline

Run scripts/clean_vector.py (or import its clean_vector() function) as the standard hygiene pass: drops empty/null geometries, repairs invalid ones with make_valid, de-duplicates, reprojects, and prints an accounting report so silent data loss is impossible.

Then: normalize text attributes (trim, collapse whitespace, locale-aware casefold — beware Turkish İ/ı, German ß), coerce dtypes explicitly, and show value_counts() of every categorical you will later filter on.

Scale strategies

Fits in RAM: GeoPandas + Shapely 2 vectorized ops. Ensure the spatial index is used (sjoin, query_bulk) — hand-rolled loops are O(n²).
Bigger than RAM, single machine: DuckDB spatial extension over GeoParquet (predicate pushdown + spatial SQL), or dask-geopandas.
Served / concurrent / transactional: PostGIS — see postgis-spatial-sql.
Rasters: windowed reads (rasterio.windows), chunked xarray + dask; never read() a 50 GB mosaic into memory.

Pipeline standards

Idempotent steps with explicit inputs/outputs on disk; re-running never corrupts state.
Checkpoint after expensive stages (download, big join) in GeoParquet/GPKG.
Log an accounting line per stage: rows/features/pixels in → out.
Deterministic ordering before writing (sort by stable key) so diffs are meaningful.

Pitfalls checklist

CSV opened without declaring lon/lat columns' CRS.
Shapefile column names silently truncated on export.
Encoding mojibake from legacy files (try encoding="utf-8" then cp1252).
Mixed geometry types in one layer (Polygon + MultiPolygon breaks some tools — normalize with .explode() or promote to Multi*).
Antimeridian and pole-crossing geometries after naive reprojection.
Downloaded "latest" data with no recorded version/date — unreproducible.

Execution contract

Workflow: inventory sources and contracts; acquire with provenance; inspect CRS, schema, geometry, and scale; clean deterministically; validate; write an analysis-ready artifact.
Decision rules: select formats and engines from size, geometry, concurrency, and downstream access needs; never infer CRS or destructive repairs silently.
Verification protocol: reconcile feature or pixel counts at every stage, assert CRS and geometry invariants, sample outputs spatially, and rerun to confirm idempotence.
Failure modes: quarantine ambiguous CRS, mixed units, invalid encodings, lossy format conversions, or unexplained row loss instead of guessing.
Deliverables: validated dataset, machine-readable schema and CRS, provenance manifest, accounting log, rejected-record report, and reproducible pipeline.
Source freshness: consult the authoritative source registry before using version-sensitive formats or APIs and record the checked date.

Files (geoai-skills)

agents

openai.yaml 220 B

interface:
  display_name: "Geospatial Data Engineering"
  short_description: "Build reliable geospatial data pipelines"
  default_prompt: "Use $geo-data-engineering to inspect, clean, and prepare this spatial dataset."

references

authoritative-sources.md 811 B

# Authoritative sources

- Last verified: 2026-07-19
- Review cadence: every 3 months
- Refresh triggers: GDAL, PROJ, GeoParquet, or STAC specification release

## Canonical sources

- [GeoParquet specification](https://geoparquet.org/releases/v1.1.0/) — interoperable vector metadata and geometry encoding.
- [GDAL documentation](https://gdal.org/en/stable/) — format drivers and transformation behavior.
- [PROJ documentation](https://proj.org/en/stable/) — coordinate operations and grid requirements.
- [STAC specification](https://github.com/radiantearth/stac-spec) — catalog and asset metadata contracts.

Pin concrete format/specification versions in pipelines. Verify driver capability against the deployed GDAL/PROJ build rather than assuming the latest documentation matches runtime behavior.

scripts

clean_vector.py 3.6 KB

"""Standard vector hygiene pass with loud accounting.

Run:    python clean_vector.py input.gpkg output.parquet 32636
Import: from clean_vector import clean_vector
"""
from __future__ import annotations

import argparse
from pathlib import Path

import geopandas as gpd
from pyproj import CRS
from shapely import make_valid


def clean_vector(gdf: gpd.GeoDataFrame, target_epsg: int) -> gpd.GeoDataFrame:
    """Validity, emptiness, duplicates, CRS — returns a cleaned copy.

    Prints an accounting report of every change so silent data loss is
    impossible.

    Args:
        gdf: Input GeoDataFrame (any CRS, must be defined).
        target_epsg: EPSG code of the projected CRS to analyze in.

    Returns:
        Cleaned GeoDataFrame in the target CRS. Only exact duplicate features
        (same attributes and same geometry) are removed.

    Raises:
        ValueError: If the input CRS is undefined, the target CRS is not
            projected, or geometry repair leaves invalid features.
    """
    if gdf.crs is None:
        raise ValueError("Input CRS undefined — resolve it before cleaning.")

    target_crs = CRS.from_epsg(target_epsg)
    if not target_crs.is_projected:
        raise ValueError(
            f"Target EPSG:{target_epsg} is not projected — choose a CRS with "
            "linear units for analysis."
        )

    n0 = len(gdf)
    result = gdf.copy()

    missing_or_empty = result.geometry.isna() | result.geometry.is_empty
    removed_missing = int(missing_or_empty.sum())
    result = result.loc[~missing_or_empty].copy()

    invalid = ~result.geometry.is_valid
    repaired = int(invalid.sum())
    if repaired:
        result.loc[invalid, result.geometry.name] = result.loc[
            invalid, result.geometry.name
        ].apply(make_valid)

    invalid_after = ~result.geometry.is_valid
    if invalid_after.any():
        raise ValueError(
            f"Geometry repair left {int(invalid_after.sum())} invalid features."
        )

    empty_after_repair = result.geometry.isna() | result.geometry.is_empty
    removed_after_repair = int(empty_after_repair.sum())
    result = result.loc[~empty_after_repair].copy()

    # Include geometry in the duplicate key. Attribute-only deduplication can
    # silently delete distinct features that happen to share the same fields.
    duplicate_subset = list(result.columns)
    exact_duplicates = result.duplicated(subset=duplicate_subset, keep="first")
    removed_duplicates = int(exact_duplicates.sum())
    result = result.loc[~exact_duplicates].copy()

    result = result.to_crs(target_crs)
    geometry_types = ",".join(sorted(result.geometry.geom_type.unique()))
    print(
        f"rows {n0} -> {len(result)} | removed null/empty {removed_missing} "
        f"| repaired {repaired} | removed after repair {removed_after_repair} "
        f"| removed exact duplicates {removed_duplicates} "
        f"| geometry types {geometry_types or 'none'} "
        f"| CRS -> EPSG:{target_epsg}"
    )
    return result


def main() -> None:
    """Run the vector hygiene pass from the command line."""
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("source", type=Path)
    parser.add_argument("destination", type=Path)
    parser.add_argument("target_epsg", type=int)
    args = parser.parse_args()

    output = clean_vector(gpd.read_file(args.source), args.target_epsg)
    if args.destination.suffix.lower() == ".parquet":
        output.to_parquet(args.destination)
    else:
        output.to_file(args.destination)
    print(f"wrote {args.destination}")


if __name__ == "__main__":
    main()

SKILL.md 5.6 KB

---
name: geo-data-engineering
description: >-
  Always invoke when geospatial data must be acquired, prepared, repaired,
  scaled, or moved through a repeatable pipeline. Covers open-data/OSM/STAC
  acquisition, spatial formats, CRS transforms, quality checks, and batch ETL
  architecture for growing or recurring joins. Invoke alongside PostGIS for
  database execution and alongside SWE standards when code is delivered. Do
  not trigger merely because another specialist reads analysis-ready data.
license: MIT
metadata:
  author: Muhammed Enes Duran
---

# Geospatial Data Engineering

Purpose: get spatial data into a clean, validated, analysis-ready state with
a repeatable pipeline — the stage where most real-world GIS time is spent
and most silent errors are born.

## Format selection

| Format | Use for | Avoid because |
|---|---|---|
| **GeoParquet** | Analysis interchange, big vector, columnar workflows | Not yet readable by some legacy desktop GIS |
| **GeoPackage** | Desktop GIS exchange, multi-layer projects | Slower than Parquet at scale; SQLite locking |
| **FlatGeobuf** | Streaming, HTTP range reads | Single layer |
| **COG** (Cloud-Optimized GeoTIFF) | All raster deliverables | — (make every GeoTIFF a COG) |
| **Zarr/NetCDF** | Multi-dimensional (time × band × y × x) | Overkill for single rasters |
| Shapefile | Only when a legacy tool demands it | 10-char columns, 2 GB cap, encoding chaos, multi-file fragility |
| CSV + WKT/lon-lat | Simple point exchange | No CRS metadata — document it explicitly |

## Acquisition playbook

- **OpenStreetMap**: small areas → `osmnx`; large extracts → Geofabrik PBF +
  `pyrosm`/`osmium`. Respect tag heterogeneity: always inspect tag value
  distributions before filtering.
- **Buildings/places at scale**: Overture Maps (GeoParquet on S3/Azure,
  query with DuckDB spatial — often the fastest path).
- **Satellite/raster**: STAC APIs via `pystac-client` + `odc-stac` — see
  `remote-sensing-analysis`; planetary archives → `google-earth-engine`.
- **Boundaries**: authoritative national source first; Natural Earth / GADM /
  geoBoundaries for global work — record which, versions differ materially.
- Record every acquisition: source URL, query parameters, retrieval date,
  license. Put it in a `DATA_SOURCES.md` next to the data.

## CRS engineering

- Store in EPSG:4326 or source CRS; **analyze** in a projected CRS suited to
  the extent: local UTM zone (`gdf.estimate_utm_crs()`), national grid, or
  equal-area (EPSG:6933/Mollweide) for cross-region area stats.
- Datum shifts matter at sub-meter precision: transformations between datums
  need the right transformation grid (`pyproj.network.set_network_enabled(True)`
  when accuracy matters).
- Never strip or overwrite a CRS to "fix" misaligned layers — diagnose which
  layer is wrong with a known landmark instead.

## Cleaning pipeline

Run `scripts/clean_vector.py` (or import its `clean_vector()` function) as
the standard hygiene pass: drops empty/null geometries, repairs invalid ones
with `make_valid`, de-duplicates, reprojects, and **prints an accounting
report** so silent data loss is impossible.

Then: normalize text attributes (trim, collapse whitespace, locale-aware
casefold — beware Turkish İ/ı, German ß), coerce dtypes explicitly, and show
`value_counts()` of every categorical you will later filter on.

## Scale strategies

- **Fits in RAM**: GeoPandas + Shapely 2 vectorized ops. Ensure the spatial
  index is used (`sjoin`, `query_bulk`) — hand-rolled loops are O(n²).
- **Bigger than RAM, single machine**: DuckDB `spatial` extension over
  GeoParquet (predicate pushdown + spatial SQL), or `dask-geopandas`.
- **Served / concurrent / transactional**: PostGIS — see `postgis-spatial-sql`.
- Rasters: windowed reads (`rasterio.windows`), chunked xarray + dask;
  never `read()` a 50 GB mosaic into memory.

## Pipeline standards

- Idempotent steps with explicit inputs/outputs on disk; re-running never
  corrupts state.
- Checkpoint after expensive stages (download, big join) in GeoParquet/GPKG.
- Log an accounting line per stage: rows/features/pixels in → out.
- Deterministic ordering before writing (sort by stable key) so diffs are
  meaningful.

## Pitfalls checklist

- CSV opened without declaring lon/lat columns' CRS.
- Shapefile column names silently truncated on export.
- Encoding mojibake from legacy files (try `encoding="utf-8"` then cp1252).
- Mixed geometry types in one layer (Polygon + MultiPolygon breaks some
  tools — normalize with `.explode()` or promote to Multi*).
- Antimeridian and pole-crossing geometries after naive reprojection.
- Downloaded "latest" data with no recorded version/date — unreproducible.

## Execution contract

- **Workflow:** inventory sources and contracts; acquire with provenance; inspect CRS, schema, geometry, and scale; clean deterministically; validate; write an analysis-ready artifact.
- **Decision rules:** select formats and engines from size, geometry, concurrency, and downstream access needs; never infer CRS or destructive repairs silently.
- **Verification protocol:** reconcile feature or pixel counts at every stage, assert CRS and geometry invariants, sample outputs spatially, and rerun to confirm idempotence.
- **Failure modes:** quarantine ambiguous CRS, mixed units, invalid encodings, lossy format conversions, or unexplained row loss instead of guessing.
- **Deliverables:** validated dataset, machine-readable schema and CRS, provenance manifest, accounting log, rejected-record report, and reproducible pipeline.
- **Source freshness:** consult [the authoritative source registry](references/authoritative-sources.md) before using version-sensitive formats or APIs and record the checked date.