Claude Skill

public-data-access

Plan, configure, validate, and document portable public-bioinformatics data acquisition. Use for GEO/GSE/GDS, SRA/ENA, TCGA/GDC, GTEx, DepMap, public expression matrices, raw reads, release files, manifests, resumable downloads, and reusable local caches. Keep the workflow provid

LLM Mart · 0 points · 13 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download xuzhougeng-wisp-science-skills_public-data-access-a3f7f7b.zip · 13 KB
Part of xuzhougeng/wisp-science — 25 skills

Install

skills CLI npx skills add https://github.com/xuzhougeng/wisp-science/tree/main/skills/public-data-access
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install xuzhougeng-wisp-science@llmmart
Git git clone https://github.com/xuzhougeng/wisp-science.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole xuzhougeng/wisp-science collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Public Bioinformatics Data Access

Build a reproducible acquisition plan before downloading. Treat GEO, SRA/ENA, GDC, GTEx, and DepMap as independent providers behind one provider-neutral workflow. Do not require a provider-specific toolkit or machine-specific checkout.

Workflow

  1. Clarify the dataset contract. Identify the provider, accession/project, release, modality, smallest useful data product, filters, target directory, expected scale, and downstream analysis.
  2. Inspect before transfer. Use available MCP/connectors or official metadata endpoints to list releases, files, samples, sizes, and checksums. Do not start a bulk transfer during discovery.
  3. Write a provider-neutral plan. Run scripts/public_data_plan.py init. Store the plan next to the future dataset as download-plan.json.
  4. Validate and review. Run validate, show the user the resolved provider, transport, filters, limits, output location, and known size. For a large, paid, authenticated, or overwrite-capable job, confirm that the user's authorization covers this concrete transfer; ask only when it does not.
  5. Select the adapter at runtime. Prefer an already available Wisp MCP tool for metadata and small queries. Prefer official HTTPS/FTP or provider clients for bulk files. Use an external project only when it is installed and record its version in the plan/manifest.
  6. Acquire safely. Reuse existing valid files, resume partial transfers when supported, keep raw files immutable, and never place credentials in the plan.
  7. Verify and hand off. Check expected files, byte sizes, checksums when available, and sample/file counts. Generate manifest.json with the script.

Provider routing

Provider Discovery and small queries Bulk acquisition Typical products
GEO GEO metadata connector, NCBI E-utilities NCBI GEO HTTPS/FTP; optional geokit in R series matrix, SOFT, supplementary files
SRA/ENA RunInfo or ENA Portal API ENA HTTPS/FTP or SRA Toolkit FASTQ, run metadata
GDC GDC files/cases API manifest + gdc-client, or HTTPS for bounded files expression, mutation, CNV, clinical, methylation
GTEx GTEx expression connector/API official release files for matrices gene/tissue queries, median or sample expression
DepMap DepMap model/release metadata official release file endpoint model metadata, expression, mutation, dependency
custom User-provided catalog/API explicit HTTPS/FTP URLs provider-specific files

Read references/provider-routing.md before implementing or changing a provider adapter. DepMap-specific flags or release semantics must stay inside the DepMap adapter; they must not shape the common plan schema.

For GEO SOFT/Series Matrix parsing, sample metadata preparation, or ExpressionSet acquisition in an R workflow, read references/geokit.md. geokit is optional; ordinary GEO discovery does not require R or package installation.

Create and validate a plan

Resolve scripts/public_data_plan.py against this skill's directory (the use_skill result lists its path), and invoke that resolved script with a Python 3.10+ interpreter. Keep the working directory at the project root so relative plan/output paths belong to the project. The examples below abbreviate the script path; quote the resolved path when it contains spaces. In an SSH/WSL context, stage the helper there or use an existing copy in that context; a desktop skill path is not automatically available remotely.

python scripts/public_data_plan.py init \
  --provider geo \
  --identifier GSE12345 \
  --data-type series-matrix \
  --output-dir data/public/geo/GSE12345 \
  --plan data/public/geo/GSE12345/download-plan.json

python scripts/public_data_plan.py validate \
  data/public/geo/GSE12345/download-plan.json

Filters are provider-specific but encoded uniformly as repeated key=value pairs:

python scripts/public_data_plan.py init \
  --provider gdc \
  --identifier TCGA-BRCA \
  --data-type expression \
  --filter workflow_type="STAR - Counts" \
  --filter sample_type="Primary Tumor" \
  --max-files 20 \
  --transport gdc-client \
  --plan data/public/gdc/TCGA-BRCA/download-plan.json

The planner does not download data. It produces a reviewable contract. See references/download-plan-schema.md for the complete schema. Validation checks the plan structure; it does not probe URLs, enforce transfer limits, verify installed packages, or approve a pending transfer. The selected adapter must honor the plan's limits and resume behavior.

Generate a manifest

After acquisition:

python scripts/public_data_plan.py manifest \
  data/public/geo/GSE12345/download-plan.json \
  --scan-dir data/public/geo/GSE12345 \
  --output data/public/geo/GSE12345/manifest.json

Use SHA-256 for modest datasets and provider checksums for large archives. For very large datasets, --checksum none is acceptable only when official checksums or immutable object identifiers are recorded elsewhere.

Safety and reproducibility rules

  • Default to overwrite=false, resume=true, and the minimum useful subset.
  • Never translate an exploratory request into “download everything.”
  • Keep provider metadata, query/filter payloads, release/version, transport, tool version, URLs/object identifiers, and validation results.
  • Separate immutable source files from normalized/derived outputs.
  • Do not treat a successful HTTP response as a valid dataset; verify content.
  • Do not embed API keys, cookies, signed URLs, SSH keys, or bearer tokens.
  • Use structured runs or a remote execution context for long transfers rather than extending an interactive shell timeout.
  • If an adapter or connector cannot perform the requested transfer, stop after producing the validated plan and report the missing capability explicitly.

Wisp Science integration

  • Discover the live connector/tool catalog instead of assuming exact MCP tool names; installations can expose different provider adapters.
  • Use connectors for discovery and bounded queries, then official transfer mechanisms for large files.
  • Keep outputs under the active project, normally data/public/<provider>/....
  • Invoke the planner as a standalone CLI; no Python REPL helper loading is required. R-based acquisition can use geokit independently of the planner.
  • Treat this skill as an acquisition/orchestration layer. Downstream QC, statistics, annotation, and visualization belong to other skills.
Files (wisp-science)
  • references
    • download-plan-schema.md 2.1 KB
      # Download plan schema
      
      The plan is a provider-neutral JSON contract. Provider adapters may add filter
      keys, but must not add provider-specific top-level fields.
      
      ## Top-level fields
      
      | Field | Meaning |
      |---|---|
      | `schema_version` | Integer schema version; currently `1` |
      | `created_at` | UTC creation timestamp |
      | `dataset` | Provider, identifier, data type, release, and filters |
      | `acquisition` | Transport, resume/overwrite behavior, limits, checksum policy |
      | `output` | Dataset directory and manifest path |
      | `approval` | Whether user review is required and current status |
      | `provenance` | Adapter/tool metadata populated during execution |
      | `notes` | Optional human-readable constraints |
      
      ## Example
      
      ```json
      {
        "schema_version": 1,
        "created_at": "2026-07-17T09:00:00Z",
        "dataset": {
          "provider": "gdc",
          "identifier": "TCGA-BRCA",
          "data_type": "expression",
          "release": null,
          "filters": {
            "sample_type": "Primary Tumor",
            "workflow_type": "STAR - Counts"
          }
        },
        "acquisition": {
          "transport": "gdc-client",
          "resume": true,
          "overwrite": false,
          "max_files": 20,
          "max_bytes": null,
          "checksum": "sha256"
        },
        "output": {
          "directory": "data/public/gdc/TCGA-BRCA",
          "manifest": "data/public/gdc/TCGA-BRCA/manifest.json"
        },
        "approval": {
          "required": true,
          "status": "pending"
        },
        "provenance": {
          "adapter": null,
          "adapter_version": null,
          "query_url": null
        },
        "notes": []
      }
      ```
      
      ## Status rules
      
      - `pending`: plan exists but transfer has not been approved.
      - `approved`: user or an authorized workflow approved the reviewed plan.
      - `rejected`: plan must not execute.
      
      Changing `pending` to `approved` is an authorization event, not an automatic
      planner action. Record the approval in the surrounding run/session history.
      
      ## Manifest contract
      
      The generated manifest contains the plan digest, scan root, aggregate file and
      byte counts, and one entry per file. File entries use paths relative to the scan
      root and optionally include SHA-256. Provider-native identifiers/checksums may
      be added later without changing the plan.
      
    • geokit.md 7 KB
      # GEO acquisition with geokit
      
      Use this reference for the optional R adapter within `public-data-access`.
      geokit is not a bundled Wisp MCP server. Discover the current Wisp tool catalog
      before choosing operations; this guide does not introduce new tool names.
      
      ## Choose the operation
      
      | Need | Route |
      |---|---|
      | Search studies or inspect a few series | Existing GEO connector or NCBI E-utilities; no R required |
      | Parse SOFT records for GSE/GSM/GPL/GDS | `geokit::geo_soft()` downloads and parses into GEO S4 objects |
      | Build an R ExpressionSet from series matrices | `geokit::geo_matrix()`; requires Biobase |
      | Prepare sample characteristics | `geokit::parse_sample_data()` on a supported loaded object |
      | Acquire selected supplementary files | Inspect the directory first, then `geokit::geo_suppl()` with a reviewed filename regex |
      | Build a larger metadata collection | Batch selected accessions with `geokit::geo_meta()` as a standalone Run |
      
      `geo_search(step=...)` fetches all matches in batches: `step` is not a total
      result limit. Use bounded connector queries for discovery. `geo_meta()` filters
      metadata during parsing but can still download full SOFT files.
      `geo_suppl()` returns paths and does not parse arbitrary attachment formats.
      The chat/Shiny helpers are unnecessary for acquisition through Wisp.
      
      ## Check the selected execution context
      
      Check Rscript and required packages in the context that will run the job, not
      only on the desktop. A standalone Run can declare an R preflight with packages
      `geokit` and, for ExpressionSet output, `Biobase`. Missing packages should be
      reported with setup instructions; discovery should not trigger installation.
      
      Record the installed version with `packageVersion("geokit")` and save
      `sessionInfo()` alongside the analysis. The upstream installation guide offers
      R-universe and GitHub installation; available binaries depend on platform/R
      version. Source installation requires Rust, so check the current package
      requirements before selecting it. Reuse the user's configured proxy and
      package sources.
      
      For standalone downloads, save an R script under the project and run it through
      the shared Run tools. Use the returned Run id for monitoring and cancellation.
      For subsequent work on objects already loaded in the persistent R runtime,
      continue in that runtime; a fresh Run cannot access those objects.
      
      ## Plan and acquire
      
      1. Inspect series, platforms, samples, and candidate files through the existing
         connector or official GEO HTTPS directories. Record the selected URLs and
         known sizes; distinguish unknown sizes from zero. A supplementary pattern
         must be checked against the actual inventory before transfer.
      2. Create the usual provider-neutral plan: provider `geo`, accession as the
         identifier, and data type `soft`, `series-matrix`, or `supplementary`.
         Transport describes the network path (`https` for GEO FTP over HTTPS), not
         the package name. Populate `provenance.adapter` with `geokit`,
         `adapter_version` with the installed version, and `query_url` with the stable
         source page. Record the selected file inventory and context/Run identity in
         a companion provenance file. Preserve both that file and the plan.
      3. Set an explicit accession-specific `odir` in the selected context. Keep raw
         downloads separate from derived RDS/tables. On SSH, retain large outputs as
         remote references and harvest only selected small summaries or manifests.
      4. Check that the adapter can satisfy the reviewed transfer. In the reviewed
         geokit version, existing filenames are reused without integrity checks and
         new downloads use `resume = FALSE`. Use `--no-resume` in a geokit plan and
         record that limitation. Verify cached files before reuse; an interrupted or
         unverified file is not a valid cache hit. If resumability or a strict byte
         limit is required, choose a downloader that enforces it before starting.
      5. Validate the plan and record the authorization for the concrete transfer
         using the shared workflow. Planner limits are not automatically passed into
         geokit. If the selected files cannot be bounded as required, resolve that
         before invoking the helper.
      
      For `geo_matrix()`, use `add_gpl = FALSE` unless platform annotation retrieval
      is part of the plan, and keep `pdata_from_soft = FALSE` unless the additional
      SOFT acquisition is needed and covered. These choices avoid implicit extra
      downloads but do not restrict the number of matrix files fetched. Use
      `ftp_over_https = TRUE` explicitly in download scripts.
      
      ## Validate and hand off
      
      - A GSE can have several platform matrices. Preserve separate objects and
        identify the GSE/GPL/source file for each; do not merge them automatically.
      - Missing files, a zero-row matrix, download failure, and parse failure are
        different outcomes. For an absent or empty assay, inspect supplementary
        counts or SRA/ENA references before proposing a new acquisition.
      - Check expression dimensions, sample IDs/order against phenotype rows, and
        feature IDs against any annotation. Preserve original characteristics along
        with cleaned columns so duplicate or ambiguous fields remain inspectable.
      - Keep assay values as retrieved. `log_trans()` is a heuristic transformation,
        not part of acquisition; gene-symbol conversion, normalization and
        differential expression belong to the subsequent analysis.
      - Save ExpressionSet outputs as RDS when requested, plus a bounded summary and
        sample table for inspection. Return paths or remote references, dimensions,
        and validation findings instead of printing whole matrices into chat.
      - Generate the manifest with the existing planner after checking the acquired
        files. The manifest records file inventory/checksums and a digest of the plan;
        it does not copy package provenance or validate biological correctness.
      
      ## Initial evaluation
      
      Compare the current acquisition workflow with this adapter on a single-platform
      matrix, a multi-platform series, and a study whose useful counts are in
      supplementary files. Include an unavailable package and an interrupted transfer.
      Record completion, manual interventions, file/sample completeness, and error
      clarity. Live provider checks are manual; automated regression checks use local
      fixtures and fake runners without R packages or network access.
      
      ## Sources and review boundary
      
      Reviewed geokit R package 0.0.2 at commit
      `3e0157737d4d3c948d55d155b1d73730a736ed13` on 2026-09-08. Check installed-version
      behavior before using flags; this reference is Wisp-specific routing guidance.
      
      - [Upstream skill](https://github.com/WangLabCSU/geokit/blob/3e0157737d4d3c948d55d155b1d73730a736ed13/pkgdown/assets/skills/geokit/SKILL.md)
      - [Download implementation](https://github.com/WangLabCSU/geokit/blob/3e0157737d4d3c948d55d155b1d73730a736ed13/R/download.R)
      - [Matrix implementation](https://github.com/WangLabCSU/geokit/blob/3e0157737d4d3c948d55d155b1d73730a736ed13/R/geo-matrix.R)
      - [Package requirements and installation](https://github.com/WangLabCSU/geokit/tree/3e0157737d4d3c948d55d155b1d73730a736ed13)
      - [GEO download documentation](https://www.ncbi.nlm.nih.gov/geo/info/download.html)
      
    • provider-routing.md 2.9 KB
      # Provider routing
      
      Choose a provider adapter only after the provider-neutral plan is valid.
      
      ## GEO
      
      - Accept `GSE`, `GDS`, `GSM`, and `GPL` accessions.
      - Inspect series/sample metadata before choosing a matrix or supplementary file.
      - Use series matrices for bounded expression reanalysis when available.
      - Use supplementary archives or SRA/ENA for raw data; do not pretend GEO itself
        guarantees FASTQ availability.
      - For SOFT/Series Matrix parsing or ExpressionSet acquisition in an R workflow,
        read [geokit.md](geokit.md). Use geokit when available in the selected
        execution context; keep basic discovery on the existing GEO connector.
      - Inspect files before calling geokit download helpers. `geo_suppl()` downloads
        matches immediately, and `geo_matrix()` may fetch multiple platform matrices.
        A URL constructed by `geo_url()` does not prove that a file exists.
      
      ## SRA and ENA
      
      - Resolve project/sample accessions to run accessions before transfer.
      - Prefer ENA HTTPS/FTP for directly available FASTQ with published checksums.
      - Use SRA Toolkit when conversion from SRA objects is required.
      - Record layout (single/paired), run count, bases/bytes, and checksums.
      
      ## GDC
      
      - Build an explicit files/cases query and save it with the plan or manifest.
      - Use `gdc-client` for resumable bulk transfer from a generated manifest.
      - Keep controlled-access data outside this public-data workflow unless the
        caller has explicitly configured credentials and authorization.
      - Do not merge thousands of files in memory during acquisition.
      
      ## GTEx
      
      - Use expression connectors/APIs for bounded gene/tissue questions.
      - Use official release matrices for bulk analysis and record the release.
      - Distinguish gene-level median expression from sample-level matrices.
      
      ## DepMap
      
      - Treat DepMap as an optional provider adapter, never as the common backend.
      - Inspect release and file metadata before choosing expression, mutation,
        copy-number, dependency, or model metadata products.
      - Keep release-specific names and authentication behavior inside this adapter.
      - Do not expose `--all` as a provider-neutral option.
      
      ## Custom HTTPS/FTP
      
      - Require explicit URLs or a machine-readable catalog response.
      - Record final resolved URLs, object identifiers, expected bytes, and checksums.
      - Reject short-lived signed URLs in durable plans; store a stable object ID and
        resolve a fresh URL only at execution time.
      
      ## Adapter selection order
      
      1. Installed Wisp connector/MCP for discovery and small results.
      2. Official provider API or bulk client.
      3. A versioned external CLI or language package already installed in the
         execution environment, such as geokit for GEO in R.
      4. Manual instructions when no safe executable adapter exists.
      
      Do not silently fall through from one adapter to another after a partial
      transfer. Record the failure, preserve resumable state, and ask before changing
      transport when the existing authorization does not cover that change.
      
  • scripts
    • public_data_plan.py 15.8 KB
      #!/usr/bin/env python3
      """Create, validate, and inventory provider-neutral public-data download plans."""
      
      from __future__ import annotations
      
      import argparse
      import hashlib
      import json
      import re
      import sys
      from datetime import datetime, timezone
      from pathlib import Path
      from typing import Any
      
      
      PROVIDERS: dict[str, dict[str, Any]] = {
          "geo": {
              "data_types": ["metadata", "series-matrix", "soft", "supplementary", "raw-reads"],
              "transports": ["auto", "mcp", "api", "https", "ftp", "manual"],
              "identifier_hint": "GSE, GDS, GSM, or GPL accession",
          },
          "sra": {
              "data_types": ["metadata", "run-info", "raw-reads"],
              "transports": ["auto", "mcp", "api", "https", "ftp", "sra-toolkit", "manual"],
              "identifier_hint": "SRP/SRR/SRS/SRX, ERP/ERR, DRP/DRR, or BioProject accession",
          },
          "ena": {
              "data_types": ["metadata", "run-info", "raw-reads"],
              "transports": ["auto", "mcp", "api", "https", "ftp", "manual"],
              "identifier_hint": "study, experiment, sample, or run accession",
          },
          "gdc": {
              "data_types": [
                  "manifest",
                  "files",
                  "expression",
                  "mutations",
                  "copy-number",
                  "clinical",
                  "methylation",
              ],
              "transports": ["auto", "mcp", "api", "https", "gdc-client", "manual"],
              "identifier_hint": "TCGA project, case, file UUID, or saved query identifier",
          },
          "gtex": {
              "data_types": [
                  "gene-expression",
                  "median-expression",
                  "sample-expression",
                  "tissue-metadata",
                  "bulk-files",
              ],
              "transports": ["auto", "mcp", "api", "https", "manual"],
              "identifier_hint": "release, gene, tissue, or named query",
          },
          "depmap": {
              "data_types": [
                  "model-metadata",
                  "expression",
                  "mutations",
                  "copy-number",
                  "dependency",
                  "release-files",
              ],
              "transports": ["auto", "mcp", "api", "https", "manual"],
              "identifier_hint": "release, model, file, or named query",
          },
          "custom": {
              "data_types": ["metadata", "files"],
              "transports": ["auto", "api", "https", "ftp", "manual"],
              "identifier_hint": "stable catalog identifier or URL",
          },
      }
      
      
      def utc_now() -> str:
          return (
              datetime.now(timezone.utc)
              .replace(microsecond=0)
              .isoformat()
              .replace("+00:00", "Z")
          )
      
      
      def parse_size(value: str) -> int:
          match = re.fullmatch(r"\s*(\d+(?:\.\d+)?)\s*([KMGTPE]?I?B)?\s*", value, re.I)
          if not match:
              raise argparse.ArgumentTypeError(
                  "size must be an integer byte count or a value such as 500MB, 10GB, or 2GiB"
              )
          number = float(match.group(1))
          unit = (match.group(2) or "B").upper()
          decimal = {"B": 1, "KB": 10**3, "MB": 10**6, "GB": 10**9, "TB": 10**12, "PB": 10**15, "EB": 10**18}
          binary = {"KIB": 2**10, "MIB": 2**20, "GIB": 2**30, "TIB": 2**40, "PIB": 2**50, "EIB": 2**60}
          multiplier = decimal.get(unit, binary.get(unit))
          if multiplier is None:
              raise argparse.ArgumentTypeError(f"unsupported size unit: {unit}")
          return int(number * multiplier)
      
      
      def parse_filters(values: list[str]) -> dict[str, str]:
          filters: dict[str, str] = {}
          for item in values:
              if "=" not in item:
                  raise ValueError(f"filter must use key=value syntax: {item!r}")
              key, value = item.split("=", 1)
              key = key.strip()
              value = value.strip()
              if not key or not value:
                  raise ValueError(f"filter key and value must be non-empty: {item!r}")
              if key in filters:
                  raise ValueError(f"duplicate filter key: {key}")
              filters[key] = value
          return filters
      
      
      def safe_segment(value: str) -> str:
          value = re.sub(r"[^A-Za-z0-9._-]+", "-", value.strip()).strip("-.")
          return value or "dataset"
      
      
      def build_plan(args: argparse.Namespace) -> dict[str, Any]:
          provider = args.provider.lower()
          identifier = args.identifier.strip()
          output_dir = args.output_dir or f"data/public/{provider}/{safe_segment(identifier)}"
          manifest_path = args.manifest_path or str(Path(output_dir) / "manifest.json")
          return {
              "schema_version": 1,
              "created_at": utc_now(),
              "dataset": {
                  "provider": provider,
                  "identifier": identifier,
                  "data_type": args.data_type,
                  "release": args.release,
                  "filters": parse_filters(args.filters),
              },
              "acquisition": {
                  "transport": args.transport,
                  "resume": args.resume,
                  "overwrite": args.allow_overwrite,
                  "max_files": args.max_files,
                  "max_bytes": args.max_bytes,
                  "checksum": args.checksum,
              },
              "output": {
                  "directory": output_dir,
                  "manifest": manifest_path,
              },
              "approval": {
                  "required": True,
                  "status": "pending",
              },
              "provenance": {
                  "adapter": None,
                  "adapter_version": None,
                  "query_url": None,
              },
              "notes": args.notes,
          }
      
      
      def looks_absolute(path: str) -> bool:
          return path.startswith(("/", "\\\\")) or bool(re.match(r"^[A-Za-z]:[\\/]", path))
      
      
      def validate_plan(plan: Any) -> dict[str, list[str]]:
          errors: list[str] = []
          warnings: list[str] = []
          if not isinstance(plan, dict):
              return {"errors": ["plan must be a JSON object"], "warnings": []}
          if plan.get("schema_version") != 1:
              errors.append("schema_version must be 1")
      
          dataset = plan.get("dataset")
          acquisition = plan.get("acquisition")
          output = plan.get("output")
          approval = plan.get("approval")
          if not isinstance(dataset, dict):
              errors.append("dataset must be an object")
              dataset = {}
          if not isinstance(acquisition, dict):
              errors.append("acquisition must be an object")
              acquisition = {}
          if not isinstance(output, dict):
              errors.append("output must be an object")
              output = {}
          if not isinstance(approval, dict):
              errors.append("approval must be an object")
              approval = {}
      
          provider = str(dataset.get("provider") or "").lower()
          identifier = str(dataset.get("identifier") or "").strip()
          data_type = str(dataset.get("data_type") or "")
          provider_info = PROVIDERS.get(provider)
          if provider_info is None:
              errors.append(f"unsupported provider: {provider!r}")
          else:
              if data_type not in provider_info["data_types"]:
                  errors.append(
                      f"unsupported data_type {data_type!r} for {provider}; "
                      f"choose one of {provider_info['data_types']}"
                  )
              transport = str(acquisition.get("transport") or "")
              if transport not in provider_info["transports"]:
                  errors.append(
                      f"unsupported transport {transport!r} for {provider}; "
                      f"choose one of {provider_info['transports']}"
                  )
          if not identifier:
              errors.append("dataset.identifier is required")
      
          accession_patterns = {
              "geo": r"^(GSE|GDS|GSM|GPL)\d+$",
              "sra": r"^((SR|ER|DR)[APRSX]\d+|PRJ(NA|EB|DB)\d+)$",
              "ena": r"^((SR|ER|DR)[APRSX]\d+|PRJ(NA|EB|DB)\d+)$",
              "gdc": r"^(TCGA-[A-Z0-9-]+|[0-9a-fA-F-]{32,36}|[A-Za-z0-9._:-]+)$",
          }
          pattern = accession_patterns.get(provider)
          if identifier and pattern and not re.match(pattern, identifier, re.I):
              warnings.append(
                  f"identifier {identifier!r} is unusual for provider {provider}; verify it during discovery"
              )
      
          filters = dataset.get("filters", {})
          if not isinstance(filters, dict):
              errors.append("dataset.filters must be an object")
          if provider == "gtex" and data_type == "gene-expression" and not (
              isinstance(filters, dict) and any(k in filters for k in ("gene", "genes"))
          ):
              warnings.append("GTEx gene-expression plans normally include a gene or genes filter")
          if provider == "gdc" and data_type not in ("manifest", "files") and not filters:
              warnings.append("GDC analysis-product plans should record workflow/sample filters")
      
          for field in ("max_files", "max_bytes"):
              value = acquisition.get(field)
              if value is not None and (not isinstance(value, int) or isinstance(value, bool) or value <= 0):
                  errors.append(f"acquisition.{field} must be a positive integer or null")
          checksum = acquisition.get("checksum")
          if checksum not in ("sha256", "none"):
              errors.append("acquisition.checksum must be 'sha256' or 'none'")
          if acquisition.get("overwrite") is True:
              warnings.append("overwrite is enabled; require explicit user confirmation before execution")
          if acquisition.get("max_files") is None and acquisition.get("max_bytes") is None:
              warnings.append("no transfer limit is set; resolve expected scale before bulk acquisition")
      
          for field in ("directory", "manifest"):
              value = str(output.get(field) or "").strip()
              if not value:
                  errors.append(f"output.{field} is required")
              elif looks_absolute(value):
                  warnings.append(f"output.{field} is absolute and reduces plan portability: {value}")
      
          if approval.get("required") is not True:
              errors.append("approval.required must be true for public-data acquisition plans")
          if approval.get("status") not in ("pending", "approved", "rejected"):
              errors.append("approval.status must be pending, approved, or rejected")
      
          provenance = plan.get("provenance")
          if not isinstance(provenance, dict):
              errors.append("provenance must be an object")
          notes = plan.get("notes")
          if not isinstance(notes, list) or any(not isinstance(x, str) for x in notes):
              errors.append("notes must be an array of strings")
          return {"errors": errors, "warnings": warnings}
      
      
      def read_json(path: Path) -> Any:
          try:
              return json.loads(path.read_text(encoding="utf-8"))
          except FileNotFoundError as exc:
              raise ValueError(f"file not found: {path}") from exc
          except json.JSONDecodeError as exc:
              raise ValueError(f"invalid JSON in {path}: {exc}") from exc
      
      
      def write_json(path: Path, value: Any, replace: bool = False) -> None:
          if path.exists() and not replace:
              raise ValueError(f"refusing to replace existing file without an explicit flag: {path}")
          path.parent.mkdir(parents=True, exist_ok=True)
          path.write_text(json.dumps(value, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
      
      
      def sha256_file(path: Path) -> str:
          digest = hashlib.sha256()
          with path.open("rb") as handle:
              for chunk in iter(lambda: handle.read(1024 * 1024), b""):
                  digest.update(chunk)
          return digest.hexdigest()
      
      
      def plan_digest(plan: Any) -> str:
          payload = json.dumps(plan, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
          return hashlib.sha256(payload.encode("utf-8")).hexdigest()
      
      
      def build_manifest(
          plan: dict[str, Any], scan_dir: Path, output: Path, checksum: str
      ) -> dict[str, Any]:
          if not scan_dir.is_dir():
              raise ValueError(f"scan directory does not exist: {scan_dir}")
          files: list[dict[str, Any]] = []
          output_resolved = output.resolve()
          for path in sorted(p for p in scan_dir.rglob("*") if p.is_file()):
              if path.resolve() == output_resolved:
                  continue
              stat = path.stat()
              item: dict[str, Any] = {
                  "path": path.relative_to(scan_dir).as_posix(),
                  "bytes": stat.st_size,
              }
              if checksum == "sha256":
                  item["sha256"] = sha256_file(path)
              files.append(item)
          return {
              "schema_version": 1,
              "created_at": utc_now(),
              "plan_sha256": plan_digest(plan),
              "provider": plan.get("dataset", {}).get("provider"),
              "identifier": plan.get("dataset", {}).get("identifier"),
              "scan_root": str(scan_dir),
              "checksum": checksum,
              "summary": {
                  "file_count": len(files),
                  "total_bytes": sum(item["bytes"] for item in files),
              },
              "files": files,
          }
      
      
      def make_parser() -> argparse.ArgumentParser:
          parser = argparse.ArgumentParser(description=__doc__)
          sub = parser.add_subparsers(dest="command", required=True)
      
          sub.add_parser("providers", help="print supported providers and adapter capabilities")
      
          init = sub.add_parser("init", help="create a provider-neutral download plan")
          init.add_argument("--provider", required=True, choices=sorted(PROVIDERS))
          init.add_argument("--identifier", required=True)
          init.add_argument("--data-type", required=True)
          init.add_argument("--release")
          init.add_argument("--filter", dest="filters", action="append", default=[], metavar="KEY=VALUE")
          init.add_argument("--transport", default="auto")
          init.add_argument("--output-dir")
          init.add_argument("--manifest-path")
          init.add_argument("--max-files", type=int)
          init.add_argument("--max-bytes", type=parse_size)
          init.add_argument("--checksum", choices=("sha256", "none"), default="sha256")
          init.add_argument("--no-resume", dest="resume", action="store_false", default=True)
          init.add_argument("--allow-overwrite", action="store_true")
          init.add_argument("--note", dest="notes", action="append", default=[])
          init.add_argument("--plan", required=True, type=Path)
          init.add_argument("--replace-plan", action="store_true")
      
          validate = sub.add_parser("validate", help="validate an existing plan")
          validate.add_argument("plan", type=Path)
      
          manifest = sub.add_parser("manifest", help="inventory acquired files")
          manifest.add_argument("plan", type=Path)
          manifest.add_argument("--scan-dir", type=Path)
          manifest.add_argument("--output", type=Path)
          manifest.add_argument("--checksum", choices=("auto", "sha256", "none"), default="auto")
          manifest.add_argument("--replace", action="store_true")
          return parser
      
      
      def main(argv: list[str] | None = None) -> int:
          parser = make_parser()
          args = parser.parse_args(argv)
          try:
              if args.command == "providers":
                  print(json.dumps(PROVIDERS, indent=2, ensure_ascii=False))
                  return 0
              if args.command == "init":
                  plan = build_plan(args)
                  result = validate_plan(plan)
                  if result["errors"]:
                      print(json.dumps(result, indent=2, ensure_ascii=False), file=sys.stderr)
                      return 2
                  write_json(args.plan, plan, replace=args.replace_plan)
                  print(json.dumps({"plan": str(args.plan), "validation": result, "content": plan}, indent=2, ensure_ascii=False))
                  return 0
              if args.command == "validate":
                  plan = read_json(args.plan)
                  result = validate_plan(plan)
                  print(json.dumps(result, indent=2, ensure_ascii=False))
                  return 0 if not result["errors"] else 2
              if args.command == "manifest":
                  plan = read_json(args.plan)
                  result = validate_plan(plan)
                  if result["errors"]:
                      print(json.dumps(result, indent=2, ensure_ascii=False), file=sys.stderr)
                      return 2
                  output_info = plan["output"]
                  scan_dir = args.scan_dir or Path(output_info["directory"])
                  output = args.output or Path(output_info["manifest"])
                  checksum = args.checksum
                  if checksum == "auto":
                      checksum = plan["acquisition"]["checksum"]
                  value = build_manifest(plan, scan_dir, output, checksum)
                  write_json(output, value, replace=args.replace)
                  print(json.dumps({"manifest": str(output), "summary": value["summary"]}, indent=2))
                  return 0
          except (OSError, ValueError) as exc:
              print(f"error: {exc}", file=sys.stderr)
              return 2
          parser.error("unknown command")
          return 2
      
      
      if __name__ == "__main__":
          raise SystemExit(main())
      
  • SKILL.md 6.8 KB
    ---
    name: public-data-access
    description: "Plan, validate, and document public-bioinformatics data acquisition for GEO/GSE/GSM/GPL/GDS, SRA/ENA, TCGA/GDC, GTEx, and DepMap. Covers expression matrices, raw reads, download manifests, caches, and optional geokit SOFT/Series Matrix acquisition for R workflows."
    ---
    
    # Public Bioinformatics Data Access
    
    Build a reproducible acquisition plan before downloading. Treat GEO, SRA/ENA,
    GDC, GTEx, and DepMap as independent providers behind one provider-neutral
    workflow. Do not require a provider-specific toolkit or machine-specific checkout.
    
    ## Workflow
    
    1. **Clarify the dataset contract.** Identify the provider, accession/project,
       release, modality, smallest useful data product, filters, target directory,
       expected scale, and downstream analysis.
    2. **Inspect before transfer.** Use available MCP/connectors or official
       metadata endpoints to list releases, files, samples, sizes, and checksums.
       Do not start a bulk transfer during discovery.
    3. **Write a provider-neutral plan.** Run `scripts/public_data_plan.py init`.
       Store the plan next to the future dataset as `download-plan.json`.
    4. **Validate and review.** Run `validate`, show the user the resolved provider,
       transport, filters, limits, output location, and known size. For a large,
       paid, authenticated, or overwrite-capable job, confirm that the user's
       authorization covers this concrete transfer; ask only when it does not.
    5. **Select the adapter at runtime.** Prefer an already available Wisp MCP tool
       for metadata and small queries. Prefer official HTTPS/FTP or provider clients
       for bulk files. Use an external project only when it is installed and record
       its version in the plan/manifest.
    6. **Acquire safely.** Reuse existing valid files, resume partial transfers when
       supported, keep raw files immutable, and never place credentials in the plan.
    7. **Verify and hand off.** Check expected files, byte sizes, checksums when
       available, and sample/file counts. Generate `manifest.json` with the script.
    
    ## Provider routing
    
    | Provider | Discovery and small queries | Bulk acquisition | Typical products |
    |---|---|---|---|
    | GEO | GEO metadata connector, NCBI E-utilities | NCBI GEO HTTPS/FTP; optional geokit in R | series matrix, SOFT, supplementary files |
    | SRA/ENA | RunInfo or ENA Portal API | ENA HTTPS/FTP or SRA Toolkit | FASTQ, run metadata |
    | GDC | GDC files/cases API | manifest + `gdc-client`, or HTTPS for bounded files | expression, mutation, CNV, clinical, methylation |
    | GTEx | GTEx expression connector/API | official release files for matrices | gene/tissue queries, median or sample expression |
    | DepMap | DepMap model/release metadata | official release file endpoint | model metadata, expression, mutation, dependency |
    | custom | User-provided catalog/API | explicit HTTPS/FTP URLs | provider-specific files |
    
    Read `references/provider-routing.md` before implementing or changing a
    provider adapter. DepMap-specific flags or release semantics must stay inside
    the DepMap adapter; they must not shape the common plan schema.
    
    For GEO SOFT/Series Matrix parsing, sample metadata preparation, or
    ExpressionSet acquisition in an R workflow, read
    [references/geokit.md](references/geokit.md). geokit is optional; ordinary GEO
    discovery does not require R or package installation.
    
    ## Create and validate a plan
    
    Resolve `scripts/public_data_plan.py` against this skill's directory (the
    `use_skill` result lists its path), and invoke that resolved script with a
    Python 3.10+ interpreter. Keep the working directory at the project root so
    relative plan/output paths belong to the project. The examples below abbreviate
    the script path; quote the resolved path when it contains spaces. In an SSH/WSL
    context, stage the helper there or use an existing copy in that context; a
    desktop skill path is not automatically available remotely.
    
    ```bash
    python scripts/public_data_plan.py init \
      --provider geo \
      --identifier GSE12345 \
      --data-type series-matrix \
      --output-dir data/public/geo/GSE12345 \
      --plan data/public/geo/GSE12345/download-plan.json
    
    python scripts/public_data_plan.py validate \
      data/public/geo/GSE12345/download-plan.json
    ```
    
    Filters are provider-specific but encoded uniformly as repeated `key=value`
    pairs:
    
    ```bash
    python scripts/public_data_plan.py init \
      --provider gdc \
      --identifier TCGA-BRCA \
      --data-type expression \
      --filter workflow_type="STAR - Counts" \
      --filter sample_type="Primary Tumor" \
      --max-files 20 \
      --transport gdc-client \
      --plan data/public/gdc/TCGA-BRCA/download-plan.json
    ```
    
    The planner does not download data. It produces a reviewable contract. See
    `references/download-plan-schema.md` for the complete schema. Validation checks
    the plan structure; it does not probe URLs, enforce transfer limits, verify
    installed packages, or approve a pending transfer. The selected adapter must
    honor the plan's limits and resume behavior.
    
    ## Generate a manifest
    
    After acquisition:
    
    ```bash
    python scripts/public_data_plan.py manifest \
      data/public/geo/GSE12345/download-plan.json \
      --scan-dir data/public/geo/GSE12345 \
      --output data/public/geo/GSE12345/manifest.json
    ```
    
    Use SHA-256 for modest datasets and provider checksums for large archives. For
    very large datasets, `--checksum none` is acceptable only when official
    checksums or immutable object identifiers are recorded elsewhere.
    
    ## Safety and reproducibility rules
    
    - Default to `overwrite=false`, `resume=true`, and the minimum useful subset.
    - Never translate an exploratory request into “download everything.”
    - Keep provider metadata, query/filter payloads, release/version, transport,
      tool version, URLs/object identifiers, and validation results.
    - Separate immutable source files from normalized/derived outputs.
    - Do not treat a successful HTTP response as a valid dataset; verify content.
    - Do not embed API keys, cookies, signed URLs, SSH keys, or bearer tokens.
    - Use structured runs or a remote execution context for long transfers rather
      than extending an interactive shell timeout.
    - If an adapter or connector cannot perform the requested transfer, stop after
      producing the validated plan and report the missing capability explicitly.
    
    ## Wisp Science integration
    
    - Discover the live connector/tool catalog instead of assuming exact MCP tool
      names; installations can expose different provider adapters.
    - Use connectors for discovery and bounded queries, then official transfer
      mechanisms for large files.
    - Keep outputs under the active project, normally `data/public/<provider>/...`.
    - Invoke the planner as a standalone CLI; no Python REPL helper loading is
      required. R-based acquisition can use geokit independently of the planner.
    - Treat this skill as an acquisition/orchestration layer. Downstream QC,
      statistics, annotation, and visualization belong to other skills.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related