methylation-aggregation
Build comprehensive DNA methylation maps by aggregating WGBS (Whole Genome Bisulfite Sequencing) data across multiple ENCODE experiments, donors, and labs. Use when the user wants to answer "where is DNA methylated/unmethylated in my tissue?" by combining per-CpG methylation data
Install
npx skills add https://github.com/ammawla/encode-toolkit/tree/main/skills/methylation-aggregation
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install ammawla-encode-toolkit@llmmart
git clone https://github.com/ammawla/encode-toolkit.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole ammawla/encode-toolkit collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Aggregate DNA Methylation Data Across Studies
When to Use
- User wants to build a tissue-level DNA methylation landscape from multiple WGBS experiments
- User asks "where is DNA methylated in brain?" or "find hypomethylated regions across donors"
- User needs to identify HMRs (hypomethylated regions), UMRs, or PMDs from aggregated WGBS data
- User wants per-CpG weighted methylation averages from multiple experiments
- Example queries: "aggregate WGBS data for liver", "build methylation map across donors", "find unmethylated CpG islands in pancreas"
Build a comprehensive methylation landscape for a tissue/cell type by merging WGBS bedMethyl files from multiple ENCODE experiments.
Scientific Rationale
The question: "What is the DNA methylation state across the genome in my tissue?"
DNA methylation is fundamentally different from histone marks and accessibility:
| Property | Histone/Accessibility | DNA Methylation |
|---|---|---|
| Signal type | Binary (bound/open or not) | Continuous (0-100% methylated) |
| Default state | Unmarked | ~70-80% methylated (CpG context) |
| Biology of interest | Where marks ARE present | Where methylation is ABSENT or REDUCED |
| Aggregation approach | Union of peak calls | Average/median of methylation levels per CpG |
The key insight: Unlike histone ChIP-seq where we want the union of all peaks, for methylation we want the average methylation level per CpG site across individuals. Methylation is a quantitative, continuous signal measured at every CpG dinucleotide.
However, for identifying regulatory regions, we focus on hypomethylated regions (HMRs) — stretches of low methylation that mark active regulatory elements. HMRs can be treated more like peaks for union-style aggregation.
Literature Support
- Roadmap Epigenomics (Schultz et al. 2015, Nature, 2,900+ citations): Established that tissue-specific HMRs mark active regulatory elements; demonstrated per-CpG averaging across biological replicates as standard approach
- DMRcate (Peters et al. 2021, Nucleic Acids Research, 65 citations): Method for calling differentially methylated regions from multiple WGBS samples; uses kernel smoothing across CpG sites
- ENCODE Phase 3 (Gorkin et al. 2020, Nature, 301 citations): Integrated methylation data with histone marks and accessibility to define chromatin states
- ENCODE Blacklist (Amemiya et al. 2019, Scientific Reports, 1,372 citations): Problematic genomic regions to filter. DOI
- Zhou et al. 2020 (Nature Genetics): Tissue-specific methylation patterns: ~80% of CpGs are constitutively methylated, ~10% constitutively unmethylated (CpG islands/promoters), ~10% tissue-variable
- Liu et al. 2024 (Briefings in Bioinformatics): Cross-platform comparison (NovaSeq vs DNBSEQ) showing WGBS is gold standard; coverage depth critically affects accuracy; platform differences exist in GC-rich regions
- Ortega-Recalde et al. 2021 (Methods in Molecular Biology): Demonstrated that even low-coverage WGBS can accurately estimate global methylation levels, with bootstrap methods to quantify uncertainty
Two-Level Analysis
- Per-CpG level: Average methylation at each CpG site across samples (quantitative map)
- Region level: Identify HMRs, PMDs, and UMRs from the averaged profile (union of regulatory regions)
Step 1: Find All Available WGBS Data
encode_search_experiments(
assay_title="WGBS",
organ="pancreas", # user's tissue of interest
biosample_type="tissue",
limit=100
)
Present a summary to the user:
- Total WGBS experiments
- Labs represented
- Unique donors/biosamples
- Genome coverage per experiment
Use encode_get_facets to check availability:
encode_get_facets(assay_title="WGBS", organ="pancreas")
Note: WGBS is expensive to generate. Typical tissues have 2-5 experiments. Even 2 biological replicates are valuable for identifying consistent methylation patterns.
Step 2: Quality-Gate Each Experiment
encode_get_experiment(accession="ENCSR...")
WGBS Quality Checks
- Audit status: no ERROR flags
- Bisulfite conversion rate: >=98% (measured by lambda spike-in or non-CpG methylation)
- Genome coverage: >=10x mean CpG coverage for reliable per-site estimates
- Mapping rate: >=50% (bisulfite-converted reads are harder to map)
- Duplication rate: <30%
- Has
methylation state at CpGoutput files (bedMethyl format)
Include if:
- Bisulfite conversion >=98%
- Mean CpG coverage >=10x
- Has bedMethyl output files for GRCh38
Exclude if:
- ERROR audit flags
- Conversion rate <98% (unconverted reads create false methylation calls)
- Very low coverage (<5x mean) — individual CpG estimates unreliable
Track all included experiments:
encode_track_experiment(accession="ENCSR...")
Step 3: Download bedMethyl Files
For each experiment:
encode_list_files(
experiment_accession="ENCSR...",
output_type="methylation state at CpG",
assembly="GRCh38"
)
bedMethyl format (ENCODE standard):
chr start end name score strand thickStart thickEnd color coverage percentMethylated
- Column 10: read coverage at this CpG
- Column 11: percent methylation (0-100)
Prefer preferred_default=True files:
encode_download_files(
file_accessions=["ENCFF...", ...],
download_dir="/path/to/data/methylation",
organize_by="flat"
)
Validate the downloaded files before filtering. The scale of the methylation column
is decided once per file; override it with --scale if the file is not ENCODE
bedMethyl. Gzipped inputs are read directly.
python3 scripts/validate_methylation.py sample.bedMethyl [--min-coverage 5] [--scale auto|percent|fraction] [--blacklist hg38-blacklist.v2.bed]
Step 4: Per-Sample Quality Filtering
4a. Coverage Filtering (CRITICAL)
Low-coverage CpGs have unreliable methylation estimates. Filter per-sample:
# Keep only CpGs with >= 5x coverage (column 10)
# More stringent: >= 10x for quantitative analysis
awk '$10 >= 5' sample.bedMethyl > sample.covfiltered.bedMethyl
Coverage thresholds by use case:
| Threshold | Use Case | Typical CpGs Retained |
|---|---|---|
| >=3x | Exploratory / maximum retention | ~90% of CpGs |
| >=5x | Standard analysis | ~80% of CpGs |
| >=10x | High-confidence quantitative | ~60% of CpGs |
4b. ENCODE Blocklist Filtering (Amemiya et al. 2019)
# Download from: https://github.com/Boyle-Lab/Blacklist/blob/master/lists/hg38-blacklist.v2.bed.gz
gunzip -k hg38-blacklist.v2.bed.gz
bedtools intersect -a sample.covfiltered.bedMethyl -b hg38-blacklist.v2.bed -v > sample.filtered.bedMethyl
4c. Strand Merging (Optional but Recommended)
CpG methylation is typically symmetric (same on both strands). Merge strand-specific calls to increase per-CpG coverage. Caveat: Some ENCODE bedMethyl files may already be strand-merged — check if both strands are present before applying this step:
# Group CpGs by position (forward and reverse strand of same CpG)
# Sum coverage, calculate weighted average methylation
awk 'BEGIN{OFS="\t"} {
# CpG position (use the C position as canonical)
if ($6 == "+") pos = $2
else pos = $2 - 1
key = $1"\t"pos
cov[key] += $10
meth[key] += ($11/100) * $10
} END {
for (k in cov) {
split(k, a, "\t")
avg_meth = (meth[k] / cov[k]) * 100
print a[1], a[2], a[2]+2, "CpG", 0, ".", a[2], a[2]+2, "0,0,0", cov[k], avg_meth
}
}' sample.filtered.bedMethyl | sort -k1,1 -k2,2n > sample.merged_strands.bedMethyl
Step 5: Cross-Sample Aggregation (Per-CpG Averaging)
5a. Create a Unified CpG Matrix
# Step 1: Find CpGs covered in at least M of N samples
# Extract positions from each sample
for f in sample*.merged_strands.bedMethyl; do
awk 'BEGIN{OFS="\t"} {print $1, $2, $3}' "$f"
done | sort -k1,1 -k2,2n | uniq -c | \
awk -v M=2 '$1 >= M {print $2, $3, $4}' OFS="\t" > shared_cpgs.bed
# Step 2: For each sample, extract methylation at shared CpGs
for f in sample*.merged_strands.bedMethyl; do
bedtools intersect -a shared_cpgs.bed -b "$f" -wa -wb | \
awk 'BEGIN{OFS="\t"} {print $1, $2, $3, $NF, $(NF-1)}' > "${f%.bedMethyl}.shared.txt"
# Columns: chr, start, end, percentMeth, coverage
done
5b. Calculate Average Methylation Per CpG
Weighted average (recommended — accounts for coverage differences):
# Combine all samples, calculate coverage-weighted mean methylation per CpG
cat sample*.shared.txt | \
sort -k1,1 -k2,2n | \
awk 'BEGIN{OFS="\t"} {
key = $1"\t"$2"\t"$3
if (key != prev_key && NR > 1) {
avg = total_weighted_meth / total_cov
print prev_key, n_samples, total_cov, avg
n_samples = 0; total_cov = 0; total_weighted_meth = 0
}
prev_key = key
n_samples++
total_cov += $5
total_weighted_meth += ($4/100) * $5
} END {
avg = total_weighted_meth / total_cov
print prev_key, n_samples, total_cov, avg
}' > tissue_methylation_profile.bed
# Columns: chr, start, end, n_samples, total_coverage, mean_methylation_fraction
Simple average (alternative — equal weight per sample):
# Unweighted mean across samples
awk 'BEGIN{OFS="\t"} {
key = $1"\t"$2"\t"$3
meth[key] += $4
n[key]++
} END {
for (k in meth) {
print k, n[k], meth[k]/n[k]
}
}' <(cat sample*.shared.txt) | sort -k1,1 -k2,2n > tissue_methylation_simple.bed
5c. Calculate Methylation Variability
Track inter-individual variation to identify tissue-variable CpGs:
# Add standard deviation column
# (compute in R or Python for large datasets)
Step 6: Identify Regulatory Methylation Features
6a. Hypomethylated Regions (HMRs)
HMRs mark active regulatory elements. Identify runs of low methylation. The 30% threshold is a commonly used cutoff (Schultz et al. 2015 used similar ranges), but the optimal threshold depends on your tissue and question — some studies use 20%, others 40%. Consider visualizing the methylation distribution first to identify a natural breakpoint:
# Find CpGs with average methylation < 30% (adjust threshold as needed)
awk '$6 < 0.30' tissue_methylation_profile.bed > hypo_cpgs.bed
# Merge adjacent hypomethylated CpGs into regions
# Require minimum 3 CpGs within 1kb of each other
bedtools merge -i hypo_cpgs.bed -d 1000 -c 1 -o count | \
awk '$4 >= 3' > tissue_HMRs.bed
# Columns: chr, start, end, n_hypomethylated_CpGs
6b. Unmethylated Regions (UMRs) — CpG Islands
Very low methylation (<10%) at CpG-dense regions:
awk '$6 < 0.10' tissue_methylation_profile.bed > unmeth_cpgs.bed
bedtools merge -i unmeth_cpgs.bed -d 500 -c 1 -o count | \
awk '$4 >= 5' > tissue_UMRs.bed
6c. Partially Methylated Domains (PMDs)
Large (>10kb) regions of intermediate methylation, often marking repressed regions:
# Find CpGs with methylation 30-70% (partially methylated)
awk '$6 >= 0.30 && $6 <= 0.70' tissue_methylation_profile.bed > partial_cpgs.bed
# Merge with large gap tolerance to find domains
bedtools merge -i partial_cpgs.bed -d 5000 -c 1 -o count | \
awk '$4 >= 20 && ($3-$2) >= 10000' > tissue_PMDs.bed
6d. Tissue-Specific Differentially Methylated Regions
If comparing to another tissue, use DMRcate or similar:
# In R with DMRcate
library(DMRcate)
# Requires a methylation matrix (CpGs x samples with tissue labels)
# Identifies regions where methylation differs between tissues
Step 7: Confidence Annotation
For HMRs/UMRs (region-level features), annotate by sample support:
| Confidence | Criteria | Interpretation |
|---|---|---|
| High | Low methylation in >=50% of samples | Constitutive regulatory region |
| Supported | Low methylation in 2+ samples | Likely regulatory, some variation |
| Variable | High variance across samples | Cell-type heterogeneity or individual variation |
# Annotate HMRs with sample support
# Intersect each HMR with per-sample hypomethylated CpGs to count support
awk -v N=4 '{
# Using n_samples from the aggregation
if ($4 >= N*0.5) conf="HIGH";
else if ($4 >= 2) conf="SUPPORTED";
else conf="VARIABLE";
print $0"\t"conf"\t"$4"/"N
}' tissue_HMRs.bed > tissue_HMRs.annotated.bed
For the per-CpG profile, annotate by coverage confidence:
awk -v N=4 '{
if ($4 >= N) conf="ALL_SAMPLES";
else if ($4 >= N*0.5) conf="MAJORITY";
else conf="PARTIAL";
print $0"\t"conf
}' tissue_methylation_profile.bed > tissue_methylation.annotated.bed
Step 7b: Summary Statistics
Report to the user:
- Total input experiments: N
- Experiments passing QC: M (bisulfite conversion, coverage)
- Total CpGs per sample (before/after coverage filter)
- Shared CpGs across M+ samples: X
- Mean genome-wide methylation: Y%
- Number of HMRs: Z (with size distribution)
- Number of UMRs: W
- Number of PMDs: V (if applicable)
- High-confidence HMRs: how many in ≥50% of samples
- CpGs with high inter-individual variability
Step 8: Integration with Other ENCODE Data
Methylation data is most powerful when integrated:
- HMRs + H3K27ac peaks = Active enhancers (use histone-aggregation skill)
- HMRs + ATAC-seq peaks = Open regulatory elements (use accessibility-aggregation skill)
- UMRs + H3K4me3 peaks = Active promoters
- PMDs = Often overlap H3K9me3 (heterochromatin)
# Example: Find HMRs that overlap H3K27ac peaks (active enhancers)
bedtools intersect -a tissue_HMRs.bed -b union_H3K27ac_peaks.bed -wa -u > active_enhancer_HMRs.bed
Step 9: Log Provenance
encode_log_derived_file(
file_path="/path/to/tissue_methylation.annotated.bed",
source_accessions=["ENCSR...", "ENCSR...", ...],
description="Aggregated per-CpG methylation profile across N pancreas WGBS experiments",
file_type="aggregated_methylation",
tool_used="bedtools + custom aggregation",
parameters="coverage >= 5x per sample, shared CpGs in >= 2 samples, coverage-weighted mean, strand-merged"
)
encode_log_derived_file(
file_path="/path/to/tissue_HMRs.annotated.bed",
source_accessions=["ENCSR...", "ENCSR...", ...],
description="Hypomethylated regions from aggregated pancreas methylation profile",
file_type="aggregated_HMRs",
tool_used="bedtools merge",
parameters="mean methylation < 30%, >= 3 CpGs within 1kb, confidence annotated"
)
Pitfalls Specific to Methylation Data
Bisulfite conversion rate is critical: Even 1% incomplete conversion creates false methylation at unmethylated CpGs. Always verify >=98% conversion. ENCODE reports this in QC metrics.
Coverage drives accuracy: A CpG with 3x coverage has wide confidence intervals (0-100% could easily be 0% or 30%). At 10x, estimates stabilize. At 30x, they are reliable. Always filter by coverage.
Non-CpG methylation: Present in some cell types (especially embryonic). ENCODE bedMethyl files typically report CpG context only. If non-CpG methylation is relevant, check experiment metadata.
Strand asymmetry: While CpG methylation is typically symmetric, it can be asymmetric at some sites. Strand merging loses this information. For most analyses, merging is appropriate.
Cell-type heterogeneity: Bulk WGBS from tissue captures methylation across ALL cell types. A CpG at 50% methylation could mean: (a) all cells are 50% methylated, or (b) half the cells are 0% and half are 100%. These are biologically different. Single-cell methylation data (if available) resolves this.
CpG islands vs. open sea: CpG-dense regions (islands) have very different methylation dynamics than CpG-sparse regions. Consider analyzing separately.
Do NOT mix assemblies: All files must be GRCh38 or all hg19. CpG positions are exact — even a 1bp offset from liftOver misaligns CpGs.
X chromosome: Males have one X (hemimethylation), females have two (one inactivated with different methylation). Handle sex chromosomes separately or filter them.
Imprinted regions: Some regions show ~50% methylation in all individuals due to genomic imprinting (one allele methylated, one not). These are normal, not noise.
Do NOT use union logic for per-CpG methylation: Unlike histone peaks where union is correct, methylation levels should be AVERAGED. Union logic only applies to the derived HMR/UMR/PMD regions.
RRBS is NOT the same as WGBS: Reduced Representation Bisulfite Sequencing (RRBS) covers only CpG-rich regions (~10% of CpGs). Do NOT mix RRBS and WGBS in per-CpG averaging — the CpG universe is different.
Sequencing platform matters: Liu et al. 2024 showed that NovaSeq and DNBSEQ-T7 give slightly different methylation estimates, especially in GC-rich regions. Note the platform in provenance if mixing experiments from different sequencers.
Walkthrough: Cross-Tissue CpG Methylation Atlas for Imprinted Gene Regions
Goal: Aggregate whole-genome bisulfite sequencing (WGBS) data across tissues to identify tissue-invariant vs. tissue-specific methylation patterns at imprinted gene loci. Context: Imprinted genes show parent-of-origin-specific methylation. Comparing across tissues reveals which imprinting control regions (ICRs) maintain methylation universally.
Step 1: Find WGBS experiments across tissues
encode_search_experiments(assay_title="WGBS", organism="Homo sapiens", limit=50)
Expected output:
{
"results": [
{"accession": "ENCSR765JPC", "assay_title": "WGBS", "organ": "liver", "biosample_summary": "liver tissue male adult (54 years)", "status": "released"},
{"accession": "ENCSR832HMR", "assay_title": "WGBS", "organ": "brain", "biosample_summary": "brain tissue female adult (53 years)", "status": "released"}
],
"total": 147,
"limit": 50,
"offset": 0,
"has_more": true,
"next_offset": 50
}
Interpretation: 147 WGBS experiments available. Select tissues with ≥2 replicates for reliable per-CpG averaging.
Step 2: List methylation bedGraph files
encode_list_files(experiment_accession="ENCSR765JPC", file_format="bed", output_type="methylation state at CpG", assembly="GRCh38")
Expected output (a JSON array of file records; fields abridged):
[
{"accession": "ENCFF123BED", "output_type": "methylation state at CpG", "file_format": "bed", "file_type": "bed bedMethyl", "assembly": "GRCh38", "file_size": 256901120, "file_size_human": "245.0 MB", "status": "released"}
]
Step 3: Download methylation files
encode_download_files(file_accessions=["ENCFF123BED", "ENCFF456MET", "ENCFF789CPG"], download_dir="/data/wgbs")
Step 4: Per-CpG weighted averaging across replicates
For each tissue:
- Merge replicates using weighted average: β = Σ(methylated reads) / Σ(total reads)
- Filter CpGs with <10× combined coverage
- Output: per-tissue methylation BED with columns: chr, start, end, β-value, coverage
Step 5: Identify tissue-invariant ICRs
# H19/IGF2 ICR: chr11:2,016,000-2,022,000
bedtools intersect -a merged_methylation.bed -b imprinted_icrs.bed -wa -wb | \
awk '{sum+=$4; n++} END {print sum/n}'
Interpretation: ICRs showing ~50% methylation across ALL tissues confirm maintained imprinting. Tissue-variable ICRs (range >20%) suggest tissue-specific imprinting loss.
Integration with downstream skills
- Feed differentially methylated regions into → peak-annotation for nearest gene assignment
- Overlay with → histone-aggregation H3K4me3 to find promoter methylation–expression anticorrelation
- Cross-reference CpG variants via → clinvar-annotation for methylation-disrupting mutations
- Compare methylation at regulatory elements from → regulatory-elements
Code Examples
1. Survey WGBS data availability by organ
encode_get_facets(assay_title="WGBS", organism="Homo sapiens")
Expected output:
{
"biosample_ontology.organ_slims": [
{"term": "brain", "count": 32},
{"term": "liver", "count": 18},
{"term": "heart", "count": 12},
{"term": "lung", "count": 10},
{"term": "blood", "count": 8},
{"term": "kidney", "count": 6}
]
}
2. Check experiment quality before aggregation
encode_get_experiment(accession="ENCSR765JPC")
Expected output:
{
"accession": "ENCSR765JPC",
"assay_title": "WGBS",
"biosample_summary": "liver tissue male adult (54 years)",
"assembly": ["GRCh38"],
"bio_replicate_count": 2,
"status": "released",
"audit_error_count": 0,
"audit_warning_count": 0
}
3. Track aggregated experiments
encode_track_experiment(accession="ENCSR765JPC", notes="Liver WGBS for cross-tissue methylation atlas")
Expected output:
{
"tracking": {"accession": "ENCSR765JPC", "action": "tracked"},
"publications_found": 0,
"publications": [],
"pipelines_found": 1,
"pipelines": [
{"title": "WGBS paired-end pipeline", "version": "1.1.6", "software": [{"name": "bismark", "version": "0.22.3"}], "status": "released"}
]
}
Integration
| This skill produces... | Feed into... | Purpose |
|---|---|---|
| Per-CpG β-value matrix | regulatory-elements | Identify methylation at cCREs and enhancers |
| Differentially methylated regions (DMRs) | peak-annotation | Assign DMRs to nearest genes |
| Tissue-specific hypomethylated regions | histone-aggregation | Correlate with H3K4me3 active promoter marks |
| Methylation at CpG islands | variant-annotation | Find variants disrupting CpG sites |
| HMR/UMR/PMD boundaries | accessibility-aggregation | Overlay open chromatin at unmethylated regions |
| Cross-tissue methylation atlas | visualization-workflow | Generate methylation heatmaps across tissues |
| CpG methylation at GWAS loci | gwas-catalog | Annotate trait-associated variants with methylation context |
Related Skills
- histone-aggregation: HMRs + H3K27ac union peaks identify active enhancers; HMRs + H3K4me3 identify active promoters
- accessibility-aggregation: HMRs typically overlap open chromatin; concordance between HMRs and ATAC/DNase peaks validates both
- hic-aggregation: Hypomethylated enhancers often anchor chromatin loops to target genes
- regulatory-elements: Combine methylation with histone and accessibility data to classify regulatory element types
- epigenome-profiling: Methylation adds a critical layer to chromatin state annotation
- pipeline-wgbs: Process raw WGBS data through the full ENCODE-aligned pipeline
- batch-analysis: Batch processing workflows for systematic methylation aggregation
- publication-trust: Verify literature claims backing analytical decisions
Presenting Results
- Present methylation summary as: total CpGs analyzed | mean coverage | methylation distribution (UMR/LMR/PMD). Show per-sample contribution. Suggest: "Would you like to correlate with histone marks?"
For the request: "$ARGUMENTS"
Files (encode-toolkit)
-
references
-
hmr-definitions.md 4.2 KB
# Hypomethylated Region (HMR) Definitions and Classification ## Overview DNA methylation landscapes contain three major domain types defined by their methylation levels. Correct identification and classification of these domains is essential for interpreting ENCODE WGBS aggregation results. ## Domain Definitions ### UMR — Unmethylated Region - **Methylation level**: <10% average CpG methylation - **Typical size**: 200bp - 5kb - **Genomic location**: CpG islands, active promoters - **Biological function**: Marks active regulatory elements, especially promoters - **Identification**: Contiguous CpGs with <10% methylation, minimum 4 CpGs - **Reference**: Schultz et al. 2015 (Nature, ~1,500 citations) DOI: 10.1038/nature14248 ### LMR — Low-Methylated Region - **Methylation level**: 10-50% average CpG methylation - **Typical size**: 200bp - 2kb - **Genomic location**: Distal enhancers, TF binding sites - **Biological function**: Marks active or poised enhancers, TF binding reduces methylation - **Identification**: Contiguous CpGs with 10-50% methylation - **Key insight**: LMRs correlate strongly with ENCODE H3K4me1 and H3K27ac peaks - **Reference**: Stadler et al. 2011 (Nature, ~1,200 citations) DOI: 10.1038/nature10716 ### PMD — Partially Methylated Domain - **Methylation level**: 50-70% average methylation across large domains - **Typical size**: 100kb - 10Mb - **Genomic location**: Gene-poor, late-replicating regions - **Biological function**: Associated with heterochromatin, gene silencing, aging - **Identification**: Large domains where bulk methylation drops below genome average (~80%) - **Important caveat**: PMDs are cell-type specific and expand with cellular aging/passaging - **Reference**: Lister et al. 2009 (Nature, ~5,000 citations) DOI: 10.1038/nature08514 ### HMR — Hypomethylated Region (general term) - Umbrella term encompassing UMRs and LMRs - Some tools (e.g., MethPipe) call HMRs without distinguishing UMR vs LMR - If using HMR calls, separate by size and CpG density: large + CpG-dense = UMR, small + CpG-poor = LMR ## Coverage Sensitivity HMR identification is highly sensitive to sequencing coverage: | Coverage | Reliability | Notes | |----------|------------|-------| | <5x per CpG | Unreliable | High noise, many false HMRs | | 5-10x | Acceptable | Sufficient for most analyses | | 10-30x | Good | ENCODE standard for WGBS | | >30x | Excellent | Diminishing returns above 30x | ### Minimum Coverage Thresholds - **Per-CpG minimum**: Require >=5 reads covering each CpG before including in calculations - **Per-region minimum**: Require >=4 CpGs meeting coverage threshold within each domain - **Strand merging**: Merge + and - strand counts at each CpG (doubles effective coverage) ## Tools for Domain Calling | Tool | Calls | Method | Reference | |------|-------|--------|-----------| | MethPipe | HMRs | HMM-based, 2-state model | Song et al. 2013 | | MethylSeekR | UMRs, LMRs, PMDs | Segmentation with FDR | Burger et al. 2013 | | DMRcate | DMRs | Kernel smoothing | Peters et al. 2021 | | Roadmap method | UMRs, LMRs, PMDs | Thresholds + size | Schultz et al. 2015 | ### Recommended Approach (from Aggregation Skill) For union-based aggregation of ENCODE WGBS data: 1. Call per-CpG methylation with >=5x coverage filter 2. Compute weighted average across samples (coverage-weighted) 3. Segment into UMR/LMR/PMD using Schultz 2015 thresholds 4. For HMR union: a CpG detected as hypomethylated in ANY sample enters the union catalog ## Cross-Reference with ENCODE Marks | Domain | Expected ENCODE marks | |--------|-----------------------| | UMR | H3K4me3 (active promoter), ATAC-seq peak, DNase-seq peak | | LMR | H3K4me1, H3K27ac (active enhancer), ATAC-seq peak | | PMD | H3K9me3 (heterochromatin), low ATAC/DNase signal | | Fully methylated | H3K36me3 (gene body), or no marks (intergenic) | ## References - Schultz et al. 2015 — Roadmap Epigenomics methylation landscape (Nature, ~1,500 cit) - Stadler et al. 2011 — LMRs at distal regulatory elements (Nature, ~1,200 cit) - Lister et al. 2009 — First whole-genome methylome, PMD discovery (Nature, ~5,000 cit) - Burger et al. 2013 — MethylSeekR UMR/LMR/PMD segmentation (Genome Biology) - Peters et al. 2021 — DMRcate differential methylation (NAR) -
literature.md 7.9 KB
# Methylation Aggregation — Literature References **Last updated:** 2026-03-07 **Purpose:** Reference catalog for the methylation-aggregation skill — papers supporting per-CpG weighted averaging for methylation aggregation, HMR/UMR/PMD identification, and cross-platform WGBS comparability. --- ## DNA Methylation Landscape --- ### Schultz et al. 2015 — Human body epigenome maps (Roadmap Epigenomics) - **Citation:** Schultz MD, He Y, Whitaker JW, Hariharan M, Mukamel EA, Leung D, Rajagopal N, Nery JR, Urich MA, Chen H, Lin S, Lin Y, Jung I, Schmitt AD, Selvaraj S, Ren B, Sejnowski TJ, Wang W, Ecker JR. Human body epigenome maps reveal noncanonical DNA methylation variation. Nature, 523(7559):212-216, 2015. - **DOI:** [10.1038/nature14248](https://doi.org/10.1038/nature14248) - **PMID:** 26030523 | **PMC:** PMC4539777 - **Citations:** ~2,900 - **Key findings:** Generated WGBS methylomes for 18 human tissues as part of the Roadmap Epigenomics Project. Established per-CpG averaging across biological replicates as the standard approach for building tissue-level methylation profiles. Identified three key genomic methylation classes: highly methylated regions (HMRs, >80%), partially methylated domains (PMDs, 50-80%), and unmethylated regions (UMRs, <20%). Tissue-specific UMRs mark active enhancers and promoters, while PMDs are associated with late-replicating heterochromatin. This paper provides the analytical framework for the two-level aggregation approach in this skill: per-CpG averaging followed by region-level classification. --- ### Zhou et al. 2020 — Tissue-specific methylation patterns - **Citation:** Zhou W, Dinh HQ, Ramjan Z, Weisenberger DJ, Nicolet CM, Shen H, Laird PW, Berman BP. DNA methylation loss in late-replicating domains is linked to mitotic cell division. Nature Genetics, 50(4):591-602, 2018. - **DOI:** [10.1038/s41588-018-0073-4](https://doi.org/10.1038/s41588-018-0073-4) - **PMID:** 29610480 | **PMC:** PMC5893360 - **Citations:** ~300 - **Key findings:** Characterized the genome-wide distribution of tissue-specific methylation: ~80% of CpGs are constitutively methylated across tissues, ~10% are constitutively unmethylated (CpG islands and active promoters), and ~10% show tissue-variable methylation. This 80/10/10 distribution informs the expected output of methylation aggregation — the tissue-variable fraction is where biologically interesting differences occur, while the constitutive fractions serve as internal validation that the averaging is working correctly. --- ## Methylation Region Analysis --- ### Peters et al. 2015 — DMRcate: differentially methylated region detection - **Citation:** Peters TJ, Buckley MJ, Statham AL, Pidsley R, Samaras K, Lord RV, Clark SJ, Molloy PL. De novo identification of differentially methylated regions in the human genome. Epigenetics & Chromatin, 8:6, 2015. - **DOI:** [10.1186/1756-8935-8-6](https://doi.org/10.1186/1756-8935-8-6) - **PMID:** 25972926 | **PMC:** PMC4429357 - **Citations:** ~900 - **Key findings:** Introduced DMRcate, which uses kernel smoothing across CpG sites followed by bump-hunting to identify differentially methylated regions (DMRs). Works with both array (450K/EPIC) and WGBS data. Relevant for downstream analysis of aggregated methylation profiles — after per-CpG averaging, DMRcate can identify regions where the averaged methylation level differs from a reference or between tissues. --- ### Hansen et al. 2012 — BSmooth: WGBS smoothing - **Citation:** Hansen KD, Langmead B, Irizarry RA. BSmooth: from whole genome bisulfite sequencing reads to differentially methylated regions. Genome Biology, 13(10):R83, 2012. - **DOI:** [10.1186/gb-2012-13-10-r83](https://doi.org/10.1186/gb-2012-13-10-r83) - **PMID:** 23034175 | **PMC:** PMC3491411 - **Citations:** ~1,200 - **Key findings:** Demonstrated that single-CpG methylation estimates from WGBS are inherently noisy even at high coverage, and that smoothing across neighboring CpGs substantially improves accuracy. The bsseq R/Bioconductor package implements local-likelihood smoothing for WGBS data. Relevant for the aggregation workflow because it establishes that per-CpG averaging should be followed by spatial smoothing to produce reliable methylation profiles for HMR/UMR/PMD calling. --- ## Technical Considerations --- ### Liu et al. 2024 — Cross-platform WGBS comparison - **Citation:** Liu Z, et al. Systematic assessment of WGBS between platforms: implications for DNA methylation profiling. Briefings in Bioinformatics, 2024. - **Key findings:** Compared WGBS performance across sequencing platforms (NovaSeq 6000 vs DNBSEQ-T7), confirming WGBS as the gold standard for comprehensive methylation profiling. Identified platform-specific differences in GC-rich regions and demonstrated that coverage depth critically affects methylation estimation accuracy — sites with <5x coverage have unreliable estimates. Supports the coverage-weighted averaging approach in this skill, where each CpG's contribution is weighted by coverage depth, and the minimum coverage threshold of 5x for inclusion in the averaged profile. --- ### Ortega-Recalde et al. 2021 — Low-coverage WGBS accuracy - **Citation:** Ortega-Recalde O, et al. Accurate estimation of global DNA methylation from low-coverage sequencing data. Methods in Molecular Biology, 2021. - **Key findings:** Demonstrated that even low-coverage WGBS can accurately estimate global and regional methylation levels when bootstrap methods are used to quantify uncertainty. For per-CpG aggregation, low-coverage samples contribute to the weighted average with lower weight (proportional to coverage), preventing them from distorting the aggregated profile while still contributing information. This validates the coverage-weighted averaging formula used in this skill: averaged_methylation = sum(methylation_i * coverage_i) / sum(coverage_i). --- ### Lister et al. 2009 — Human DNA methylomes at base resolution - **Citation:** Lister R, Pelizzola M, Dowen RH, et al. Human DNA methylomes at base resolution show widespread epigenomic differences. Nature, 462(7271):315-322, 2009. - **DOI:** [10.1038/nature08514](https://doi.org/10.1038/nature08514) - **PMID:** 19829295 | **PMC:** PMC2857523 - **Citations:** ~5,000 - **Key findings:** First base-resolution WGBS methylomes establishing the foundational concepts for methylation aggregation: PMDs as large domains of partial methylation in differentiated cells, non-CpG methylation in stem cells, and UMRs at active regulatory elements. The PMD/HMR/UMR classification scheme introduced here is the basis for the region-level analysis performed after per-CpG averaging in this skill. --- ## Data Integration Framework --- ### ENCODE Project Consortium 2020 — Expanded encyclopaedias of DNA elements - **Citation:** ENCODE Project Consortium et al. Expanded encyclopaedias of DNA elements in the human and mouse genomes. Nature, 583(7818):699-710, 2020. - **DOI:** [10.1038/s41586-020-2493-4](https://doi.org/10.1038/s41586-020-2493-4) - **PMID:** 32728249 | **PMC:** PMC7410828 - **Citations:** ~2,500 - **Key findings:** ENCODE Phase 3 established DNA methylation as a core epigenomic layer integrated with histone modifications and chromatin accessibility for candidate cis-Regulatory Element (cCRE) classification. WGBS data is used to identify methylation valleys (large UMRs at key developmental genes) and to provide the methylation context for regulatory element classification. Established ENCODE WGBS processing standards used as quality gates in this skill's aggregation workflow. --- ### Amemiya et al. 2019 — ENCODE Blacklist - **DOI:** [10.1038/s41598-019-45839-z](https://doi.org/10.1038/s41598-019-45839-z) | **PMID:** 31249361 | **Citations:** ~1,372 - **Methylation aggregation role:** CpGs within blacklisted regions may show aberrant methylation estimates due to alignment artifacts in repetitive sequences. These CpGs should be filtered from the aggregated profile to prevent false HMR/UMR calls at blacklisted loci.
-
-
scripts
-
validate_methylation.py 16.5 KB
#!/usr/bin/env python3 """Validate bedMethyl files from ENCODE WGBS methylation aggregation. Checks bedMethyl format, methylation value ranges, strand validity, coverage values, and reports summary statistics with warnings for low-coverage sites. Usage: python validate_methylation.py input.bedMethyl [--min-coverage 5] [--blacklist hg38-blacklist.v2.bed] python validate_methylation.py input.bed --min-coverage 10 python validate_methylation.py input.bedMethyl --scale fraction Plain and gzipped (.gz) inputs and blacklists are both accepted. """ import argparse import gzip import statistics import sys from collections import Counter, defaultdict from pathlib import Path VALID_CHROMS = {f"chr{i}" for i in range(1, 23)} | {"chrX", "chrY", "chrM"} VALID_STRANDS = {"+", "-", "."} # ENCODE bedMethyl format has 11 columns: # chr start end name score strand thickStart thickEnd color coverage percentMethylated BEDMETHYL_COLS = 11 # Alternative minimal format: chr start end name score strand coverage methylation% MINIMAL_COLS = 8 # layout -> (coverage column, methylation column, how the column count is described), 0-indexed LAYOUTS = { "encode_bedmethyl": (9, 10, "11+"), "minimal": (6, 7, "8"), } MAX_COLUMN_ERRORS = 5 def parse_args(): parser = argparse.ArgumentParser( description="Validate bedMethyl files from ENCODE WGBS methylation aggregation.", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=( "Examples:\n" " python validate_methylation.py sample.bedMethyl\n" " python validate_methylation.py sample.bedMethyl --min-coverage 10\n" " python validate_methylation.py sample.bed --blacklist hg38-blacklist.v2.bed\n" ), ) parser.add_argument("input", type=Path, help="Input bedMethyl file") parser.add_argument( "--min-coverage", type=int, default=5, help="Minimum coverage threshold to flag low-coverage CpGs. Default: 5", ) parser.add_argument( "--blacklist", type=Path, default=None, help="ENCODE blacklist BED file (e.g., hg38-blacklist.v2.bed)", ) parser.add_argument( "--scale", choices=["auto", "percent", "fraction"], default="auto", help=( "Scale of the methylation column, decided once for the whole file. " "auto: percent for the ENCODE 11-column layout, and for the minimal " "layout fraction only when the file's maximum value is <= 1. Default: auto" ), ) return parser.parse_args() def open_text(path): """Open a plain or gzipped text file for reading.""" if str(path).endswith(".gz"): return gzip.open(path, "rt") return open(path) def quartiles(values): """25th percentile, median and 75th percentile (interpolated).""" median = statistics.median(values) if len(values) < 2: return values[0], median, values[0] q1, _, q3 = statistics.quantiles(values, n=4, method="inclusive") return q1, median, q3 def load_blacklist(path): """Load blacklist regions as a dict of chrom -> list of (start, end).""" regions = defaultdict(list) with open_text(path) as f: for line in f: if line.startswith("#") or line.strip() == "": continue parts = line.strip().split("\t") if len(parts) >= 3: regions[parts[0]].append((int(parts[1]), int(parts[2]))) for chrom in regions: regions[chrom].sort() return regions def overlaps_blacklist(chrom, start, end, blacklist): """Check if a region overlaps any blacklist interval.""" if chrom not in blacklist: return False for bl_start, bl_end in blacklist[chrom]: if bl_start >= end: break if bl_end > start: return True return False def detect_format(n_cols): """ENCODE bedMethyl (11 or more columns), minimal (exactly 8), or None for anything else.""" if n_cols >= BEDMETHYL_COLS: return "encode_bedmethyl" if n_cols == MINIMAL_COLS: return "minimal" return None def layout_matches(detected_format, n_cols): """Check a later line against the layout detected from the first data line.""" if detected_format == "encode_bedmethyl": return n_cols >= BEDMETHYL_COLS return n_cols == MINIMAL_COLS def decide_scale(requested, detected_format, values): """Decide once per file whether the methylation column holds percentages or fractions.""" if requested != "auto": return requested # ENCODE bedMethyl column 11 is a percentage (0-100) by specification. if detected_format == "encode_bedmethyl": return "percent" return "fraction" if values and max(values) <= 1.0 else "percent" def validate_methylation(input_path, min_coverage, blacklist_path, requested_scale): errors = [] warnings = [] chrom_counts = Counter() strand_counts = Counter() coverage_values = [] records = [] # rows whose fields all parsed: (line, chrom, strand, coverage, methylation, in blacklist) total_lines = 0 # Every line excluded from the statistics below counts as malformed, so # total_lines == valid_records + bad_lines always holds. bad_lines = 0 column_errors = 0 low_coverage = 0 blacklist_overlaps = 0 blacklist = None if blacklist_path: if not blacklist_path.exists(): print(f"ERROR: Blacklist file not found: {blacklist_path}", file=sys.stderr) sys.exit(1) blacklist = load_blacklist(blacklist_path) if not input_path.exists(): print(f"ERROR: Input file not found: {input_path}", file=sys.stderr) sys.exit(1) detected_format = None expected_desc = None cov_col = None # 0-indexed column for coverage meth_col = None # 0-indexed column for methylation with open_text(input_path) as f: for line_num, line in enumerate(f, 1): if line.startswith("#") or line.startswith("track") or line.startswith("browser"): continue line = line.strip() if not line: continue total_lines += 1 fields = line.split("\t") # The first data line fixes the layout for the whole file if detected_format is None: detected_format = detect_format(len(fields)) if detected_format is None: print( f"ERROR: Could not detect bedMethyl format. " f"Expected 8 columns (minimal) or >= 11 columns (ENCODE bedMethyl), got {len(fields)}", file=sys.stderr, ) sys.exit(1) cov_col, meth_col, expected_desc = LAYOUTS[detected_format] if not layout_matches(detected_format, len(fields)): bad_lines += 1 column_errors += 1 if column_errors <= MAX_COLUMN_ERRORS: errors.append( f"Line {line_num}: expected {expected_desc} columns ({detected_format}), got {len(fields)}" ) elif column_errors == MAX_COLUMN_ERRORS + 1: errors.append("... suppressing further column-count errors") continue chrom = fields[0] if chrom not in VALID_CHROMS: if not chrom.startswith("chr"): errors.append(f"Line {line_num}: invalid chromosome '{chrom}'") bad_lines += 1 continue # Coordinate validation try: start = int(fields[1]) end = int(fields[2]) except ValueError: errors.append(f"Line {line_num}: non-integer coordinates") bad_lines += 1 continue if start < 0: errors.append(f"Line {line_num}: negative start coordinate ({start})") if end < 0: errors.append(f"Line {line_num}: negative end coordinate ({end})") if start >= end: errors.append(f"Line {line_num}: start ({start}) >= end ({end})") # an impossible interval is malformed: count it once and keep it out of the statistics if start < 0 or end < 0 or start >= end: bad_lines += 1 continue # Every field is checked before anything is counted: a row with an unusable strand, # coverage or methylation value is malformed and stays out of every statistic. row_ok = True strand = fields[5] if strand not in VALID_STRANDS: errors.append(f"Line {line_num}: invalid strand '{strand}' (expected +, -, or .)") row_ok = False try: coverage = int(float(fields[cov_col])) # some tools write coverage as a float except ValueError: errors.append(f"Line {line_num}: invalid coverage in column {cov_col + 1}") row_ok = False else: if coverage < 0: errors.append(f"Line {line_num}: negative coverage ({coverage})") row_ok = False try: meth = float(fields[meth_col]) except ValueError: errors.append(f"Line {line_num}: invalid methylation value in column {meth_col + 1}") row_ok = False else: if meth < 0: errors.append(f"Line {line_num}: negative methylation value ({meth})") row_ok = False if not row_ok: bad_lines += 1 continue # The methylation scale is known only after the whole file is read, so the row is # kept here and counted below, once its value is known to be in range. in_blacklist = bool(blacklist) and overlaps_blacklist(chrom, start, end, blacklist) records.append((line_num, chrom, strand, coverage, meth, in_blacklist)) if total_lines == 0: print(f"ERROR: no data rows in {input_path} (only comments, headers or blank lines)", file=sys.stderr) sys.exit(1) # --- Apply one scale to every methylation value, then count the rows that are in range --- scale = decide_scale(requested_scale, detected_format, [record[4] for record in records]) upper, factor = (1.0, 100) if scale == "fraction" else (100, 1) methylation_values = [] for line_num, chrom, strand, coverage, meth, in_blacklist in records: if meth > upper: if scale == "fraction": errors.append(f"Line {line_num}: methylation value {meth} > 1 with --scale fraction (expected 0-1).") else: errors.append(f"Line {line_num}: methylation value > 100 ({meth}). Expected 0-100 (percentage).") bad_lines += 1 continue chrom_counts[chrom] += 1 strand_counts[strand] += 1 coverage_values.append(coverage) if coverage < min_coverage: low_coverage += 1 if in_blacklist: blacklist_overlaps += 1 methylation_values.append(meth * factor) valid_records = total_lines - bad_lines # --- Report Statistics --- print("=== bedMethyl Validation Report ===") print(f"File: {input_path}") print(f"Detected format: {detected_format} ({expected_desc} columns)") print() print("--- Summary ---") print(f"Data lines: {total_lines:,}") print(f"Valid CpGs: {valid_records:,}") print(f"Malformed lines: {bad_lines}") print(f"Low-coverage CpGs (<{min_coverage}x): {low_coverage:,} ({100 * low_coverage / max(valid_records, 1):.1f}%)") if blacklist_path: print(f"Blacklist overlaps: {blacklist_overlaps:,} ({100 * blacklist_overlaps / max(valid_records, 1):.1f}%)") print() scale_source = "auto-detected" if requested_scale == "auto" else f"--scale {requested_scale}" scale_label = "fraction (0-1), reported as percent" if scale == "fraction" else "percent (0-100)" print(f"Methylation scale: {scale_label} [{scale_source}]") print() if coverage_values: sorted_cov = sorted(coverage_values) n = len(sorted_cov) cov_q1, cov_median, cov_q3 = quartiles(sorted_cov) print("--- Coverage Distribution ---") print(f"Min: {sorted_cov[0]:>6}") print(f"25th: {cov_q1:>6.1f}") print(f"Median: {cov_median:>6.1f}") print(f"75th: {cov_q3:>6.1f}") print(f"Max: {sorted_cov[-1]:>6}") # Coverage buckets buckets = [ ("<3x", 0, 3), ("3-5x", 3, 5), ("5-10x", 5, 10), ("10-20x", 10, 20), ("20-50x", 20, 50), (">=50x", 50, float("inf")), ] print("\n Coverage buckets:") for label, lo, hi in buckets: count = sum(1 for c in coverage_values if lo <= c < hi) print(f" {label:<8} {count:>10,} ({100 * count / n:.1f}%)") print() if methylation_values: sorted_meth = sorted(methylation_values) n_m = len(sorted_meth) meth_q1, meth_median, meth_q3 = quartiles(sorted_meth) print("--- Methylation Distribution (as %) ---") print(f"Min: {sorted_meth[0]:>6.1f}%") print(f"25th: {meth_q1:>6.1f}%") print(f"Median: {meth_median:>6.1f}%") print(f"75th: {meth_q3:>6.1f}%") print(f"Max: {sorted_meth[-1]:>6.1f}%") # Methylation state buckets buckets = [ ("Unmethylated (0-10%)", 0, 10), ("Low (10-30%)", 10, 30), ("Intermediate (30-70%)", 30, 70), ("High (70-90%)", 70, 90), ("Methylated (90-100%)", 90, 100.01), ] print("\n Methylation state distribution:") for label, lo, hi in buckets: count = sum(1 for m in methylation_values if lo <= m < hi) print(f" {label:<30} {count:>10,} ({100 * count / n_m:.1f}%)") print() print("--- Strand Breakdown ---") for strand in ["+", "-", "."]: count = strand_counts.get(strand, 0) pct = 100 * count / max(valid_records, 1) print(f" {strand:<3} {count:>10,} ({pct:5.1f}%)") print() print("--- Chromosome Distribution ---") for chrom in sorted(chrom_counts.keys(), key=lambda c: (len(c), c)): count = chrom_counts[chrom] pct = 100 * count / max(valid_records, 1) print(f" {chrom:<6} {count:>10,} ({pct:5.1f}%)") print() # --- Warnings --- if low_coverage > valid_records * 0.3: msg = ( f"WARNING: {100 * low_coverage / max(valid_records, 1):.0f}% of CpGs have " f"coverage <{min_coverage}x. Consider filtering these for reliable " f"methylation estimates." ) print(msg, file=sys.stderr) if valid_records and strand_counts.get(".", 0) == valid_records: msg = ( "INFO: All CpGs have strand '.'. This file may already be strand-merged. " "Skip the strand-merge step in aggregation." ) print(msg, file=sys.stderr) elif strand_counts.get("+", 0) > 0 and strand_counts.get("-", 0) > 0: plus_count = strand_counts.get("+", 0) minus_count = strand_counts.get("-", 0) ratio = plus_count / max(minus_count, 1) if 0.8 <= ratio <= 1.2: msg = ( f"INFO: Both strands present ({plus_count:,} forward, {minus_count:,} reverse). " f"Consider strand-merging for increased per-CpG coverage." ) print(msg, file=sys.stderr) if blacklist_overlaps > 0: msg = f"WARNING: {blacklist_overlaps:,} CpGs overlap ENCODE blacklist regions. Remove these before aggregation." print(msg, file=sys.stderr) for w in warnings[:20]: print(w, file=sys.stderr) # --- Errors --- if errors: print(f"\n--- Errors ({len(errors)}) ---", file=sys.stderr) for e in errors[:50]: print(f" {e}", file=sys.stderr) if len(errors) > 50: print(f" ... and {len(errors) - 50} more errors", file=sys.stderr) has_errors = len(errors) > 0 if has_errors: print(f"\nRESULT: FAIL -- {len(errors)} error(s) found", file=sys.stderr) else: print(f"\nRESULT: PASS -- file is valid {detected_format}") return 1 if has_errors else 0 if __name__ == "__main__": args = parse_args() exit_code = validate_methylation(args.input, args.min_coverage, args.blacklist, args.scale) sys.exit(exit_code)
-
-
SKILL.md 23.2 KB
--- name: methylation-aggregation description: Build comprehensive DNA methylation maps by aggregating WGBS (Whole Genome Bisulfite Sequencing) data across multiple ENCODE experiments, donors, and labs. Use when the user wants to answer "where is DNA methylated/unmethylated in my tissue?" by combining per-CpG methylation data into tissue-level methylation profiles. Handles coverage filtering, identifies hypomethylated regions (HMRs) and partially methylated domains (PMDs), and manages cross-lab variation. --- # Aggregate DNA Methylation Data Across Studies ## When to Use - User wants to build a tissue-level DNA methylation landscape from multiple WGBS experiments - User asks "where is DNA methylated in brain?" or "find hypomethylated regions across donors" - User needs to identify HMRs (hypomethylated regions), UMRs, or PMDs from aggregated WGBS data - User wants per-CpG weighted methylation averages from multiple experiments - Example queries: "aggregate WGBS data for liver", "build methylation map across donors", "find unmethylated CpG islands in pancreas" Build a comprehensive methylation landscape for a tissue/cell type by merging WGBS bedMethyl files from multiple ENCODE experiments. ## Scientific Rationale **The question**: "What is the DNA methylation state across the genome in my tissue?" DNA methylation is **fundamentally different** from histone marks and accessibility: | Property | Histone/Accessibility | DNA Methylation | |----------|----------------------|-----------------| | Signal type | Binary (bound/open or not) | Continuous (0-100% methylated) | | Default state | Unmarked | ~70-80% methylated (CpG context) | | Biology of interest | Where marks ARE present | Where methylation is ABSENT or REDUCED | | Aggregation approach | Union of peak calls | Average/median of methylation levels per CpG | **The key insight**: Unlike histone ChIP-seq where we want the union of all peaks, for methylation we want the **average methylation level per CpG site** across individuals. Methylation is a quantitative, continuous signal measured at every CpG dinucleotide. **However**, for identifying regulatory regions, we focus on **hypomethylated regions (HMRs)** — stretches of low methylation that mark active regulatory elements. HMRs can be treated more like peaks for union-style aggregation. ### Literature Support - **Roadmap Epigenomics** (Schultz et al. 2015, Nature, 2,900+ citations): Established that tissue-specific HMRs mark active regulatory elements; demonstrated per-CpG averaging across biological replicates as standard approach - **DMRcate** (Peters et al. 2021, Nucleic Acids Research, 65 citations): Method for calling differentially methylated regions from multiple WGBS samples; uses kernel smoothing across CpG sites - **ENCODE Phase 3** (Gorkin et al. 2020, Nature, 301 citations): Integrated methylation data with histone marks and accessibility to define chromatin states - **ENCODE Blacklist** (Amemiya et al. 2019, Scientific Reports, 1,372 citations): Problematic genomic regions to filter. [DOI](https://doi.org/10.1038/s41598-019-45839-z) - **Zhou et al. 2020** (Nature Genetics): Tissue-specific methylation patterns: ~80% of CpGs are constitutively methylated, ~10% constitutively unmethylated (CpG islands/promoters), ~10% tissue-variable - **Liu et al. 2024** (Briefings in Bioinformatics): Cross-platform comparison (NovaSeq vs DNBSEQ) showing WGBS is gold standard; coverage depth critically affects accuracy; platform differences exist in GC-rich regions - **Ortega-Recalde et al. 2021** (Methods in Molecular Biology): Demonstrated that even low-coverage WGBS can accurately estimate global methylation levels, with bootstrap methods to quantify uncertainty ### Two-Level Analysis 1. **Per-CpG level**: Average methylation at each CpG site across samples (quantitative map) 2. **Region level**: Identify HMRs, PMDs, and UMRs from the averaged profile (union of regulatory regions) ## Step 1: Find All Available WGBS Data ``` encode_search_experiments( assay_title="WGBS", organ="pancreas", # user's tissue of interest biosample_type="tissue", limit=100 ) ``` Present a summary to the user: - Total WGBS experiments - Labs represented - Unique donors/biosamples - Genome coverage per experiment Use `encode_get_facets` to check availability: ``` encode_get_facets(assay_title="WGBS", organ="pancreas") ``` **Note**: WGBS is expensive to generate. Typical tissues have 2-5 experiments. Even 2 biological replicates are valuable for identifying consistent methylation patterns. ## Step 2: Quality-Gate Each Experiment ``` encode_get_experiment(accession="ENCSR...") ``` ### WGBS Quality Checks - Audit status: no ERROR flags - **Bisulfite conversion rate**: >=98% (measured by lambda spike-in or non-CpG methylation) - **Genome coverage**: >=10x mean CpG coverage for reliable per-site estimates - **Mapping rate**: >=50% (bisulfite-converted reads are harder to map) - **Duplication rate**: <30% - Has `methylation state at CpG` output files (bedMethyl format) ### Include if: - Bisulfite conversion >=98% - Mean CpG coverage >=10x - Has bedMethyl output files for GRCh38 ### Exclude if: - ERROR audit flags - Conversion rate <98% (unconverted reads create false methylation calls) - Very low coverage (<5x mean) — individual CpG estimates unreliable Track all included experiments: ``` encode_track_experiment(accession="ENCSR...") ``` ## Step 3: Download bedMethyl Files For each experiment: ``` encode_list_files( experiment_accession="ENCSR...", output_type="methylation state at CpG", assembly="GRCh38" ) ``` **bedMethyl format** (ENCODE standard): ``` chr start end name score strand thickStart thickEnd color coverage percentMethylated ``` - Column 10: read coverage at this CpG - Column 11: percent methylation (0-100) Prefer `preferred_default=True` files: ``` encode_download_files( file_accessions=["ENCFF...", ...], download_dir="/path/to/data/methylation", organize_by="flat" ) ``` Validate the downloaded files before filtering. The scale of the methylation column is decided once per file; override it with `--scale` if the file is not ENCODE bedMethyl. Gzipped inputs are read directly. ```bash python3 scripts/validate_methylation.py sample.bedMethyl [--min-coverage 5] [--scale auto|percent|fraction] [--blacklist hg38-blacklist.v2.bed] ``` ## Step 4: Per-Sample Quality Filtering ### 4a. Coverage Filtering (CRITICAL) Low-coverage CpGs have unreliable methylation estimates. Filter per-sample: ```bash # Keep only CpGs with >= 5x coverage (column 10) # More stringent: >= 10x for quantitative analysis awk '$10 >= 5' sample.bedMethyl > sample.covfiltered.bedMethyl ``` **Coverage thresholds by use case:** | Threshold | Use Case | Typical CpGs Retained | |-----------|----------|----------------------| | >=3x | Exploratory / maximum retention | ~90% of CpGs | | >=5x | Standard analysis | ~80% of CpGs | | >=10x | High-confidence quantitative | ~60% of CpGs | ### 4b. ENCODE Blocklist Filtering (Amemiya et al. 2019) ```bash # Download from: https://github.com/Boyle-Lab/Blacklist/blob/master/lists/hg38-blacklist.v2.bed.gz gunzip -k hg38-blacklist.v2.bed.gz bedtools intersect -a sample.covfiltered.bedMethyl -b hg38-blacklist.v2.bed -v > sample.filtered.bedMethyl ``` ### 4c. Strand Merging (Optional but Recommended) CpG methylation is typically symmetric (same on both strands). Merge strand-specific calls to increase per-CpG coverage. **Caveat**: Some ENCODE bedMethyl files may already be strand-merged — check if both strands are present before applying this step: ```bash # Group CpGs by position (forward and reverse strand of same CpG) # Sum coverage, calculate weighted average methylation awk 'BEGIN{OFS="\t"} { # CpG position (use the C position as canonical) if ($6 == "+") pos = $2 else pos = $2 - 1 key = $1"\t"pos cov[key] += $10 meth[key] += ($11/100) * $10 } END { for (k in cov) { split(k, a, "\t") avg_meth = (meth[k] / cov[k]) * 100 print a[1], a[2], a[2]+2, "CpG", 0, ".", a[2], a[2]+2, "0,0,0", cov[k], avg_meth } }' sample.filtered.bedMethyl | sort -k1,1 -k2,2n > sample.merged_strands.bedMethyl ``` ## Step 5: Cross-Sample Aggregation (Per-CpG Averaging) ### 5a. Create a Unified CpG Matrix ```bash # Step 1: Find CpGs covered in at least M of N samples # Extract positions from each sample for f in sample*.merged_strands.bedMethyl; do awk 'BEGIN{OFS="\t"} {print $1, $2, $3}' "$f" done | sort -k1,1 -k2,2n | uniq -c | \ awk -v M=2 '$1 >= M {print $2, $3, $4}' OFS="\t" > shared_cpgs.bed # Step 2: For each sample, extract methylation at shared CpGs for f in sample*.merged_strands.bedMethyl; do bedtools intersect -a shared_cpgs.bed -b "$f" -wa -wb | \ awk 'BEGIN{OFS="\t"} {print $1, $2, $3, $NF, $(NF-1)}' > "${f%.bedMethyl}.shared.txt" # Columns: chr, start, end, percentMeth, coverage done ``` ### 5b. Calculate Average Methylation Per CpG **Weighted average** (recommended — accounts for coverage differences): ```bash # Combine all samples, calculate coverage-weighted mean methylation per CpG cat sample*.shared.txt | \ sort -k1,1 -k2,2n | \ awk 'BEGIN{OFS="\t"} { key = $1"\t"$2"\t"$3 if (key != prev_key && NR > 1) { avg = total_weighted_meth / total_cov print prev_key, n_samples, total_cov, avg n_samples = 0; total_cov = 0; total_weighted_meth = 0 } prev_key = key n_samples++ total_cov += $5 total_weighted_meth += ($4/100) * $5 } END { avg = total_weighted_meth / total_cov print prev_key, n_samples, total_cov, avg }' > tissue_methylation_profile.bed # Columns: chr, start, end, n_samples, total_coverage, mean_methylation_fraction ``` **Simple average** (alternative — equal weight per sample): ```bash # Unweighted mean across samples awk 'BEGIN{OFS="\t"} { key = $1"\t"$2"\t"$3 meth[key] += $4 n[key]++ } END { for (k in meth) { print k, n[k], meth[k]/n[k] } }' <(cat sample*.shared.txt) | sort -k1,1 -k2,2n > tissue_methylation_simple.bed ``` ### 5c. Calculate Methylation Variability Track inter-individual variation to identify tissue-variable CpGs: ```bash # Add standard deviation column # (compute in R or Python for large datasets) ``` ## Step 6: Identify Regulatory Methylation Features ### 6a. Hypomethylated Regions (HMRs) HMRs mark active regulatory elements. Identify runs of low methylation. The 30% threshold is a commonly used cutoff (Schultz et al. 2015 used similar ranges), but **the optimal threshold depends on your tissue and question** — some studies use 20%, others 40%. Consider visualizing the methylation distribution first to identify a natural breakpoint: ```bash # Find CpGs with average methylation < 30% (adjust threshold as needed) awk '$6 < 0.30' tissue_methylation_profile.bed > hypo_cpgs.bed # Merge adjacent hypomethylated CpGs into regions # Require minimum 3 CpGs within 1kb of each other bedtools merge -i hypo_cpgs.bed -d 1000 -c 1 -o count | \ awk '$4 >= 3' > tissue_HMRs.bed # Columns: chr, start, end, n_hypomethylated_CpGs ``` ### 6b. Unmethylated Regions (UMRs) — CpG Islands Very low methylation (<10%) at CpG-dense regions: ```bash awk '$6 < 0.10' tissue_methylation_profile.bed > unmeth_cpgs.bed bedtools merge -i unmeth_cpgs.bed -d 500 -c 1 -o count | \ awk '$4 >= 5' > tissue_UMRs.bed ``` ### 6c. Partially Methylated Domains (PMDs) Large (>10kb) regions of intermediate methylation, often marking repressed regions: ```bash # Find CpGs with methylation 30-70% (partially methylated) awk '$6 >= 0.30 && $6 <= 0.70' tissue_methylation_profile.bed > partial_cpgs.bed # Merge with large gap tolerance to find domains bedtools merge -i partial_cpgs.bed -d 5000 -c 1 -o count | \ awk '$4 >= 20 && ($3-$2) >= 10000' > tissue_PMDs.bed ``` ### 6d. Tissue-Specific Differentially Methylated Regions If comparing to another tissue, use DMRcate or similar: ```r # In R with DMRcate library(DMRcate) # Requires a methylation matrix (CpGs x samples with tissue labels) # Identifies regions where methylation differs between tissues ``` ## Step 7: Confidence Annotation For HMRs/UMRs (region-level features), annotate by sample support: | Confidence | Criteria | Interpretation | |-----------|----------|----------------| | **High** | Low methylation in >=50% of samples | Constitutive regulatory region | | **Supported** | Low methylation in 2+ samples | Likely regulatory, some variation | | **Variable** | High variance across samples | Cell-type heterogeneity or individual variation | ```bash # Annotate HMRs with sample support # Intersect each HMR with per-sample hypomethylated CpGs to count support awk -v N=4 '{ # Using n_samples from the aggregation if ($4 >= N*0.5) conf="HIGH"; else if ($4 >= 2) conf="SUPPORTED"; else conf="VARIABLE"; print $0"\t"conf"\t"$4"/"N }' tissue_HMRs.bed > tissue_HMRs.annotated.bed ``` For the per-CpG profile, annotate by coverage confidence: ```bash awk -v N=4 '{ if ($4 >= N) conf="ALL_SAMPLES"; else if ($4 >= N*0.5) conf="MAJORITY"; else conf="PARTIAL"; print $0"\t"conf }' tissue_methylation_profile.bed > tissue_methylation.annotated.bed ``` ## Step 7b: Summary Statistics Report to the user: - Total input experiments: N - Experiments passing QC: M (bisulfite conversion, coverage) - Total CpGs per sample (before/after coverage filter) - Shared CpGs across M+ samples: X - Mean genome-wide methylation: Y% - Number of HMRs: Z (with size distribution) - Number of UMRs: W - Number of PMDs: V (if applicable) - High-confidence HMRs: how many in ≥50% of samples - CpGs with high inter-individual variability ## Step 8: Integration with Other ENCODE Data Methylation data is most powerful when integrated: 1. **HMRs + H3K27ac peaks** = Active enhancers (use histone-aggregation skill) 2. **HMRs + ATAC-seq peaks** = Open regulatory elements (use accessibility-aggregation skill) 3. **UMRs + H3K4me3 peaks** = Active promoters 4. **PMDs** = Often overlap H3K9me3 (heterochromatin) ```bash # Example: Find HMRs that overlap H3K27ac peaks (active enhancers) bedtools intersect -a tissue_HMRs.bed -b union_H3K27ac_peaks.bed -wa -u > active_enhancer_HMRs.bed ``` ## Step 9: Log Provenance ``` encode_log_derived_file( file_path="/path/to/tissue_methylation.annotated.bed", source_accessions=["ENCSR...", "ENCSR...", ...], description="Aggregated per-CpG methylation profile across N pancreas WGBS experiments", file_type="aggregated_methylation", tool_used="bedtools + custom aggregation", parameters="coverage >= 5x per sample, shared CpGs in >= 2 samples, coverage-weighted mean, strand-merged" ) encode_log_derived_file( file_path="/path/to/tissue_HMRs.annotated.bed", source_accessions=["ENCSR...", "ENCSR...", ...], description="Hypomethylated regions from aggregated pancreas methylation profile", file_type="aggregated_HMRs", tool_used="bedtools merge", parameters="mean methylation < 30%, >= 3 CpGs within 1kb, confidence annotated" ) ``` ## Pitfalls Specific to Methylation Data 1. **Bisulfite conversion rate is critical**: Even 1% incomplete conversion creates false methylation at unmethylated CpGs. Always verify >=98% conversion. ENCODE reports this in QC metrics. 2. **Coverage drives accuracy**: A CpG with 3x coverage has wide confidence intervals (0-100% could easily be 0% or 30%). At 10x, estimates stabilize. At 30x, they are reliable. Always filter by coverage. 3. **Non-CpG methylation**: Present in some cell types (especially embryonic). ENCODE bedMethyl files typically report CpG context only. If non-CpG methylation is relevant, check experiment metadata. 4. **Strand asymmetry**: While CpG methylation is typically symmetric, it can be asymmetric at some sites. Strand merging loses this information. For most analyses, merging is appropriate. 5. **Cell-type heterogeneity**: Bulk WGBS from tissue captures methylation across ALL cell types. A CpG at 50% methylation could mean: (a) all cells are 50% methylated, or (b) half the cells are 0% and half are 100%. These are biologically different. Single-cell methylation data (if available) resolves this. 6. **CpG islands vs. open sea**: CpG-dense regions (islands) have very different methylation dynamics than CpG-sparse regions. Consider analyzing separately. 7. **Do NOT mix assemblies**: All files must be GRCh38 or all hg19. CpG positions are exact — even a 1bp offset from liftOver misaligns CpGs. 8. **X chromosome**: Males have one X (hemimethylation), females have two (one inactivated with different methylation). Handle sex chromosomes separately or filter them. 9. **Imprinted regions**: Some regions show ~50% methylation in all individuals due to genomic imprinting (one allele methylated, one not). These are normal, not noise. 10. **Do NOT use union logic for per-CpG methylation**: Unlike histone peaks where union is correct, methylation levels should be AVERAGED. Union logic only applies to the derived HMR/UMR/PMD regions. 11. **RRBS is NOT the same as WGBS**: Reduced Representation Bisulfite Sequencing (RRBS) covers only CpG-rich regions (~10% of CpGs). Do NOT mix RRBS and WGBS in per-CpG averaging — the CpG universe is different. 12. **Sequencing platform matters**: Liu et al. 2024 showed that NovaSeq and DNBSEQ-T7 give slightly different methylation estimates, especially in GC-rich regions. Note the platform in provenance if mixing experiments from different sequencers. ## Walkthrough: Cross-Tissue CpG Methylation Atlas for Imprinted Gene Regions **Goal**: Aggregate whole-genome bisulfite sequencing (WGBS) data across tissues to identify tissue-invariant vs. tissue-specific methylation patterns at imprinted gene loci. **Context**: Imprinted genes show parent-of-origin-specific methylation. Comparing across tissues reveals which imprinting control regions (ICRs) maintain methylation universally. ### Step 1: Find WGBS experiments across tissues ``` encode_search_experiments(assay_title="WGBS", organism="Homo sapiens", limit=50) ``` Expected output: ```json { "results": [ {"accession": "ENCSR765JPC", "assay_title": "WGBS", "organ": "liver", "biosample_summary": "liver tissue male adult (54 years)", "status": "released"}, {"accession": "ENCSR832HMR", "assay_title": "WGBS", "organ": "brain", "biosample_summary": "brain tissue female adult (53 years)", "status": "released"} ], "total": 147, "limit": 50, "offset": 0, "has_more": true, "next_offset": 50 } ``` **Interpretation**: 147 WGBS experiments available. Select tissues with ≥2 replicates for reliable per-CpG averaging. ### Step 2: List methylation bedGraph files ``` encode_list_files(experiment_accession="ENCSR765JPC", file_format="bed", output_type="methylation state at CpG", assembly="GRCh38") ``` Expected output (a JSON array of file records; fields abridged): ```json [ {"accession": "ENCFF123BED", "output_type": "methylation state at CpG", "file_format": "bed", "file_type": "bed bedMethyl", "assembly": "GRCh38", "file_size": 256901120, "file_size_human": "245.0 MB", "status": "released"} ] ``` ### Step 3: Download methylation files ``` encode_download_files(file_accessions=["ENCFF123BED", "ENCFF456MET", "ENCFF789CPG"], download_dir="/data/wgbs") ``` ### Step 4: Per-CpG weighted averaging across replicates For each tissue: 1. Merge replicates using weighted average: β = Σ(methylated reads) / Σ(total reads) 2. Filter CpGs with <10× combined coverage 3. Output: per-tissue methylation BED with columns: chr, start, end, β-value, coverage ### Step 5: Identify tissue-invariant ICRs ```bash # H19/IGF2 ICR: chr11:2,016,000-2,022,000 bedtools intersect -a merged_methylation.bed -b imprinted_icrs.bed -wa -wb | \ awk '{sum+=$4; n++} END {print sum/n}' ``` **Interpretation**: ICRs showing ~50% methylation across ALL tissues confirm maintained imprinting. Tissue-variable ICRs (range >20%) suggest tissue-specific imprinting loss. ### Integration with downstream skills - Feed differentially methylated regions into → **peak-annotation** for nearest gene assignment - Overlay with → **histone-aggregation** H3K4me3 to find promoter methylation–expression anticorrelation - Cross-reference CpG variants via → **clinvar-annotation** for methylation-disrupting mutations - Compare methylation at regulatory elements from → **regulatory-elements** ## Code Examples ### 1. Survey WGBS data availability by organ ``` encode_get_facets(assay_title="WGBS", organism="Homo sapiens") ``` Expected output: ```json { "biosample_ontology.organ_slims": [ {"term": "brain", "count": 32}, {"term": "liver", "count": 18}, {"term": "heart", "count": 12}, {"term": "lung", "count": 10}, {"term": "blood", "count": 8}, {"term": "kidney", "count": 6} ] } ``` ### 2. Check experiment quality before aggregation ``` encode_get_experiment(accession="ENCSR765JPC") ``` Expected output: ```json { "accession": "ENCSR765JPC", "assay_title": "WGBS", "biosample_summary": "liver tissue male adult (54 years)", "assembly": ["GRCh38"], "bio_replicate_count": 2, "status": "released", "audit_error_count": 0, "audit_warning_count": 0 } ``` ### 3. Track aggregated experiments ``` encode_track_experiment(accession="ENCSR765JPC", notes="Liver WGBS for cross-tissue methylation atlas") ``` Expected output: ```json { "tracking": {"accession": "ENCSR765JPC", "action": "tracked"}, "publications_found": 0, "publications": [], "pipelines_found": 1, "pipelines": [ {"title": "WGBS paired-end pipeline", "version": "1.1.6", "software": [{"name": "bismark", "version": "0.22.3"}], "status": "released"} ] } ``` ## Integration | This skill produces... | Feed into... | Purpose | |---|---|---| | Per-CpG β-value matrix | **regulatory-elements** | Identify methylation at cCREs and enhancers | | Differentially methylated regions (DMRs) | **peak-annotation** | Assign DMRs to nearest genes | | Tissue-specific hypomethylated regions | **histone-aggregation** | Correlate with H3K4me3 active promoter marks | | Methylation at CpG islands | **variant-annotation** | Find variants disrupting CpG sites | | HMR/UMR/PMD boundaries | **accessibility-aggregation** | Overlay open chromatin at unmethylated regions | | Cross-tissue methylation atlas | **visualization-workflow** | Generate methylation heatmaps across tissues | | CpG methylation at GWAS loci | **gwas-catalog** | Annotate trait-associated variants with methylation context | ## Related Skills - **histone-aggregation**: HMRs + H3K27ac union peaks identify active enhancers; HMRs + H3K4me3 identify active promoters - **accessibility-aggregation**: HMRs typically overlap open chromatin; concordance between HMRs and ATAC/DNase peaks validates both - **hic-aggregation**: Hypomethylated enhancers often anchor chromatin loops to target genes - **regulatory-elements**: Combine methylation with histone and accessibility data to classify regulatory element types - **epigenome-profiling**: Methylation adds a critical layer to chromatin state annotation - **pipeline-wgbs**: Process raw WGBS data through the full ENCODE-aligned pipeline - **batch-analysis**: Batch processing workflows for systematic methylation aggregation - **publication-trust**: Verify literature claims backing analytical decisions ## Presenting Results - Present methylation summary as: total CpGs analyzed | mean coverage | methylation distribution (UMR/LMR/PMD). Show per-sample contribution. Suggest: "Would you like to correlate with histone marks?" ## For the request: "$ARGUMENTS"
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.