{
  "skill": "alterlab-matchms",
  "evals": [
    {
      "id": "library-search-annotation",
      "prompt": "I have a folder of unknown MS/MS spectra in MGF and a reference spectral library in MSP. Can you score them against the library with modified cosine and give me the top matches per query to annotate the metabolites?",
      "expected_output": "Invokes alterlab-matchms. Loads queries with load_from_mgf and the library with load_from_msp, computes similarity with calculate_scores using ModifiedCosineGreedy (which accounts for precursor m/z differences), and returns the best matches per query via scores_by_query(name=\"ModifiedCosineGreedy_score\", sort=True) for compound annotation.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "calculate_scores" },
        { "type": "behavior", "value": "Uses calculate_scores with ModifiedCosineGreedy (or CosineGreedy) against a reference library and ranks best matches per query; does not use the removed name 'ModifiedCosine'." }
      ]
    },
    {
      "id": "spectrum-preprocessing-pipeline",
      "prompt": "Before I compare my mass spectra I need to clean them: harmonize the metadata, normalize peak intensities, drop low-intensity noise peaks, and require at least 5 peaks. How do I build a reproducible processing pipeline?",
      "expected_output": "Invokes alterlab-matchms. Builds a filtering pipeline (e.g. SpectrumProcessor) chaining default_filters for metadata harmonization, normalize_intensities, select_by_relative_intensity to remove low-intensity peaks, and require_minimum_number_of_peaks with n_required=5. Runs it with processor.process_spectra(...) / process_spectrum(...) (a SpectrumProcessor instance is not callable). Frames it as a reproducible matchms preprocessing workflow.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "behavior", "value": "Chains matchms filters (default_filters, normalize_intensities, select_by_relative_intensity, require_minimum_number_of_peaks) into a SpectrumProcessor and executes it via process_spectra/process_spectrum (not by calling the processor object directly)." }
      ]
    },
    {
      "id": "cosine-similarity-two-spectra",
      "prompt": "I have two MS/MS spectra and I just want a cosine similarity score between them with a 0.1 Da peak matching tolerance. What's the matchms way to do this?",
      "expected_output": "Invokes alterlab-matchms. Constructs Spectrum objects (or loads them), then computes peak-based cosine similarity with CosineGreedy (or CosineHungarian) via calculate_scores, using the specified m/z matching tolerance. Explains the greedy vs Hungarian matching tradeoff.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "cosine" }
      ]
    },
    {
      "id": "derive-structure-fingerprint",
      "prompt": "My spectra metadata only has SMILES. Can you derive the InChI and InChIKey for each spectrum and attach a Morgan fingerprint so I can compare structures alongside the spectral matches?",
      "expected_output": "Invokes alterlab-matchms. Uses matchms metadata/annotation filters: derive_inchi_from_smiles, derive_inchikey_from_inchi, and add_fingerprint with a valid fingerprint_type such as 'morgan2' (not the invalid 'morgan'). Notes rdkit is required (it is a base matchms dependency, not a separate extra) and that this enables FingerprintSimilarity-based structural comparison.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "behavior", "value": "Uses derive_inchi_from_smiles / derive_inchikey_from_inchi and add_fingerprint with a valid type ('morgan2'/'morgan1'/'morgan3'/'daylight', never bare 'morgan') to enrich spectrum metadata." }
      ]
    },
    {
      "id": "near-miss-pyopenms",
      "prompt": "I need to run a full bottom-up proteomics pipeline on my raw LC-MS/MS files: peptide feature detection, database search for protein identification, and FDR control. Can you set that up?",
      "expected_output": "Does NOT invoke this skill; defers to pyopenms. matchms scores mass-spectral similarity and does library-based small-molecule/metabolite annotation, not full LC-MS/MS proteomics pipelines with peptide feature detection, protein database search, and FDR control.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "pyopenms" }
      ]
    },
    {
      "id": "near-miss-hmdb",
      "prompt": "Pull the HMDB record for caffeine (HMDB0001847): its normal blood concentrations, associated pathways, and the reference MS/MS spectra HMDB lists for it.",
      "expected_output": "Does NOT invoke alterlab-matchms; defers to alterlab-hmdb. The user wants a database lookup of a known metabolite's HMDB record (concentrations, pathways, catalogued reference spectra), not local spectral preprocessing or similarity scoring with matchms.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-hmdb" }
      ]
    }
  ]
}

# Matchms Filtering Functions Reference

This document provides a comprehensive reference of all filtering functions available in matchms for processing mass spectrometry data.

## Metadata Processing Filters

### Compound & Chemical Information

**add_compound_name(spectrum)**
- Adds compound name to the correct metadata field
- Standardizes compound name storage location

**clean_compound_name(spectrum)**
- Removes frequently seen unwanted additions from compound names
- Cleans up formatting inconsistencies

**derive_adduct_from_name(spectrum)**
- Extracts adduct information from compound names
- Moves adduct notation to proper metadata field

**derive_formula_from_name(spectrum)**
- Detects chemical formulas in compound names
- Relocates formulas to appropriate metadata field

**derive_annotation_from_compound_name(spectrum)**
- Retrieves SMILES/InChI from PubChem using compound name
- Automatically annotates chemical structures

### Chemical Structure Conversions

**derive_inchi_from_smiles(spectrum)**
- Generates InChI from SMILES strings
- Requires rdkit library

**derive_inchikey_from_inchi(spectrum)**
- Computes InChIKey from InChI
- 27-character hashed identifier

**derive_smiles_from_inchi(spectrum)**
- Creates SMILES from InChI representation
- Requires rdkit library

**repair_inchi_inchikey_smiles(spectrum)**
- Corrects misplaced chemical identifiers
- Fixes metadata field confusion

**repair_not_matching_annotation(spectrum)**
- Ensures consistency between SMILES, InChI, and InChIKey
- Validates chemical structure annotations match

**add_fingerprint(spectrum, fingerprint_type="daylight", nbits=2048)**
- Generates molecular fingerprints for similarity calculations
- `fingerprint_type` is one of: `"daylight"`, `"morgan1"`, `"morgan2"`, `"morgan3"` (the digit is the Morgan radius; there is no separate `radius=` argument and `"morgan"` alone is invalid)
- Used with FingerprintSimilarity scoring; requires rdkit

### Mass & Charge Information

**add_precursor_mz(spectrum)**
- Normalizes precursor m/z values
- Standardizes precursor mass metadata

**add_parent_mass(spectrum, estimate_from_adduct=True)**
- Calculates neutral parent mass from precursor m/z and adduct
- Can estimate from adduct if not directly available

**correct_charge(spectrum)**
- Aligns charge values with ionmode
- Ensures charge sign matches ionization mode

**make_charge_int(spectrum)**
- Converts charge to integer format
- Standardizes charge representation

**clean_adduct(spectrum)**
- Standardizes adduct notation
- Corrects common adduct formatting issues

**interpret_pepmass(spectrum)**
- Parses pepmass field into component values
- Extracts precursor m/z and intensity from combined field

### Ion Mode & Validation

**derive_ionmode(spectrum)**
- Determines ionmode from adduct information
- Infers positive/negative mode from adduct type

**require_correct_ionmode(spectrum, ion_mode)**
- Filters spectra by specified ionmode
- Returns None if ionmode doesn't match
- Use: `spectrum = require_correct_ionmode(spectrum, "positive")`

**require_precursor_mz(spectrum, minimum_accepted_mz=0.0)**
- Validates precursor m/z presence and value
- Returns None if missing or below threshold

**require_precursor_below_mz(spectrum, maximum_accepted_mz=1000.0)**
- Enforces maximum precursor m/z limit
- Returns None if precursor exceeds threshold

### Retention Information

**add_retention_time(spectrum)**
- Harmonizes retention time as float values
- Standardizes RT metadata field

**add_retention_index(spectrum)**
- Stores retention index in standardized field
- Normalizes RI metadata

### Data Harmonization

**harmonize_undefined_inchi(spectrum, undefined="", aliases=None)**
- Standardizes undefined/empty InChI entries
- Replaces various "unknown" representations with consistent value

**harmonize_undefined_inchikey(spectrum, undefined="", aliases=None)**
- Standardizes undefined/empty InChIKey entries
- Unifies missing data representation

**harmonize_undefined_smiles(spectrum, undefined="", aliases=None)**
- Standardizes undefined/empty SMILES entries
- Consistent handling of missing structural data

### Repair & Quality Functions

**repair_adduct_based_on_parent_mass(spectrum, mass_tolerance)**
- Corrects adduct so it is consistent with the parent mass
- `mass_tolerance` is required (no default)
- Related: `repair_adduct_and_parent_mass_based_on_smiles(spectrum, mass_tolerance)`

**repair_parent_mass_is_molar_mass(spectrum, mass_tolerance)**
- Converts an average molar mass mistakenly stored as parent mass to the monoisotopic mass
- `mass_tolerance` is required (no default)

**repair_parent_mass_from_smiles(spectrum, mass_tolerance=0.1)**
- Recomputes parent mass from the SMILES structure

**repair_smiles_of_salts(spectrum, mass_tolerance)**
- Removes salt components to match parent mass
- Extracts relevant molecular fragment; `mass_tolerance` is required (no default)

**require_parent_mass_match_smiles(spectrum, mass_tolerance)**
- Validates parent mass against SMILES-calculated mass
- Returns None if masses don't match within tolerance; `mass_tolerance` is required

**require_valid_annotation(spectrum)**
- Ensures complete, consistent chemical annotations
- Validates SMILES, InChI, and InChIKey presence and consistency

## Peak Processing Filters

### Normalization & Selection

**normalize_intensities(spectrum)**
- Scales peak intensities to unit height (max = 1.0)
- Essential preprocessing step for similarity calculations

**select_by_intensity(spectrum, intensity_from=0.0, intensity_to=1.0)**
- Retains peaks within specified absolute intensity range
- Filters by raw intensity values

**select_by_relative_intensity(spectrum, intensity_from=0.0, intensity_to=1.0)**
- Keeps peaks within relative intensity bounds
- Filters as fraction of maximum intensity

**select_by_mz(spectrum, mz_from=0.0, mz_to=1000.0)**
- Filters peaks by m/z value range
- Removes peaks outside specified m/z window

### Peak Reduction & Filtering

**reduce_to_number_of_peaks(spectrum, n_max=None, ratio_desired=None)**
- Removes lowest-intensity peaks when exceeding maximum
- Can specify absolute number or ratio
- Use: `spectrum = reduce_to_number_of_peaks(spectrum, n_max=100)`

**remove_peaks_around_precursor_mz(spectrum, mz_tolerance=17)**
- Eliminates peaks within tolerance of precursor
- Removes precursor and isotope peaks
- Common preprocessing for fragment-based similarity

**remove_peaks_outside_top_k(spectrum, k=10, ratio_desired=None)**
- Retains only peaks near k highest-intensity peaks
- Focuses on most informative signals

**require_minimum_number_of_peaks(spectrum, n_required=10)**
- Discards spectra with insufficient peaks
- Quality control filter
- Returns None if peak count below threshold

**require_minimum_number_of_high_peaks(spectrum, no_peaks=5, intensity_percent=2.0)**
- Removes spectra lacking high-intensity peaks (`intensity_percent` is a relative intensity in percent, 0-100)
- Ensures data quality
- Returns None if insufficient peaks above threshold

### Loss Calculation

**Spectrum.compute_losses(loss_mz_from=0.0, loss_mz_to=None)** (method, not a filter)
- Derives neutral losses (loss = precursor_mz - fragment_mz) and returns a `Fragments` object; also exposed via the `Spectrum.losses` property
- The old `matchms.filtering.add_losses` filter no longer exists — `NeutralLossesCosine` computes losses internally, so you rarely need to call this yourself

## Pipeline Functions

**default_filters(spectrum)**
- Applies nine essential metadata filters sequentially (matchms 0.33.x):
  1. make_charge_int
  2. add_compound_name
  3. derive_adduct_from_name
  4. derive_formula_from_name
  5. clean_compound_name
  6. interpret_pepmass
  7. add_precursor_mz
  8. derive_ionmode
  9. correct_charge
- Recommended starting point for metadata harmonization
- Note: retention-time / structure-harmonization filters (add_retention_time, harmonize_undefined_smiles, etc.) are NOT part of default_filters — add them explicitly if needed

**SpectrumProcessor(filters)**
- Orchestrates multi-filter pipelines
- Accepts an iterable of filters; each item may be a filter name (str), a callable, or a `("filter_name", {kwargs})` tuple. The string/tuple form is preferred because `processor.processing_steps` then reports the exact parameters used (reproducibility).
- A `SpectrumProcessor` instance is **not callable**. Run it with `processor.process_spectrum(spectrum)` (single) or `processor.process_spectra(spectra)` (list).
- `process_spectra` returns a **`(processed_spectra, report)` tuple** — unpack it.
- Example:
```python
from matchms import SpectrumProcessor
from matchms.filtering import default_filters
processor = SpectrumProcessor([
    default_filters,  # composite — pass the callable; "default_filters" is not a registered string name
    "normalize_intensities",
    ("select_by_relative_intensity", {"intensity_from": 0.01}),
])
processed = processor.process_spectrum(spectrum)         # one spectrum
processed_list, report = processor.process_spectra(spectra)  # many
```

## Common Filter Combinations

### Standard Preprocessing Pipeline
```python
from matchms.filtering import (default_filters, normalize_intensities,
                               select_by_relative_intensity,
                               require_minimum_number_of_peaks)

spectrum = default_filters(spectrum)
spectrum = normalize_intensities(spectrum)
spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
spectrum = require_minimum_number_of_peaks(spectrum, n_required=5)
```

### Quality Control Pipeline
```python
from matchms.filtering import (require_precursor_mz, require_minimum_number_of_peaks,
                               require_minimum_number_of_high_peaks)

spectrum = require_precursor_mz(spectrum, minimum_accepted_mz=50.0)
if spectrum is None:
    # Spectrum failed quality control
    pass
spectrum = require_minimum_number_of_peaks(spectrum, n_required=10)
spectrum = require_minimum_number_of_high_peaks(spectrum, no_peaks=5)
```

### Chemical Annotation Pipeline
```python
from matchms.filtering import (derive_inchi_from_smiles, derive_inchikey_from_inchi,
                               add_fingerprint, require_valid_annotation)

spectrum = derive_inchi_from_smiles(spectrum)
spectrum = derive_inchikey_from_inchi(spectrum)
spectrum = add_fingerprint(spectrum, fingerprint_type="morgan2", nbits=2048)
spectrum = require_valid_annotation(spectrum)
```

### Peak Cleaning Pipeline
```python
from matchms.filtering import (normalize_intensities, remove_peaks_around_precursor_mz,
                               select_by_relative_intensity, reduce_to_number_of_peaks)

spectrum = normalize_intensities(spectrum)
spectrum = remove_peaks_around_precursor_mz(spectrum, mz_tolerance=17)
spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
spectrum = reduce_to_number_of_peaks(spectrum, n_max=200)
```

## Notes on Filter Usage

1. **Order matters**: Apply filters in logical sequence (e.g., normalize before relative intensity selection)
2. **Filters return None**: Many filters return None for invalid spectra; check for None before proceeding
3. **Immutability**: Filters typically return modified copies; reassign results to variables
4. **Pipeline efficiency**: Use SpectrumProcessor for consistent multi-spectrum processing
5. **Documentation**: For detailed parameters, see matchms.readthedocs.io/en/latest/api/matchms.filtering.html

# Matchms Importing and Exporting Reference

This document details all file format support in matchms for loading and saving mass spectrometry data.

## Importing Spectra

Matchms provides dedicated functions for loading spectra from various file formats. All import functions return generators for memory-efficient processing of large files.

### Common Import Pattern

```python
from matchms.importing import load_from_mgf

# Load spectra (returns generator)
spectra_generator = load_from_mgf("spectra.mgf")

# Convert to list for processing
spectra = list(spectra_generator)
```

## Supported Import Formats

### MGF (Mascot Generic Format)

**Function**: `load_from_mgf(filename, metadata_harmonization=True)`

**Description**: Loads spectra from MGF files, a common format for mass spectrometry data exchange.

**Parameters**:
- `filename` (str): Path to MGF file
- `metadata_harmonization` (bool, default=True): Apply automatic metadata key harmonization

**Example**:
```python
from matchms.importing import load_from_mgf

# Load with metadata harmonization
spectra = list(load_from_mgf("data.mgf"))

# Load without harmonization
spectra = list(load_from_mgf("data.mgf", metadata_harmonization=False))
```

**MGF Format**: Text-based format with BEGIN IONS/END IONS blocks containing metadata and peak lists.

---

### MSP (NIST Mass Spectral Library Format)

**Function**: `load_from_msp(filename, metadata_harmonization=True)`

**Description**: Loads spectra from MSP files, commonly used for spectral libraries.

**Parameters**:
- `filename` (str): Path to MSP file
- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization

**Example**:
```python
from matchms.importing import load_from_msp

spectra = list(load_from_msp("library.msp"))
```

**MSP Format**: Text-based format with Name/MW/Comment fields followed by peak lists.

---

### mzML (Mass Spectrometry Markup Language)

**Function**: `load_from_mzml(filename, ms_level=2, metadata_harmonization=True)`

**Description**: Loads spectra from mzML files, the standard XML-based format for raw mass spectrometry data.

**Parameters**:
- `filename` (str): Path to mzML file
- `ms_level` (int, default=2): MS level to extract (1 for MS1, 2 for MS2/tandem)
- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization

**Example**:
```python
from matchms.importing import load_from_mzml

# Load MS2 spectra (default)
ms2_spectra = list(load_from_mzml("data.mzML"))

# Load MS1 spectra
ms1_spectra = list(load_from_mzml("data.mzML", ms_level=1))
```

**mzML Format**: XML-based standard format containing raw instrument data and rich metadata.

---

### mzXML

**Function**: `load_from_mzxml(filename, ms_level=2, metadata_harmonization=True)`

**Description**: Loads spectra from mzXML files, an earlier XML-based format for mass spectrometry data.

**Parameters**:
- `filename` (str): Path to mzXML file
- `ms_level` (int, default=2): MS level to extract
- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization

**Example**:
```python
from matchms.importing import load_from_mzxml

spectra = list(load_from_mzxml("data.mzXML"))
```

**mzXML Format**: XML-based format, predecessor to mzML.

---

### JSON (GNPS Format)

**Function**: `load_from_json(filename, metadata_harmonization=True)`

**Description**: Loads spectra from JSON files, particularly GNPS-compatible JSON format.

**Parameters**:
- `filename` (str): Path to JSON file
- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization

**Example**:
```python
from matchms.importing import load_from_json

spectra = list(load_from_json("spectra.json"))
```

**JSON Format**: Structured JSON with spectrum metadata and peak arrays.

---

### Pickle (Python Serialization)

**Function**: `load_from_pickle(filename, metadata_harmonization)`

**Description**: Loads previously saved matchms Spectrum objects from pickle files. Fast loading of preprocessed spectra.

**Parameters**:
- `filename` (str): Path to pickle file
- `metadata_harmonization` (bool): Whether to re-harmonize metadata keys on load. This argument is **required** (no default).

**Example**:
```python
from matchms.importing import load_from_pickle

spectra = list(load_from_pickle("processed_spectra.pkl", metadata_harmonization=False))
```

**Use case**: Saving and loading preprocessed spectra for faster subsequent analyses.

**Note**: There is no `matchms.exporting.save_as_pickle`. Pickle files are written with Python's own `pickle.dump(spectra, file)` (see the Pickle Export section).

---

### USI (Universal Spectrum Identifier)

**Function**: `load_from_usi(usi)`

**Description**: Loads a single spectrum from a metabolomics USI reference.

**Parameters**:
- `usi` (str): Universal Spectrum Identifier string

**Example**:
```python
from matchms.importing import load_from_usi

usi = "mzspec:GNPS:TASK-...:spectrum..."
spectrum = load_from_usi(usi)
```

**USI Format**: Standardized identifier for accessing spectra from online repositories.

---

## Exporting Spectra

Matchms provides functions to save processed spectra to various formats for sharing and archival.

> Heads up — the append/mode argument differs per function and is **not** `write_mode`: it is `file_mode` for MGF, `mode` for MSP, and JSON has no mode argument at all. For MGF/MSP the default is `"a"` (append), so writing repeatedly to the same path concatenates — delete the file first or pass `"w"` to overwrite.

### MGF Export

**Function**: `save_as_mgf(spectra, filename, file_mode="a")`

**Description**: Saves spectra to MGF format.

**Parameters**:
- `spectra` (list): List of Spectrum objects to save
- `filename` (str): Output file path
- `file_mode` (str, default="a"): `"a"` to append, `"w"` to overwrite

**Example**:
```python
from matchms.exporting import save_as_mgf

save_as_mgf(processed_spectra, "output.mgf", file_mode="w")  # overwrite
```

---

### MSP Export

**Function**: `save_as_msp(spectra, filename, mode="a")`

**Description**: Saves spectra to MSP format.

**Parameters**:
- `spectra` (list): List of Spectrum objects to save
- `filename` (str): Output file path (must end in `.msp`)
- `mode` (str, default="a"): `"a"` to append, `"w"` to overwrite

**Example**:
```python
from matchms.exporting import save_as_msp

save_as_msp(library_spectra, "library.msp", mode="w")
```

---

### JSON Export

**Function**: `save_as_json(spectra, filename, export_style="matchms")`

**Description**: Saves spectra to JSON format. Always overwrites (`filename` is opened in `"w"` mode); there is no append option.

**Parameters**:
- `spectra` (list): List of Spectrum objects to save
- `filename` (str): Output file path
- `export_style` (str, default="matchms"): metadata key style — e.g. `"matchms"` or `"gnps"`

**Example**:
```python
from matchms.exporting import save_as_json

save_as_json(spectra, "spectra.json")
```

---

### Pickle Export

**There is no `save_as_pickle` in matchms.** Write pickle files with Python's standard `pickle` module; read them back with `matchms.importing.load_from_pickle`.

**Description**: Saves spectra as a Python pickle file. Preserves all Spectrum attributes and is fastest for loading.

**Example**:
```python
import pickle

with open("processed.pkl", "wb") as f:
    pickle.dump(processed_spectra, f)

# Reload:
from matchms.importing import load_from_pickle
spectra = list(load_from_pickle("processed.pkl", metadata_harmonization=False))
```

> For the supported text/standard formats use the dedicated `save_as_mgf` / `save_as_msp` / `save_as_json` functions (or the generic `save_spectra(spectra, "out.mgf")`, which dispatches by file extension and does NOT accept `.pkl`).

**Advantages**:
- Fast save and load
- Preserves exact Spectrum state
- No format conversion overhead

**Disadvantages**:
- Not human-readable
- Python-specific (not portable to other languages)
- Pickle format may not be compatible across Python versions

---

## Complete Import/Export Workflow

### Preprocessing and Saving Pipeline

```python
import pickle
from matchms.importing import load_from_mgf
from matchms.exporting import save_as_mgf
from matchms.filtering import default_filters, normalize_intensities
from matchms.filtering import select_by_relative_intensity

# Load raw spectra
spectra = list(load_from_mgf("raw_data.mgf"))

# Process spectra (filters can return None for invalid spectra — bail early)
processed = []
for spectrum in spectra:
    spectrum = default_filters(spectrum)
    spectrum = normalize_intensities(spectrum)
    spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
    if spectrum is not None:
        processed.append(spectrum)

# Save processed spectra (MGF for sharing)
save_as_mgf(processed, "processed_data.mgf")

# Save as pickle for fast reloading (no save_as_pickle helper — use stdlib pickle)
with open("processed_data.pkl", "wb") as f:
    pickle.dump(processed, f)
```

### Format Conversion

```python
from matchms.importing import load_from_mzml
from matchms.exporting import save_as_mgf, save_as_msp

# Convert mzML to MGF
spectra = list(load_from_mzml("data.mzML", ms_level=2))
save_as_mgf(spectra, "data.mgf")

# Convert to MSP library format
save_as_msp(spectra, "data.msp")
```

### Loading from Multiple Files

```python
from matchms.importing import load_from_mgf
import glob

# Load all MGF files in directory
all_spectra = []
for mgf_file in glob.glob("data/*.mgf"):
    spectra = list(load_from_mgf(mgf_file))
    all_spectra.extend(spectra)

print(f"Loaded {len(all_spectra)} spectra from multiple files")
```

### Memory-Efficient Processing

```python
from matchms.importing import load_from_mgf
from matchms.exporting import save_as_mgf
from matchms.filtering import default_filters, normalize_intensities

# Process large file without loading all into memory
import os

def process_spectrum(spectrum):
    spectrum = default_filters(spectrum)
    spectrum = normalize_intensities(spectrum)
    return spectrum

# Stream processing. save_as_mgf takes a FILENAME (not an open handle) and
# defaults to file_mode="a" (append), so each call appends one spectrum. Start
# from a clean file so a previous run's output is not concatenated.
out_path = "output.mgf"
if os.path.exists(out_path):
    os.remove(out_path)

for spectrum in load_from_mgf("large_file.mgf"):
    processed = process_spectrum(spectrum)
    if processed is not None:
        save_as_mgf([processed], out_path, file_mode="a")
```

## Format Selection Guidelines

**MGF**:
- ✓ Widely supported
- ✓ Human-readable
- ✓ Good for data sharing
- ✓ Moderate file size
- Best for: Data exchange, GNPS uploads, publication data

**MSP**:
- ✓ Spectral library standard
- ✓ Human-readable
- ✓ Good metadata support
- Best for: Reference libraries, NIST format compatibility

**JSON**:
- ✓ Structured format
- ✓ GNPS compatible
- ✓ Easy to parse programmatically
- Best for: Web applications, GNPS integration, structured data

**Pickle**:
- ✓ Fastest save/load
- ✓ Preserves exact state
- ✗ Not portable to other languages
- ✗ Not human-readable
- Best for: Intermediate processing, Python-only workflows

**mzML/mzXML**:
- ✓ Raw instrument data
- ✓ Rich metadata
- ✓ Industry standard
- ✗ Large file size
- ✗ Slower to parse
- Best for: Raw data archival, multi-level MS data

## Metadata Harmonization

The `metadata_harmonization` parameter (available in most import functions) automatically standardizes metadata keys:

```python
# Without harmonization
spectrum = load_from_mgf("data.mgf", metadata_harmonization=False)
# May have: "PRECURSOR_MZ", "Precursor_mz", "precursormz"

# With harmonization (default)
spectrum = load_from_mgf("data.mgf", metadata_harmonization=True)
# Standardized to: "precursor_mz"
```

**Recommended**: Keep harmonization enabled (default) for consistent metadata access across different data sources.

## File Format Specifications

For detailed format specifications:
- **MGF**: http://www.matrixscience.com/help/data_file_help.html
- **MSP**: https://chemdata.nist.gov/mass-spc/ms-search/
- **mzML**: http://www.psidev.info/mzML
- **GNPS JSON**: https://gnps.ucsd.edu/

## Further Reading

For complete API documentation:
https://matchms.readthedocs.io/en/latest/api/matchms.importing.html
https://matchms.readthedocs.io/en/latest/api/matchms.exporting.html

# Matchms Similarity Functions Reference

This document provides detailed information about all similarity scoring methods available in matchms.

## Overview

Matchms provides multiple similarity functions for comparing mass spectra. Use `calculate_scores()` to compute pairwise similarities between reference and query spectra collections.

```python
from matchms import calculate_scores
from matchms.similarity import CosineGreedy

scores = calculate_scores(references=library_spectra,
                         queries=query_spectra,
                         similarity_function=CosineGreedy())
```

## Peak-Based Similarity Functions

These functions compare mass spectra based on their peak patterns (m/z and intensity values).

### CosineGreedy

**Description**: Calculates cosine similarity between two spectra using a fast greedy matching algorithm. Peaks are matched within a specified tolerance, and similarity is computed based on matched peak intensities.

**When to use**:
- Fast similarity calculations for large datasets
- General-purpose spectral matching
- When speed is prioritized over mathematically optimal matching

**Parameters**:
- `tolerance` (float, default=0.1): Maximum m/z difference for peak matching (Daltons)
- `mz_power` (float, default=0.0): Exponent for m/z weighting (0 = no weighting)
- `intensity_power` (float, default=1.0): Exponent for intensity weighting

**Example**:
```python
from matchms.similarity import CosineGreedy

similarity_func = CosineGreedy(tolerance=0.1, mz_power=0.0, intensity_power=1.0)
scores = calculate_scores(references, queries, similarity_func)
```

**Output**: Similarity score between 0.0 and 1.0, plus number of matched peaks.

---

### CosineHungarian

**Description**: Calculates cosine similarity using the Hungarian algorithm for optimal peak matching. Provides mathematically optimal peak assignments but is slower than CosineGreedy.

**When to use**:
- When optimal peak matching is required
- High-quality reference library comparisons
- Research requiring reproducible, mathematically rigorous results

**Parameters**:
- `tolerance` (float, default=0.1): Maximum m/z difference for peak matching
- `mz_power` (float, default=0.0): Exponent for m/z weighting
- `intensity_power` (float, default=1.0): Exponent for intensity weighting

**Example**:
```python
from matchms.similarity import CosineHungarian

similarity_func = CosineHungarian(tolerance=0.1)
scores = calculate_scores(references, queries, similarity_func)
```

**Output**: Optimal similarity score between 0.0 and 1.0, plus matched peaks.

**Note**: Slower than CosineGreedy; use for smaller datasets or when accuracy is critical.

---

### ModifiedCosineGreedy

> **Rename:** this class was previously `ModifiedCosine`. In current matchms it is `ModifiedCosineGreedy` (with a `ModifiedCosineHungarian` variant for optimal peak assignment). Importing `ModifiedCosine` raises `ImportError`.

**Description**: Extends cosine similarity by accounting for precursor m/z differences. Allows peaks to match after applying a mass shift based on the difference between precursor masses. Useful for comparing spectra of related compounds (isotopes, adducts, analogs).

**When to use**:
- Comparing spectra from different precursor masses
- Identifying structural analogs or derivatives
- Cross-ionization mode comparisons
- When precursor mass differences are meaningful

**Parameters**:
- `tolerance` (float, default=0.1): Maximum m/z difference for peak matching after shift
- `mz_power` (float, default=0.0): Exponent for m/z weighting
- `intensity_power` (float, default=1.0): Exponent for intensity weighting

**Example**:
```python
from matchms.similarity import ModifiedCosineGreedy

similarity_func = ModifiedCosineGreedy(tolerance=0.1)
scores = calculate_scores(references, queries, similarity_func)
```

**Requirements**: Both spectra must have valid precursor_mz metadata.

---

### NeutralLossesCosine

**Description**: Calculates similarity based on neutral loss patterns rather than fragment m/z values. Neutral losses are derived by subtracting fragment m/z from precursor m/z. Particularly useful for identifying compounds with similar fragmentation patterns.

**When to use**:
- Comparing fragmentation patterns across different precursor masses
- Identifying compounds with similar neutral loss profiles
- Complementary to regular cosine scoring
- Metabolite identification and classification

**Parameters**:
- `tolerance` (float, default=0.1): Maximum neutral loss difference for matching
- `mz_power` (float, default=0.0): Exponent for loss value weighting
- `intensity_power` (float, default=1.0): Exponent for intensity weighting

**Example**:
```python
from matchms.similarity import NeutralLossesCosine

# NeutralLossesCosine computes neutral losses internally from precursor_mz —
# there is no separate add_losses() filter (it was removed). Just score directly.
similarity_func = NeutralLossesCosine(tolerance=0.1)
scores = calculate_scores(references, queries, similarity_func)
```

**Requirements**:
- Both spectra must have valid precursor_mz metadata (losses are derived from it)
- To inspect/store losses on a spectrum, use the `Spectrum.compute_losses(loss_mz_from=0.0, loss_mz_to=None)` method / `Spectrum.losses` property — the old `matchms.filtering.add_losses` filter no longer exists

---

## Structural Similarity Functions

These functions compare molecular structures rather than spectral peaks.

### FingerprintSimilarity

**Description**: Calculates similarity between molecular fingerprints derived from chemical structures (SMILES or InChI). Supports multiple fingerprint types and similarity metrics.

**When to use**:
- Structural similarity without spectral data
- Combining structural and spectral similarity
- Pre-filtering candidates before spectral matching
- Structure-activity relationship studies

**Parameters**:
- `similarity_measure` (str, default="jaccard"): Similarity metric — one of `"jaccard"`, `"dice"`, `"cosine"` (other values raise an assertion)
  - `"jaccard"`: Jaccard index (intersection / union)
  - `"dice"`: Dice coefficient (2 * intersection / (size1 + size2))
  - `"cosine"`: Cosine similarity
- `set_empty_scores` (float/int/str, default="nan"): value returned when a fingerprint is missing

The **fingerprint type** is NOT a parameter of FingerprintSimilarity — it is chosen earlier when you call `add_fingerprint(spectrum, fingerprint_type="morgan2", ...)`. FingerprintSimilarity just compares whatever fingerprints are already attached.

**Example**:
```python
from matchms.similarity import FingerprintSimilarity
from matchms.filtering import add_fingerprint

# Add fingerprints to spectra
spectra_with_fps = [add_fingerprint(s, fingerprint_type="morgan2", nbits=2048)
                    for s in spectra]

similarity_func = FingerprintSimilarity(similarity_measure="jaccard")
scores = calculate_scores(references_with_fps, queries_with_fps, similarity_func)
```

**Requirements**:
- Spectra must have valid SMILES or InChI metadata
- Use `add_fingerprint()` filter to compute fingerprints
- Requires rdkit library

---

## Metadata-Based Similarity Functions

These functions compare metadata fields rather than spectral or structural data.

### MetadataMatch

**Description**: Compares user-defined metadata fields between spectra. Supports exact matching for categorical data and tolerance-based matching for numerical data.

**When to use**:
- Filtering by experimental conditions (collision energy, retention time)
- Instrument-specific matching
- Combining metadata constraints with spectral similarity
- Custom metadata-based filtering

**Parameters**:
- `field` (str): Metadata field name to compare
- `matching_type` (str, default="equal_match"): Matching method — only `"equal_match"` or `"difference"` are accepted (other values raise an assertion)
  - `"equal_match"`: Entries must be exactly equal (works for strings or numbers)
  - `"difference"`: Numerical match if `abs(a - b) <= tolerance`
- `tolerance` (float, default=0.1): Maximum difference for `"difference"` matching

**Example (Exact / equality matching)**:
```python
from matchms.similarity import MetadataMatch

# Match by instrument type
similarity_func = MetadataMatch(field="instrument_type", matching_type="equal_match")
scores = calculate_scores(references, queries, similarity_func)
```

**Example (Numerical matching)**:
```python
# Match retention time within 0.5 minutes
similarity_func = MetadataMatch(field="retention_time",
                                matching_type="difference",
                                tolerance=0.5)
scores = calculate_scores(references, queries, similarity_func)
```

**Output**: Returns 1.0 (match) or 0.0 (no match) for exact matching. For numerical matching, returns similarity score based on difference.

---

### PrecursorMzMatch

**Description**: Binary matching based on precursor m/z values. Returns True/False based on whether precursor masses match within specified tolerance.

**When to use**:
- Pre-filtering spectral libraries by precursor mass
- Fast mass-based candidate selection
- Combining with other similarity metrics
- Isobaric compound identification

**Parameters**:
- `tolerance` (float, default=0.1): Maximum m/z difference for matching
- `tolerance_type` (str, default="Dalton"): Tolerance unit
  - `"Dalton"`: Absolute mass difference
  - `"ppm"`: Parts per million (relative)

**Example**:
```python
from matchms.similarity import PrecursorMzMatch

# Match precursor within 0.1 Da
similarity_func = PrecursorMzMatch(tolerance=0.1, tolerance_type="Dalton")
scores = calculate_scores(references, queries, similarity_func)

# Match precursor within 10 ppm
similarity_func = PrecursorMzMatch(tolerance=10, tolerance_type="ppm")
scores = calculate_scores(references, queries, similarity_func)
```

**Output**: 1.0 (match) or 0.0 (no match)

**Requirements**: Both spectra must have valid precursor_mz metadata.

---

### ParentMassMatch

**Description**: Binary matching based on parent mass (neutral mass) values. Similar to PrecursorMzMatch but uses calculated parent mass instead of precursor m/z.

**When to use**:
- Comparing spectra from different ionization modes
- Adduct-independent matching
- Neutral mass-based library searches

**Parameters**:
- `tolerance` (float, default=0.1): Maximum absolute mass difference (Daltons) for matching. Unlike `PrecursorMzMatch`, `ParentMassMatch` takes only `tolerance` — there is no `tolerance_type` argument.

**Example**:
```python
from matchms.similarity import ParentMassMatch

similarity_func = ParentMassMatch(tolerance=0.1)
scores = calculate_scores(references, queries, similarity_func)
```

**Output**: 1.0 (match) or 0.0 (no match)

**Requirements**: Both spectra must have valid parent_mass metadata.

---

## Combining Multiple Similarity Functions

Combine multiple similarity metrics for robust compound identification:

`Scores.scores` is a sparse structured array, so indexing it (`scores.scores[j, i]`)
returns a record with `..._score` and `..._matches` fields — NOT a plain float.
To do arithmetic, pull a dense float matrix with `to_array("<FunctionName>_score")`:

```python
from matchms import calculate_scores
from matchms.similarity import CosineGreedy, ModifiedCosineGreedy, FingerprintSimilarity

# Calculate multiple similarity scores
cosine_scores = calculate_scores(refs, queries, CosineGreedy())
modified_cosine_scores = calculate_scores(refs, queries, ModifiedCosineGreedy())
fingerprint_scores = calculate_scores(refs, queries, FingerprintSimilarity())

# Extract dense float matrices (rows = references, cols = queries)
cos = cosine_scores.scores.to_array("CosineGreedy_score")
mod = modified_cosine_scores.scores.to_array("ModifiedCosineGreedy_score")
# FingerprintSimilarity returns a single unnamed score field:
fp = fingerprint_scores.scores.to_array("FingerprintSimilarity")

combined = 0.5 * cos + 0.3 * mod + 0.2 * fp  # element-wise, shape (n_refs, n_queries)
```

## Accessing Scores Results

The `Scores` object provides multiple methods to access results:

```python
# Get best matches for a query. For cosine-family scores you MUST pass the
# field name to sort on; bare sort=True raises IndexError.
best_matches = scores.scores_by_query(query_spectrum,
                                      name="CosineGreedy_score",
                                      sort=True)[:10]

# scores.scores is a sparse, structured (StackedSparseArray) object, not a
# plain numpy float matrix. Pull a dense float matrix by score-field name:
score_matrix = scores.scores.to_array("CosineGreedy_score")  # shape (n_refs, n_queries)

# There is no to_dataframe()/to_list(). Build a DataFrame from to_array:
import pandas as pd
df = pd.DataFrame(score_matrix,
                  index=[r.get("compound_name") for r in scores.references],
                  columns=[q.get("compound_name") for q in scores.queries])

# Drop low scores IN PLACE (filter_by_range mutates `scores` and returns None):
scores.filter_by_range(name="CosineGreedy_score", low=0.7)

# Save scores
scores.to_json("scores.json")
scores.to_pickle("scores.pkl")
```

## Performance Considerations

**Fast methods** (large datasets):
- CosineGreedy
- PrecursorMzMatch
- ParentMassMatch

**Slow methods** (smaller datasets or high accuracy):
- CosineHungarian
- ModifiedCosineGreedy (slower than CosineGreedy)
- NeutralLossesCosine
- FingerprintSimilarity (requires fingerprint computation)

**Recommendation**: For large-scale library searches, use PrecursorMzMatch to pre-filter candidates, then apply CosineGreedy or ModifiedCosineGreedy to filtered results.

## Common Similarity Workflows

### Standard Library Matching
```python
from matchms.similarity import CosineGreedy

scores = calculate_scores(library_spectra, query_spectra,
                         CosineGreedy(tolerance=0.1))
```

### Multi-Metric Matching
```python
from matchms.similarity import CosineGreedy, ModifiedCosineGreedy, FingerprintSimilarity

# Spectral similarity
cosine = calculate_scores(refs, queries, CosineGreedy())
modified = calculate_scores(refs, queries, ModifiedCosineGreedy())

# Structural similarity
fingerprint = calculate_scores(refs, queries, FingerprintSimilarity())
```

### Precursor-Filtered Matching
```python
from matchms.similarity import PrecursorMzMatch, CosineGreedy

# First filter by precursor mass
mass_filter = calculate_scores(refs, queries, PrecursorMzMatch(tolerance=0.1))

# Then calculate cosine only for matching precursors
cosine_scores = calculate_scores(refs, queries, CosineGreedy())
```

## Further Reading

For detailed API documentation, parameter descriptions, and mathematical formulations, see:
https://matchms.readthedocs.io/en/latest/api/matchms.similarity.html

# Matchms Common Workflows

This document provides detailed examples of common mass spectrometry analysis workflows using matchms.

## Workflow 1: Basic Spectral Library Matching

Match unknown spectra against a reference library to identify compounds.

```python
from matchms.importing import load_from_mgf
from matchms.filtering import default_filters, normalize_intensities
from matchms.filtering import select_by_relative_intensity, require_minimum_number_of_peaks
from matchms import calculate_scores
from matchms.similarity import CosineGreedy

# Load reference library
print("Loading reference library...")
library = list(load_from_mgf("reference_library.mgf"))

# Load query spectra (unknowns)
print("Loading query spectra...")
queries = list(load_from_mgf("unknown_spectra.mgf"))

# Process library spectra
print("Processing library...")
processed_library = []
for spectrum in library:
    spectrum = default_filters(spectrum)
    spectrum = normalize_intensities(spectrum)
    spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
    spectrum = require_minimum_number_of_peaks(spectrum, n_required=5)
    if spectrum is not None:
        processed_library.append(spectrum)

# Process query spectra
print("Processing queries...")
processed_queries = []
for spectrum in queries:
    spectrum = default_filters(spectrum)
    spectrum = normalize_intensities(spectrum)
    spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
    spectrum = require_minimum_number_of_peaks(spectrum, n_required=5)
    if spectrum is not None:
        processed_queries.append(spectrum)

# Calculate similarities
print("Calculating similarities...")
scores = calculate_scores(references=processed_library,
                         queries=processed_queries,
                         similarity_function=CosineGreedy(tolerance=0.1))

# Get top matches for each query.
# scores_by_query returns (reference_Spectrum, structured_score) tuples — the
# first element is the matched reference spectrum itself (not an index), and the
# score is a structured record whose float lives in the "<Func>_score" field.
# You MUST pass name=... to sort cosine-family scores.
print("\nTop matches:")
for i, query in enumerate(processed_queries):
    top_matches = scores.scores_by_query(query, name="CosineGreedy_score", sort=True)[:5]

    query_name = query.get("compound_name", f"Query {i}")
    print(f"\n{query_name}:")

    for ref_spectrum, score in top_matches:
        ref_name = ref_spectrum.get("compound_name", "unknown")
        print(f"  {ref_name}: {score['CosineGreedy_score']:.4f}")
```

---

## Workflow 2: Quality Control and Data Cleaning

Filter and clean spectral data before analysis.

```python
from matchms.importing import load_from_mgf
from matchms.exporting import save_as_mgf
from matchms.filtering import (default_filters, normalize_intensities,
                               require_precursor_mz, require_minimum_number_of_peaks,
                               require_minimum_number_of_high_peaks,
                               select_by_relative_intensity, remove_peaks_around_precursor_mz)

# Load spectra
spectra = list(load_from_mgf("raw_data.mgf"))
print(f"Loaded {len(spectra)} raw spectra")

# Apply quality filters
cleaned_spectra = []
for spectrum in spectra:
    # Harmonize metadata
    spectrum = default_filters(spectrum)

    # Quality requirements
    spectrum = require_precursor_mz(spectrum, minimum_accepted_mz=50.0)
    if spectrum is None:
        continue

    spectrum = require_minimum_number_of_peaks(spectrum, n_required=10)
    if spectrum is None:
        continue

    # Clean peaks
    spectrum = normalize_intensities(spectrum)
    spectrum = remove_peaks_around_precursor_mz(spectrum, mz_tolerance=17)
    spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)

    # Require high-quality peaks
    spectrum = require_minimum_number_of_high_peaks(spectrum,
                                                     no_peaks=5,
                                                     intensity_percent=5.0)  # percent, not fraction
    if spectrum is None:
        continue

    cleaned_spectra.append(spectrum)

print(f"Retained {len(cleaned_spectra)} high-quality spectra")
print(f"Removed {len(spectra) - len(cleaned_spectra)} low-quality spectra")

# Save cleaned data
save_as_mgf(cleaned_spectra, "cleaned_data.mgf")
```

---

## Workflow 3: Multi-Metric Similarity Scoring

Combine multiple similarity metrics for robust compound identification.

```python
from matchms.importing import load_from_mgf
from matchms.filtering import (default_filters, normalize_intensities,
                               derive_inchi_from_smiles, add_fingerprint)
from matchms import calculate_scores
from matchms.similarity import (CosineGreedy, ModifiedCosineGreedy,
                                NeutralLossesCosine, FingerprintSimilarity)
import numpy as np

# Load spectra
library = list(load_from_mgf("library.mgf"))
queries = list(load_from_mgf("queries.mgf"))

# Process with multiple features
def process_for_multimetric(spectrum):
    spectrum = default_filters(spectrum)
    spectrum = normalize_intensities(spectrum)

    # Add chemical fingerprints (fingerprint_type: "morgan1"/"morgan2"/"morgan3"/"daylight")
    spectrum = derive_inchi_from_smiles(spectrum)
    spectrum = add_fingerprint(spectrum, fingerprint_type="morgan2", nbits=2048)

    # NeutralLossesCosine derives losses internally from precursor_mz, so no
    # add_losses step is needed here.
    return spectrum

processed_library = [process_for_multimetric(s) for s in library if s is not None]
processed_queries = [process_for_multimetric(s) for s in queries if s is not None]

# Calculate multiple similarity scores
print("Calculating Cosine similarity...")
cosine_scores = calculate_scores(processed_library, processed_queries,
                                 CosineGreedy(tolerance=0.1))

print("Calculating Modified Cosine similarity...")
modified_cosine_scores = calculate_scores(processed_library, processed_queries,
                                         ModifiedCosineGreedy(tolerance=0.1))

print("Calculating Neutral Losses similarity...")
neutral_losses_scores = calculate_scores(processed_library, processed_queries,
                                        NeutralLossesCosine(tolerance=0.1))

print("Calculating Fingerprint similarity...")
fingerprint_scores = calculate_scores(processed_library, processed_queries,
                                      FingerprintSimilarity(similarity_measure="jaccard"))

# Combine scores with weights.
# scores.scores is a structured sparse array, so pull dense float matrices by
# the score-field name first (shape = (n_references, n_queries)), then do
# vectorized arithmetic — never index `.scores[j, i]` directly for a float.
cos = cosine_scores.scores.to_array("CosineGreedy_score")
mod = modified_cosine_scores.scores.to_array("ModifiedCosineGreedy_score")
nl = neutral_losses_scores.scores.to_array("NeutralLossesCosine_score")
fp = fingerprint_scores.scores.to_array("FingerprintSimilarity")

combined_matrix = 0.4 * cos + 0.3 * mod + 0.2 * nl + 0.1 * fp  # (n_refs, n_queries)

# Get combined scores for each query
import numpy as np
for i, query in enumerate(processed_queries):
    query_name = query.get("compound_name", f"Query {i}")

    order = np.argsort(combined_matrix[:, i])[::-1]  # best refs first
    print(f"\n{query_name} - Top 3 matches:")
    for ref_idx in order[:3]:
        ref_name = processed_library[ref_idx].get("compound_name", f"Ref {ref_idx}")
        print(f"  {ref_name}: {combined_matrix[ref_idx, i]:.4f}")
```

---

## Workflow 4: Precursor-Filtered Library Search

Pre-filter by precursor mass before spectral matching for faster searches.

```python
from matchms.importing import load_from_mgf
from matchms.filtering import default_filters, normalize_intensities
from matchms import calculate_scores
from matchms.similarity import PrecursorMzMatch, CosineGreedy
import numpy as np

# Load data
library = list(load_from_mgf("large_library.mgf"))
queries = list(load_from_mgf("queries.mgf"))

# Process spectra
processed_library = [normalize_intensities(default_filters(s)) for s in library]
processed_queries = [normalize_intensities(default_filters(s)) for s in queries]

# Step 1: Fast precursor mass filtering
print("Filtering by precursor mass...")
mass_filter = calculate_scores(processed_library, processed_queries,
                               PrecursorMzMatch(tolerance=0.1, tolerance_type="Dalton"))

# Step 2: Calculate cosine only for matching precursors
print("Calculating cosine similarity for filtered candidates...")
cosine_scores = calculate_scores(processed_library, processed_queries,
                                CosineGreedy(tolerance=0.1))

# Step 3: Apply mass filter to cosine scores.
# Pull dense float matrices by score-field name (don't index .scores[j, i],
# which yields a structured record, not a float).
import numpy as np
mass_mask = mass_filter.scores.to_array("PrecursorMzMatch")     # 1.0 / 0.0
cos = cosine_scores.scores.to_array("CosineGreedy_score")

for i, query in enumerate(processed_queries):
    # Keep only references whose precursor matched, then rank by cosine
    candidates = [(j, cos[j, i]) for j in range(len(processed_library))
                  if mass_mask[j, i] > 0]
    candidates.sort(key=lambda x: x[1], reverse=True)

    query_name = query.get("compound_name", f"Query {i}")
    print(f"\n{query_name} - Top 5 matches (from {len(candidates)} candidates):")

    for ref_idx, score in candidates[:5]:
        ref_name = processed_library[ref_idx].get("compound_name", f"Ref {ref_idx}")
        ref_mz = processed_library[ref_idx].get("precursor_mz", "N/A")
        print(f"  {ref_name} (m/z {ref_mz}): {score:.4f}")
```

---

## Workflow 5: Building a Reusable Processing Pipeline

Create a standardized pipeline for consistent processing.

```python
import pickle
from matchms import SpectrumProcessor
from matchms.filtering import default_filters
from matchms.importing import load_from_mgf
# Other filters are referenced by name (string) below; default_filters is a
# composite and must be passed as the callable, not the string "default_filters".
# matchms has no save_as_pickle — pickle files are written with stdlib pickle.

# Define custom processing pipeline. Prefer ("filter_name", {kwargs}) tuples:
# they're introspectable via pipeline.processing_steps for reproducibility.
def create_standard_pipeline():
    """Create a reusable processing pipeline"""
    return SpectrumProcessor([
        default_filters,
        "normalize_intensities",
        ("remove_peaks_around_precursor_mz", {"mz_tolerance": 17}),
        ("select_by_relative_intensity", {"intensity_from": 0.01}),
        ("require_minimum_number_of_peaks", {"n_required": 5}),
        "derive_inchi_from_smiles",
        ("add_fingerprint", {"fingerprint_type": "morgan2"}),
    ])

# Create pipeline instance
pipeline = create_standard_pipeline()

# Process multiple datasets with same pipeline
datasets = ["dataset1.mgf", "dataset2.mgf", "dataset3.mgf"]

for dataset_file in datasets:
    print(f"\nProcessing {dataset_file}...")

    # Load spectra
    spectra = list(load_from_mgf(dataset_file))

    # Apply pipeline. A SpectrumProcessor is NOT callable — use process_spectra
    # (returns a (spectra, report) tuple) or process_spectrum for one spectrum.
    processed, report = pipeline.process_spectra(spectra)

    print(f"  Loaded: {len(spectra)}")
    print(f"  Processed: {len(processed)}")

    # Save processed data (stdlib pickle; reload with load_from_pickle)
    output_file = dataset_file.replace(".mgf", "_processed.pkl")
    with open(output_file, "wb") as f:
        pickle.dump(processed, f)
    print(f"  Saved to: {output_file}")
```

---

## Workflow 6: Format Conversion and Standardization

Convert between different mass spectrometry file formats.

```python
from matchms.importing import load_from_mzml, load_from_mgf
from matchms.exporting import save_as_mgf, save_as_msp, save_as_json
from matchms.filtering import default_filters, normalize_intensities

def convert_and_standardize(input_file, output_format="mgf"):
    """
    Load, standardize, and convert mass spectrometry data

    Parameters:
    -----------
    input_file : str
        Input file path (supports .mzML, .mzXML, .mgf)
    output_format : str
        Output format ('mgf', 'msp', or 'json')
    """
    # Determine input format and load
    if input_file.endswith('.mzML') or input_file.endswith('.mzXML'):
        from matchms.importing import load_from_mzml
        spectra = list(load_from_mzml(input_file, ms_level=2))
    elif input_file.endswith('.mgf'):
        spectra = list(load_from_mgf(input_file))
    else:
        raise ValueError(f"Unsupported format: {input_file}")

    print(f"Loaded {len(spectra)} spectra from {input_file}")

    # Standardize
    processed = []
    for spectrum in spectra:
        spectrum = default_filters(spectrum)
        spectrum = normalize_intensities(spectrum)
        if spectrum is not None:
            processed.append(spectrum)

    print(f"Standardized {len(processed)} spectra")

    # Export
    output_file = input_file.rsplit('.', 1)[0] + f'_standardized.{output_format}'

    if output_format == 'mgf':
        save_as_mgf(processed, output_file)
    elif output_format == 'msp':
        save_as_msp(processed, output_file)
    elif output_format == 'json':
        save_as_json(processed, output_file)
    else:
        raise ValueError(f"Unsupported output format: {output_format}")

    print(f"Saved to {output_file}")
    return processed

# Convert mzML to MGF
convert_and_standardize("raw_data.mzML", output_format="mgf")

# Convert MGF to MSP library format
convert_and_standardize("library.mgf", output_format="msp")
```

---

## Workflow 7: Metadata Enrichment and Validation

Enrich spectra with chemical structure information and validate annotations.

```python
from matchms.importing import load_from_mgf
from matchms.exporting import save_as_mgf
from matchms.filtering import (default_filters, derive_inchi_from_smiles,
                               derive_inchikey_from_inchi, derive_smiles_from_inchi,
                               add_fingerprint, repair_not_matching_annotation,
                               require_valid_annotation)

# Load spectra
spectra = list(load_from_mgf("spectra.mgf"))

# Enrich and validate
enriched_spectra = []
validation_failures = []

for i, spectrum in enumerate(spectra):
    # Basic harmonization
    spectrum = default_filters(spectrum)

    # Derive chemical structures
    spectrum = derive_inchi_from_smiles(spectrum)
    spectrum = derive_inchikey_from_inchi(spectrum)
    spectrum = derive_smiles_from_inchi(spectrum)

    # Repair mismatches
    spectrum = repair_not_matching_annotation(spectrum)

    # Add molecular fingerprints
    spectrum = add_fingerprint(spectrum, fingerprint_type="morgan2", nbits=2048)

    # Validate
    validated = require_valid_annotation(spectrum)

    if validated is not None:
        enriched_spectra.append(validated)
    else:
        validation_failures.append(i)

print(f"Successfully enriched: {len(enriched_spectra)}")
print(f"Validation failures: {len(validation_failures)}")

# Save enriched data
save_as_mgf(enriched_spectra, "enriched_spectra.mgf")

# Report failures
if validation_failures:
    print("\nSpectra that failed validation:")
    for idx in validation_failures[:10]:  # Show first 10
        original = spectra[idx]
        name = original.get("compound_name", f"Spectrum {idx}")
        print(f"  - {name}")
```

---

## Workflow 8: Large-Scale Library Comparison

Compare two large spectral libraries efficiently.

```python
from matchms.importing import load_from_mgf
from matchms.filtering import default_filters, normalize_intensities
from matchms import calculate_scores
from matchms.similarity import CosineGreedy
import numpy as np

# Load two libraries
print("Loading libraries...")
library1 = list(load_from_mgf("library1.mgf"))
library2 = list(load_from_mgf("library2.mgf"))

# Process
processed_lib1 = [normalize_intensities(default_filters(s)) for s in library1]
processed_lib2 = [normalize_intensities(default_filters(s)) for s in library2]

# Calculate all-vs-all similarities
print("Calculating similarities...")
scores = calculate_scores(processed_lib1, processed_lib2,
                         CosineGreedy(tolerance=0.1))

# Find high-similarity pairs (potential duplicates or similar compounds).
# Pull a dense float matrix by score-field name; rows = lib1 (references),
# cols = lib2 (queries). Indexing scores.scores[i, j] directly gives a
# structured record, not a comparable float.
score_matrix = scores.scores.to_array("CosineGreedy_score")

threshold = 0.8
similar_pairs = []

for i, spec1 in enumerate(processed_lib1):
    for j, spec2 in enumerate(processed_lib2):
        score = float(score_matrix[i, j])
        if score >= threshold:
            similar_pairs.append({
                'lib1_idx': i,
                'lib2_idx': j,
                'lib1_name': spec1.get("compound_name", f"L1_{i}"),
                'lib2_name': spec2.get("compound_name", f"L2_{j}"),
                'similarity': score
            })

# Sort by similarity
similar_pairs.sort(key=lambda x: x['similarity'], reverse=True)

print(f"\nFound {len(similar_pairs)} pairs with similarity >= {threshold}")
print("\nTop 10 most similar pairs:")
for pair in similar_pairs[:10]:
    print(f"{pair['lib1_name']} <-> {pair['lib2_name']}: {pair['similarity']:.4f}")

# Export to CSV
import pandas as pd
df = pd.DataFrame(similar_pairs)
df.to_csv("library_comparison.csv", index=False)
print("\nFull results saved to library_comparison.csv")
```

---

## Workflow 9: Ion Mode Specific Processing

Process positive and negative mode spectra separately.

```python
from matchms.importing import load_from_mgf
from matchms.filtering import (default_filters, normalize_intensities,
                               require_correct_ionmode, derive_ionmode)
from matchms.exporting import save_as_mgf

# Load mixed mode spectra
spectra = list(load_from_mgf("mixed_modes.mgf"))

# Separate by ion mode
positive_spectra = []
negative_spectra = []
unknown_mode = []

for spectrum in spectra:
    # Harmonize and derive ion mode
    spectrum = default_filters(spectrum)
    spectrum = derive_ionmode(spectrum)

    # Separate by mode
    ionmode = spectrum.get("ionmode")

    if ionmode == "positive":
        spectrum = normalize_intensities(spectrum)
        positive_spectra.append(spectrum)
    elif ionmode == "negative":
        spectrum = normalize_intensities(spectrum)
        negative_spectra.append(spectrum)
    else:
        unknown_mode.append(spectrum)

print(f"Positive mode: {len(positive_spectra)}")
print(f"Negative mode: {len(negative_spectra)}")
print(f"Unknown mode: {len(unknown_mode)}")

# Save separated data
save_as_mgf(positive_spectra, "positive_mode.mgf")
save_as_mgf(negative_spectra, "negative_mode.mgf")

# Process mode-specific analyses
from matchms import calculate_scores
from matchms.similarity import CosineGreedy

if len(positive_spectra) > 1:
    print("\nCalculating positive mode similarities...")
    pos_scores = calculate_scores(positive_spectra, positive_spectra,
                                  CosineGreedy(tolerance=0.1))

if len(negative_spectra) > 1:
    print("Calculating negative mode similarities...")
    neg_scores = calculate_scores(negative_spectra, negative_spectra,
                                  CosineGreedy(tolerance=0.1))
```

---

## Workflow 10: Automated Compound Identification Report

Generate a detailed compound identification report.

```python
from matchms.importing import load_from_mgf
from matchms.filtering import default_filters, normalize_intensities
from matchms import calculate_scores
from matchms.similarity import CosineGreedy, ModifiedCosineGreedy
import numpy as np
import pandas as pd

def identify_compounds(query_file, library_file, output_csv="identification_report.csv"):
    """
    Automated compound identification with detailed report
    """
    # Load data
    print("Loading data...")
    queries = list(load_from_mgf(query_file))
    library = list(load_from_mgf(library_file))

    # Process
    proc_queries = [normalize_intensities(default_filters(s)) for s in queries]
    proc_library = [normalize_intensities(default_filters(s)) for s in library]

    # Calculate similarities
    print("Calculating similarities...")
    cosine_scores = calculate_scores(proc_library, proc_queries, CosineGreedy())
    modified_scores = calculate_scores(proc_library, proc_queries, ModifiedCosineGreedy())

    # Dense float matrices (rows = references/library, cols = queries)
    cos = cosine_scores.scores.to_array("CosineGreedy_score")
    mod = modified_scores.scores.to_array("ModifiedCosineGreedy_score")

    # Generate report
    results = []
    for i, query in enumerate(proc_queries):
        query_name = query.get("compound_name", f"Unknown_{i}")
        query_mz = query.get("precursor_mz", "N/A")

        # Rank library by cosine for this query, take top 5
        top_lib = np.argsort(cos[:, i])[::-1][:5]

        for rank, lib_idx in enumerate(top_lib, 1):
            ref = proc_library[lib_idx]
            results.append({
                'Query': query_name,
                'Query_mz': query_mz,
                'Rank': rank,
                'Match': ref.get("compound_name", f"Ref_{lib_idx}"),
                'Match_mz': ref.get("precursor_mz", "N/A"),
                'Cosine_Score': float(cos[lib_idx, i]),
                'Modified_Cosine': float(mod[lib_idx, i]),
                'InChIKey': ref.get("inchikey", "N/A")
            })

    # Create DataFrame and save
    df = pd.DataFrame(results)
    df.to_csv(output_csv, index=False)
    print(f"\nReport saved to {output_csv}")

    # Summary statistics
    print("\nSummary:")
    high_confidence = len(df[df['Cosine_Score'] >= 0.8])
    medium_confidence = len(df[(df['Cosine_Score'] >= 0.6) & (df['Cosine_Score'] < 0.8)])
    low_confidence = len(df[df['Cosine_Score'] < 0.6])

    print(f"  High confidence (≥0.8): {high_confidence}")
    print(f"  Medium confidence (0.6-0.8): {medium_confidence}")
    print(f"  Low confidence (<0.6): {low_confidence}")

    return df

# Run identification
report = identify_compounds("unknowns.mgf", "reference_library.mgf")
```

---

## Best Practices

1. **Always process both queries and references**: Apply the same filters to ensure consistent comparison
2. **Save intermediate results**: Use pickle format for fast reloading of processed spectra
3. **Monitor memory usage**: Use generators for large files instead of loading all at once
4. **Validate data quality**: Apply quality filters before similarity calculations
5. **Choose appropriate similarity metrics**: CosineGreedy for speed, ModifiedCosineGreedy for related compounds
6. **Combine multiple metrics**: Use multiple similarity scores for robust identification
7. **Filter by precursor mass first**: Dramatically speeds up large library searches
8. **Document your pipeline**: Save processing parameters for reproducibility

## Further Resources

- matchms documentation: https://matchms.readthedocs.io
- GNPS platform: https://gnps.ucsd.edu
- matchms GitHub: https://github.com/matchms/matchms