{
  "mcpServers": {
    "scomp-link": {
      "command": "scomp-link",
      "args": ["mcp"],
      "env": {}
    }
  }
}

# scomp-link Pipeline Configuration Template
# Run with: scomp-link pipeline --config this_file.yaml

# Required fields
data: data/train.csv          # Path to training data (CSV, TSV, Parquet)
target: target_column         # Target column name
task: regression              # regression | classification

# Optional: pipeline name
name: my_pipeline

# Optional: Feature Engineering
# Uncomment to enable
# engineer:
#   interactions: true        # Polynomial feature interactions
#   log_transform: true       # Log1p for skewed distributions
#   date_features: false      # Extract year/month/dow from date columns
#   target_encode: false      # Target-encode high-cardinality categoricals
#   auto_bin: false           # Quantile binning

# Optional: Hyperparameter Tuning
# Uncomment to enable
# tune:
#   method: optuna            # optuna | halving
#   n_trials: 50              # Number of optimization trials

# Optional: Model Hint (skip auto-selection)
# model_hint: numerical_prediction  # Force specific model type

# Optional: Ensemble Learning
# ensemble: voting            # voting | stacking

# Optional: Advanced Cross-Validation
# advanced_cv: true

# Optional: Test Size
# test_size: 0.2

# Optional: Validation on separate test data
# validate:
#   test_data: data/test.csv  # Path to test data
#   report: reports/validation_report.html  # HTML report output

# Optional: Save trained model
# save: models/my_model.scomp

{
  "mcpServers": {
    "scomp-link": {
      "command": "scomp-link",
      "args": ["mcp"]
    }
  }
}

{
  "scomp-link": {
    "command": "scomp-link",
    "args": ["mcp"]
  }
}

{
  "mcpServers": {
    "scomp-link": {
      "command": "scomp-link",
      "args": ["mcp"],
      "env": {}
    }
  }
}

{
  "servers": {
    "scomp-link": {
      "command": "scomp-link",
      "args": ["mcp"]
    }
  }
}

# Python API Reference

## Core Pipeline

### `ScompLinkPipeline`

```python
from scomp_link import ScompLinkPipeline

pipe = ScompLinkPipeline("Project Name")
pipe.set_objectives(["Minimize RMSE"])
pipe.import_and_clean_data(df)                    # pandas DataFrame
pipe.select_variables(target_col='y', feature_cols=['x1', 'x2'])  # feature_cols optional
pipe.choose_model("numerical_prediction", metadata={})  # see model selection below
results = pipe.run_pipeline(
    task_type="regression",          # regression | classification | clustering | text | image | image_clustering
    test_size=0.2,
    models_to_test=None,             # dict for RegressorOptimizer/ClassifierOptimizer
    use_ensemble=False,              # voting/stacking ensemble
    ensemble_strategy='voting',      # voting | stacking
    advanced_cv=False,               # Enable LOOCV + Bootstrap
    cv_methods=['bootstrap'],        # loocv | bootstrap
    bootstrap_iterations=1000,
    # Text-specific:
    text_col=None,                   # column with text data
    use_contrastive=True,            # BERT contrastive vs TF-IDF
    text_model='bert-base-uncased',
    epochs=3, batch_size=32,
    # Image-specific:
    image_col=None,                  # column with image arrays
    # Clustering-specific:
    n_clusters=None,
)
pipe.save_model('./staging')
pipe.load_model('./staging')
predictions = pipe.predict(X_new)
```

**`choose_model` objective types:**
- `"numerical_prediction"` → auto-selects Ridge/Lasso/GBR based on data size
- `"categorical_known"` → auto-selects SVC/NaiveBayes/GBR based on features
- `"categorical_unknown"` → KMeans or MeanShift
- `"numerical_study"` → PCA / Geostatistical / UCM
- `"multi_numerical_prediction"` → VAR/VARMA or MLP

## Persistence

### `ScompArtifact`

```python
from scomp_link import ScompArtifact

# Save
artifact = ScompArtifact()
artifact.set_model(model)
artifact.set_preprocessor(scaler)                # optional sklearn transformer
artifact.set_config(task_type='regression', target_col='y', feature_cols=['x1', 'x2'])
artifact.set_metrics({'r2': 0.95, 'rmse': 2.1})
artifact.set_feature_schema(X_train)             # stores dtype/range per feature
artifact.set_sample_data(X_train, max_rows=200)  # for drift detection
artifact.set_metadata(author='team', version='1.0', description='...')
artifact.save('model.scomp')

# Load
loaded = ScompArtifact.load('model.scomp')
predictions = loaded.predict(X_new)              # chains preprocessor + model
info = loaded.info()                             # dict with all metadata
schema = loaded.feature_schema                   # dict {col: {dtype, min, max, mean}}
sample = loaded.sample_data                      # DataFrame

# Validate
ScompArtifact.is_scomp_file('model.scomp')      # returns bool
```

## Preprocessing

### `Preprocessor`

```python
from scomp_link import Preprocessor

prep = Preprocessor(df)                          # accepts pandas or polars
cleaned_df = prep.clean_data(remove_outliers=True, outlier_threshold=3.0)
integrated_df = prep.integrate_data(other_df, on='id', how='left')
features = prep.feature_selection(target_col='y', n_features=10)
eda = prep.run_eda()                             # returns dict with shape, missing, dtypes
X_train, X_test, y_train, y_test = prep.prepare_datasets(target_col='y', test_size=0.2)
```

### `FeatureEngineer`

```python
from scomp_link import FeatureEngineer

fe = FeatureEngineer(
    interactions=True,        # polynomial interactions between numeric features
    log_transform=True,       # log1p for skewed features (skew > threshold)
    skew_threshold=0.8,       # skewness threshold for log transform
    date_features=True,       # extract year/month/dow/weekend from date columns
    target_encode=True,       # target-encode high-cardinality categoricals
    target_encode_threshold=8,  # min unique values to trigger target encoding
    auto_bin=True,            # quantile binning
    n_bins=5,                 # number of bins
)
X_train_eng = fe.fit_transform(X_train, y_train)
X_test_eng = fe.transform(X_test)
```

### `DataQualityReport`

```python
from scomp_link import DataQualityReport

dqr = DataQualityReport(df)
report = dqr.generate()    # returns dict with: overview, missing, constants, duplicates, correlations, cardinality
dqr.save_html('quality.html')
```

## Models

### `RegressorOptimizer`

```python
from scomp_link import RegressorOptimizer
from sklearn.linear_model import Ridge, Lasso
from sklearn.ensemble import GradientBoostingRegressor

models_to_test = {
    'Ridge': {'model': Ridge(), 'params_grid': {'alpha': [0.1, 1.0, 10.0]}},
    'GBR': {'model': GradientBoostingRegressor(), 'params_grid': {'n_estimators': [50, 100], 'max_depth': [3, 5]}},
}

opt = RegressorOptimizer(df, 'target', x_cols, x_complexity_col='x1',
                         models_to_test=models_to_test, select_features=True)
opt.estimate_optimization_time(time_per_combination=2)
opt.test_models_regression()
opt.grafico_fit_con_errore('Ridge')  # matplotlib plot
# Results: opt.model_results dict with Model, Params, fitted values
```

### `ClassifierOptimizer`

```python
from scomp_link import ClassifierOptimizer

opt = ClassifierOptimizer(df, 'target', x_cols, models_to_test=models, select_features=True)
opt.test_models_classification()
opt.print_results()
# Results: opt.model_results dict with Model, Params, Report, Confusion_Matrix
```

### `AnomalyDetector`

```python
from scomp_link import AnomalyDetector

detector = AnomalyDetector(
    contamination=0.05,
    methods=['iforest', 'lof', 'tabnet', 'transformer'],
    consensus_threshold=2,
    tabnet_epochs=50,
    transformer_epochs=80,
)
results = detector.fit_predict(df, features=['col1', 'col2'])
# results['data'] — DataFrame with is_anomaly column
# results['comparison'] — method-by-method stats
```

### `TimeSeriesForecaster`

```python
from scomp_link import TimeSeriesForecaster

fc = TimeSeriesForecaster(method='auto', horizon=30, seasonal_period=12)
fc.fit(series)                          # pandas Series
forecast = fc.predict()                 # Series of predicted values
ci = fc.predict_with_ci(steps=10)       # DataFrame with forecast, lower, upper
cv = fc.walk_forward_cv(series, n_splits=5, horizon=12)  # dict with mean_mae, mean_rmse, mean_mape
```

### Tuning

```python
from scomp_link import OptunaOptimizer, HalvingSearchOptimizer, EarlyStoppingCV

# Optuna
def param_space(trial):
    return {'n_estimators': trial.suggest_int('n_estimators', 50, 500), ...}
opt = OptunaOptimizer(ModelClass, param_space, scoring='r2', n_trials=100)
best_model = opt.optimize(X_train, y_train)

# Halving
halving = HalvingSearchOptimizer(model, param_grid, scoring='r2')
best_model = halving.optimize(X_train, y_train)
```

## Validation

### `Validator`

```python
from scomp_link import Validator

val = Validator(model)
metrics = val.evaluate(y_true, y_pred, task_type='regression')  # mse, rmse, mae, r2
scores = val.k_fold_cv(X, y, k=5)
loocv_scores = val.loocv(X, y)
val.generate_validation_report(y_true, y_pred, task_type='regression', report_name='report.html')
```

### `FairnessMetrics`

```python
from scomp_link import FairnessMetrics

fm = FairnessMetrics(y_true, y_pred, sensitive_feature=gender_array)
report = fm.compute_all()      # demographic_parity, disparate_impact, equalized_odds
summary = fm.summary(report)   # DataFrame
fig = fm.plot_fairness_report(report)
```

## Monitoring

### `DriftDetector`

```python
from scomp_link import DriftDetector

detector = DriftDetector(reference_df, psi_threshold=0.2, ks_alpha=0.05)
report = detector.detect(current_df, features=['col1', 'col2'])  # optional feature subset
summary = detector.summary(report)   # dict: drifted_features, total_features, max_psi, worst_feature
fig = detector.plot_drift_report(report)
fig_dist = detector.plot_feature_distribution('col1', current_df)
```

## Explainability

```python
from scomp_link import ShapExplainer, LimeExplainer

# SHAP
shap_exp = ShapExplainer(model, X_background[:100])
shap_exp.explain(X_test)
importance = shap_exp.feature_importance()  # DataFrame: feature, importance
fig = shap_exp.plot_importance(top_n=10)

# LIME
lime_exp = LimeExplainer(model, X_train, task='regression')  # or 'classification'
exp = lime_exp.explain_instance(X_test.iloc[0], num_features=5)
exp.as_list()   # [(feature_rule, weight), ...]
fig = lime_exp.plot_explanation(exp)
```

## Pipeline DSL (`>>` operator)

Compose pipelines declaratively. Lazy: `>>` builds, `.run()` executes.

### ML chain

```python
from scomp_link import CleanStep, SelectStep, ModelStep, TrainStep

results = (
    CleanStep(df)
    >> SelectStep("target", features=["col1", "col2"])
    >> ModelStep("numerical_prediction")          # or "categorical_known", "categorical_unknown"
    >> TrainStep("regression")                    # or "classification", "clustering"
).run()
# → {"status": "success", "model_type": ..., "metrics": {...}}
```

### Report chain

```python
from scomp_link import SectionStep, TitleStep, TextStep, TableStep, GraphStep, RawGraphStep, SaveStep

(
    SectionStep("Section Title")     # opens collapsible section, auto-closes previous
    >> TitleStep("h2 title")
    >> TextStep("paragraph text")
    >> TableStep(df, "Table Title")
    >> GraphStep(plotly_fig, "Chart Title")
    >> RawGraphStep(svg_string, "SVG Chart")
    >> SaveStep("output.html")       # auto-closes open section before saving
).run(report)                        # pass ScompLinkHTMLReport or omit for auto-created default
```

### LogStep (works in both chain types)

```python
from scomp_link import LogStep

CleanStep(df) >> LogStep("after clean") >> SelectStep("y") >> TrainStep("regression")
# LogStep logs df shape / model_type (ML) or section_open state (Report), then passes through
```

### Type safety rules

- Mixing `MLStep` and `ReportStep` in the same chain raises `TypeError` at `>>` time
- `LogStep` is neutral — can appear anywhere in either chain type
- Chain with only `LogStep` instances raises `TypeError` on `.run()`

## Report Builder — Interactive Components (v2.1.0)

```python
from scomp_link.utils.report_html import ScompLinkHTMLReport
report = ScompLinkHTMLReport("My Report")

# KPI cards
report.add_kpi_cards({"RMSE": {"value": 0.81, "trend": "-0.05", "status": "good"}})

# Plotly grid
report.add_plotly_grid([fig1, fig2, fig3], cols=2, titles=["A", "B", "C"])

# Tabs
report.add_tabs({"Overview": "<p>text</p>", "Chart": fig, "Data": df})

# Cascading dropdowns
report.add_cascading_content("By Client x Year",
    dimensions=[{"label":"Client","options":["A","B"]}, {"label":"Year","options":["2024","2025"]}],
    content_map={("A","2024"): fig_a24, ("B","2025"): "<p>No data</p>"},
    cascade=True)

# Comparison table with deltas
report.add_comparison_table(df, baseline_col="v1", compare_cols=["v2"],
                            higher_is_better={"R2": True, "RMSE": False})

# Data profiling
report.add_summary_stats(df, title="Training Data Overview")

# Dark mode toggle
report.add_dark_mode_toggle()

# DataFrame with threshold coloring
report.add_dataframe(df, "Metrics", thresholds={"error": (0.10, 0.25, False), "r2": (0.80, 0.50, True)})
```

## Plotly Utilities — New Functions (v2.1.0)

```python
from scomp_link.utils.plotly_utils import (
    fill_timeslots, normalize_to_index, index_chart, stacked_area_comparison
)

arr = fill_timeslots([1, 2, 3], n_slots=5)           # [1, 2, 3, 0, 0]
idx = normalize_to_index([80, 100, 120])              # mean→100
fig = index_chart({"A": [100,110], "B": [90,105]}, ["Jan","Feb"], "Index")
fig = stacked_area_comparison({"A":[30,40],"B":[70,60]}, {"A":[20,30],"B":[80,70]},
                               ["A","B"], ["2024","2025"], "Comparison")
```

# CLI Reference — All 24 Commands

## Training & Prediction

### `run` — Train a model
```bash
scomp-link run --data PATH --target COL --task {regression,classification,text,clustering,image}
  [--features COL1,COL2]     # Specific features (default: all except target)
  [--text-col COL]           # Text column (for --task text)
  [--image-col COL]          # Image column (for --task image)
  [--n-clusters N]           # Clusters (for --task clustering)
  [--model-hint HINT]        # Force model type (numerical_prediction, categorical_known, etc.)
  [--test-size 0.2]          # Test split ratio
  [--engineer]               # Apply feature engineering before training
  [--ensemble {voting,stacking}]  # Enable ensemble
  [--advanced-cv]            # Run LOOCV + Bootstrap CV
  [--save-artifact PATH]     # Save as .scomp file
  [--output PATH]            # Save results to file
  [--format {json,csv,table}]  # Output format
  [--name NAME]              # Pipeline name
  [--silent]                 # Suppress output
```

### `predict` — Predict with saved artifact
```bash
scomp-link predict --artifact MODEL.scomp --data PATH
  [--output PATH]            # Output file (default: predictions.csv)
  [--silent]
```

### `text` — Text classification
```bash
scomp-link text --data PATH --text-col COL --target COL
  [--method {tfidf,contrastive}]  # tfidf=fast, contrastive=BERT (default: tfidf)
  [--model-name MODEL]       # Transformer model (default: bert-base-uncased)
  [--epochs 3]               # Training epochs
  [--batch-size 32]          # Batch size
  [--test-size 0.2]          # Test split
  [--save-artifact PATH]     # Save artifact
  [--output PATH]            # Results JSON
  [--silent]
```

### `cluster` — Clustering
```bash
scomp-link cluster --data PATH
  [--features COL1,COL2]     # Features to cluster on (default: all numeric)
  [--n-clusters 5]           # Number of clusters
  [--method {kmeans,meanshift}]  # Algorithm (default: kmeans)
  [--output PATH]            # Output CSV with cluster column
  [--plot PATH.html]         # Scatter plot visualization
  [--silent]
```

### `tune` — Hyperparameter tuning
```bash
scomp-link tune --data PATH --target COL --task {regression,classification}
  [--method {optuna,halving}]  # Tuning method (default: optuna)
  [--n-trials 50]            # Number of trials
  [--features COL1,COL2]     # Feature columns
  [--test-size 0.2]          # Test split
  [--save-artifact PATH]     # Save best model
  [--output PATH]            # Results JSON
  [--format {json,csv,table}]
  [--silent]
```

### `pipeline` — Run from YAML config
```bash
scomp-link pipeline --config PATH.yaml
  [--silent]
```

## Evaluation & Monitoring

### `validate` — Evaluate artifact on test data
```bash
scomp-link validate --artifact MODEL.scomp --data PATH --target COL
  [--output PATH]            # Metrics JSON
  [--report PATH.html]       # HTML validation report
  [--format {json,csv,table}]
  [--silent]
```

### `explain` — SHAP feature importance
```bash
scomp-link explain --artifact MODEL.scomp --data PATH
  [--n-samples 100]          # Samples to explain
  [--output PATH]            # Feature importance CSV
  [--silent]
```

### `fairness` — Bias and fairness metrics
```bash
scomp-link fairness --data PATH --target COL --predicted COL --sensitive COL
  [--output PATH]            # Report JSON
  [--silent]
```

### `monitor` — Production monitoring report
```bash
scomp-link monitor --reference PATH --current PATH
  [--artifact MODEL.scomp]   # For performance metrics
  [--target COL]             # Target column (with --artifact)
  [--threshold 0.2]          # PSI drift threshold
  [--output PATH.html]       # Output report
  [--silent]
```

### `compare` — Compare multiple artifacts
```bash
scomp-link compare --artifacts A.scomp B.scomp [C.scomp ...]
  [--output PATH]            # Comparison CSV
  [--plot PATH.html]         # Bar chart comparison
```

## Data & Features

### `describe` — Quick dataset profiling
```bash
scomp-link describe --data PATH
  [--format {table,csv,json}]  # Output format (default: table)
  [--output PATH]            # Save to file
```
Output: one row per column with dtype, missing%, unique, min, max, mean, std.

### `quality` — Full data quality report
```bash
scomp-link quality --data PATH
  [--output PATH.html]       # HTML report (default: data_quality_report.html)
  [--silent]
```

### `engineer` — Feature engineering
```bash
scomp-link engineer --data PATH
  [--target COL]             # Target for target encoding
  [--interactions]           # Polynomial interactions
  [--log-transform]          # Log-transform skewed features
  [--date-features]          # Extract year/month/dow from dates
  [--target-encode]          # Target encode high-cardinality categoricals
  [--auto-bin]               # Quantile binning
  [--n-bins 5]               # Number of bins
  [--output PATH]            # Output file (default: engineered.csv)
  [--silent]
```

### `drift` — Distribution drift detection
```bash
scomp-link drift --reference PATH --current PATH
  [--features COL1,COL2]     # Specific features (default: all numeric)
  [--threshold 0.2]          # PSI threshold
  [--output PATH]            # Drift report CSV
  [--plot PATH.html]         # PSI bar chart
  [--silent]
```

### `anomaly` — Anomaly detection
```bash
scomp-link anomaly --data PATH
  [--features COL1,COL2]     # Features (default: all numeric)
  [--methods iforest,lof,tabnet,transformer]  # Detection methods
  [--contamination 0.05]     # Expected anomaly fraction
  [--consensus 2]            # Min methods that must agree
  [--output PATH]            # Output CSV with anomaly labels
  [--plot PATH.html]         # Anomaly visualization
  [--silent]
```

### `forecast` — Time series forecasting
```bash
scomp-link forecast --data PATH --column COL
  [--horizon 10]             # Steps to forecast
  [--method {auto,arima,sarima,exp_smoothing}]
  [--seasonal-period N]      # Seasonal period
  [--cv-splits N]            # Walk-forward CV splits
  [--output PATH]            # Forecast CSV
  [--plot PATH.html]         # Forecast chart
  [--silent]
```

## Model Lifecycle

### `init` — Scaffold new project
```bash
scomp-link init NAME
  [--force]                  # Overwrite existing
```
Creates: `NAME/{data/, models/, reports/, pipeline.py, config.yaml, .gitignore, README.md}`

### `serve` — REST API server
```bash
scomp-link serve --artifact MODEL.scomp
  [--host 0.0.0.0]           # Bind address
  [--port 8080]              # Port
  [--debug]                  # Flask debug mode
```
Endpoints: `GET /health`, `GET /info`, `GET /schema`, `POST /predict`

POST /predict body: `{"instances": [{"col1": val1, "col2": val2}]}`

### `export` — Export to standard format
```bash
scomp-link export --artifact MODEL.scomp
  [--format {pickle,joblib,onnx,pmml}]  # Export format (default: pickle)
  [--output PATH]            # Output file
```

### `report` — Interactive HTML report
```bash
# EDA report
scomp-link report --data PATH --output PATH.html

# Model evaluation report (requires artifact + data)
scomp-link report --artifact MODEL.scomp --data PATH --output PATH.html
  [--silent]
```

### `info` — Inspect artifact metadata
```bash
scomp-link info --artifact MODEL.scomp
```
Output: JSON with model_type, task, target, metrics, feature_schema, metadata.

## Utilities

### `list-models` — Available model types
```bash
scomp-link list-models
```

### `check-deps` — Dependency status
```bash
scomp-link check-deps
```

### `mcp` — Start MCP server
```bash
scomp-link mcp
```
Starts the Model Context Protocol server for agent integration (stdio transport).



## LLM Commands

All LLM commands require `pip install scomp-link[llm]`.

### `llm finetune` — Fine-tune a HuggingFace model
```bash
scomp-link llm finetune --model <hf_id> --data PATH
  [--method {lora,qlora,full}]   # Training method (default: lora)
  [--epochs N]                   # Training epochs (default: 3)
  [--batch-size N]               # Batch size (default: 4)
  [--learning-rate F]            # Learning rate (default: 2e-4)
  [--lora-r N]                   # LoRA rank (default: 16)
  [--lora-alpha N]               # LoRA alpha (default: 32)
  [--max-seq-length N]           # Max sequence length (default: 2048)
  [--save-artifact PATH]         # Save as .scomp file
  [--output-dir PATH]            # Output directory
```

### `llm convert` — Convert to GGUF
```bash
scomp-link llm convert --model PATH
  [--quantization LEVEL]         # Q4_K_M, Q8_0, f16, etc.
  [--output PATH]                # Output GGUF file
```

### `llm scratch` — Build transformer from scratch
```bash
scomp-link llm scratch --config PATH.yaml --data PATH
  [--epochs N]                   # Training epochs
  [--batch-size N]               # Batch size
  [--learning-rate F]            # Learning rate
```

### `llm estimate` — Estimate VRAM/disk requirements
```bash
scomp-link llm estimate --model PATH
  [--quantization LEVEL]         # Target quantization level
```

### `llm evaluate` — Evaluate generated text
```bash
scomp-link llm evaluate --generated PATH --references PATH
  [--output PATH]                # Results JSON
```

### `llm dedup` — Deduplicate text corpus
```bash
scomp-link llm dedup --data PATH
  [--method {exact,ngram,minhash}]  # Dedup method
  [--threshold F]                # Similarity threshold (default: 0.8)
  [--output PATH]                # Deduplicated output
```

### `llm merge` — Merge models
```bash
scomp-link llm merge --base PATH --models M1,M2[,M3...]
  [--method {linear,slerp,ties,dare}]  # Merge strategy
  [--weights W1,W2[,W3...]]     # Per-model weights
  [--output PATH]                # Output directory
```

### `llm serve` — Serve model as REST API
```bash
scomp-link llm serve --model PATH
  [--port N]                     # Port (default: 8080)
  [--host ADDR]                  # Bind address (default: 127.0.0.1)
  [--load-in-4bit]               # Load with 4-bit quantization
```

### `llm format` — Convert dataset formats
```bash
scomp-link llm format --input PATH --output PATH
  [--source {alpaca,sharegpt,openai}]  # Source format
  [--target {chatml,llama,plain}]      # Target format
```

# Visualization Guide

scomp-link provides 41 chart types across three engines plus an HTML report builder.

## HTML Report Builder Pattern

```python
from scomp_link.utils.report_html import ScompLinkHTMLReport

# Initialize (Highcharts/Plotly JS included automatically)
report = ScompLinkHTMLReport(
    title='Report Title',
    main_color='#6E37FA',    # Optional: primary theme color
    light_color='#9682FF',   # Optional: light variant
    dark_color='#4614B4',    # Optional: dark variant
)

# Build content
report.open_section("Section Name")       # Collapsible section start
report.add_title("Heading")               # <h2> heading
report.add_text("Paragraph text")         # <p> paragraph
report.add_dataframe(df, "Table Name")    # Styled HTML table with CSV download
report.add_graph_to_report(fig, "Title")  # Plotly figure (interactive)
report.add_matplotlib_graph_to_report(fig, "Title")  # Matplotlib (static image)
report.add_rawgraphs_to_report(svg, "Title")  # RAWGraphs SVG chart
report.add_image_to_report("path.png", "Title")  # Local image
report.add_many_plots_with_selection_box_to_report(figs_dict, "Title")  # Combobox selector
report.html_report += html_string         # Direct HTML append (for Highcharts)
report.close_section()                    # Collapsible section end

# Save
report.save_html('output.html')           # Self-contained HTML file
report.save_pdf('output.pdf')             # PDF via Playwright (headless Chrome)
```

## Plotly Utils (5 functions)

```python
from scomp_link.utils.plotly_utils import histogram, multiple_histograms, barchart, linechart, area_chart
```

### `histogram(values, name, h=600)`
- `values`: array-like of floats
- `name`: str — title/axis label
- `h`: int — height in pixels
- Returns: plotly.graph_objects.Figure

### `multiple_histograms(variable_float_for_distribution, ...)`
- Multiple distributions overlaid

### `barchart(categories, metric_values_list, x_axis_title, y_axis_titles=None, order='asc', ...)`
- `categories`: list of str — x-axis labels
- `metric_values_list`: list of float — bar heights
- `order`: 'asc' | 'desc' | None
- `metric_values_line_list`: optional secondary y-axis (line overlay)
- `percentage_y`: bool — format y as percentage
- Returns: plotly.graph_objects.Figure

### `linechart(date_list, lines, title_text, x_label, y_labels, format_date="%Y-%m-%d")`
- `date_list`: list of date strings
- `lines`: dict `{series_name: [values]}`
- Returns: plotly.graph_objects.Figure

### `area_chart(date_list, lines, title_text, x_label, y_labels, format_date="%Y-%m-%d")`
- Same as linechart but filled area
- Returns: plotly.graph_objects.Figure

## Highcharts (3 functions)

```python
from scomp_link.utils.highcharts import streamgraphs, calendar_heatmap, calendar_gantt
```

### `streamgraphs(title, dates, series_dict, annotation=None, area=True)`
- `title`: str
- `dates`: list of str (x-axis categories, e.g. ['2024-01', '2024-02', ...])
- `series_dict`: dict `{series_name: [int_values]}` — each value list same length as dates
- `annotation`: dict `{label: int_index}` — annotations at specific x positions (optional)
- `area`: bool — True=stacked area, False=symmetric streamgraph
- Returns: HTML string (append to `report.html_report`)

### `calendar_heatmap(title, series_dict, min=0, max=1)`
- `title`: str
- `series_dict`: dict `{"yyyy-mm-dd": float_value}` — daily values (typically 28-42 days)
- `min`: float — color scale minimum
- `max`: float — color scale maximum
- Returns: HTML string

### `calendar_gantt(title, series_dict, min_date, max_date, colors=None)`
- `title`: str
- `series_dict`: list of dicts with structure:
  ```python
  [
      {
          'name': 'Phase Name',
          'data': [
              {'name': 'Task', 'id': 'task1',
               'start': "Date.UTC(2025, 5, 1)",  # JS Date.UTC format
               'end': "Date.UTC(2025, 5, 14)",
               'completed': "{ amount: 0.8 }"},  # Optional progress
              {'name': 'Milestone', 'id': 'ms1',
               'start': "Date.UTC(2025, 5, 14)",
               'end': "Date.UTC(2025, 5, 14)",
               'milestone': 'true'},
          ]
      }
  ]
  ```
- `min_date`: str "yyyy-mm-dd"
- `max_date`: str "yyyy-mm-dd"
- Returns: HTML string

## RAWGraphs SVG Charts (31 functions)

All RAWGraphs functions return SVG strings. Embed with `report.add_rawgraphs_to_report(svg, title)`.

```python
from scomp_link.utils.rawgraphs import (
    # Comparisons
    barchart, barchartmultiset, barchartstacked, piechart, radarchart, voronoidiagram,
    # Distributions
    beeswarm, boxplot, violinplot,
    # Time Series
    bumpchart, gantt_chart, horizongraph, linechart, slopechart, streamgraph,
    # Correlations
    bubblechart, contour_plot, convex_hull, hexagonal_binning, matrixplot, parallelcoordinates,
    # Hierarchies
    circlepacking, circular_dendrogram, dendrogram, sunburst, treemap, voronoi_treemap,
    # Networks
    alluvial_diagram, arc_diagram, chord_diagram, sankey_diagram,
)
```

### Comparisons

**`barchart(categories, values, title, width=800, height=500)`**
- `categories`: list[str]
- `values`: list[float]

**`barchartmultiset(categories, series_dict, title, width=800, height=500)`**
- `series_dict`: dict `{series_name: [values]}`

**`barchartstacked(categories, series_dict, title, width=800, height=500)`**
- Same as multiset but stacked

**`piechart(categories, values, title, width=500, height=500)`**

**`radarchart(categories, series_dict, title, width=600, height=600)`**
- `series_dict`: dict `{series_name: [values]}` — values on same scale per category

**`voronoidiagram(points, labels, title, width=800, height=600)`**
- `points`: list of (x, y) tuples
- `labels`: list[str]

### Distributions

**`beeswarm(groups, values, title, width=800, height=400)`**
- `groups`: list[str] — group per point
- `values`: list[float] — value per point

**`boxplot(groups_dict, title, width=800, height=400)`**
- `groups_dict`: dict `{group_name: [values]}`

**`violinplot(groups_dict, title, width=800, height=400)`**
- Same as boxplot

### Time Series

**`bumpchart(time_points, rankings_dict, title, width=900, height=500)`**
- `time_points`: list[str] — x-axis labels
- `rankings_dict`: dict `{entity: [rank_per_timepoint]}`

**`gantt_chart(tasks, title, width=900, height=400)`**
- `tasks`: list of `{'name': str, 'start': float, 'end': float, 'group': str}`

**`horizongraph(dates, series_dict, title, width=900, height=300)`**
- `series_dict`: dict `{series_name: [values]}`

**`linechart(dates, series_dict, title, width=900, height=400)`**
- `dates`: list[str]
- `series_dict`: dict `{series_name: [values]}`

**`slopechart(labels, start_values, end_values, title, start_label, end_label, width=600, height=500)`**

**`streamgraph(dates, series_dict, title, width=900, height=400)`**

### Correlations

**`bubblechart(x, y, sizes, labels, title, width=800, height=600)`**
- `x`, `y`: list[float]
- `sizes`: list[float] — bubble radius
- `labels`: list[str]

**`contour_plot(x, y, title, width=800, height=600, n_levels=10)`**

**`convex_hull(groups_dict, title, width=800, height=600)`**
- `groups_dict`: dict `{group: [(x,y), ...]}`

**`hexagonal_binning(x, y, title, width=800, height=600, gridsize=20)`**

**`matrixplot(matrix, row_labels, col_labels, title, width=700, height=700)`**
- `matrix`: 2D list or numpy array

**`parallelcoordinates(df, group_col, title, width=900, height=400)`**
- `df`: pandas DataFrame
- `group_col`: str — column to color by

### Hierarchies

**`circlepacking(labels, parents, values, title, width=700, height=700)`**
- `labels`: list[str] — node names
- `parents`: list[str] — parent of each node ("" for root)
- `values`: list[float] — size of each node

**`circular_dendrogram(labels, parents, title, width=700, height=700)`**

**`dendrogram(labels, parents, title, width=900, height=500)`**

**`sunburst(labels, parents, values, title, width=700, height=700)`**

**`treemap(labels, parents, values, title, width=900, height=600)`**

**`voronoi_treemap(labels, parents, values, title, width=700, height=700)`**

### Networks

**`alluvial_diagram(flows, title, width=900, height=500)`**
- `flows`: list of `{'source': str, 'target': str, 'value': float}`

**`arc_diagram(nodes, links, title, width=900, height=400)`**
- `nodes`: list[str]
- `links`: list of `{'source': int, 'target': int, 'value': float}`

**`chord_diagram(matrix, labels, title, width=700, height=700)`**
- `matrix`: square 2D array (flow between nodes)
- `labels`: list[str]

**`sankey_diagram(nodes, links, title, width=900, height=500)`**
- `nodes`: list[str]
- `links`: list of `{'source': int, 'target': int, 'value': float}`

## Chart Selection Guide

| I have... | Use this chart |
|-----------|---------------|
| Categories + values | `barchart`, `piechart` |
| Categories + multiple series | `barchartmultiset`, `barchartstacked`, `radarchart` |
| Numeric distribution | `histogram` (Plotly), `boxplot`, `violinplot`, `beeswarm` |
| Time series (single) | `linechart`, `area_chart` (Plotly) |
| Time series (multi-series) | `streamgraphs` (Highcharts), `streamgraph` (RAWGraphs) |
| Rankings over time | `bumpchart`, `slopechart` |
| Project timeline | `calendar_gantt` (Highcharts), `gantt_chart` (RAWGraphs) |
| Daily values (calendar) | `calendar_heatmap` (Highcharts) |
| Two numeric variables | `bubblechart`, `contour_plot`, `hexagonal_binning` |
| Correlation matrix | `matrixplot` |
| Multi-dimensional | `parallelcoordinates` |
| Hierarchical data | `treemap`, `sunburst`, `circlepacking`, `dendrogram` |
| Flow/connections | `sankey_diagram`, `alluvial_diagram`, `chord_diagram`, `arc_diagram` |
| Grouped points | `convex_hull`, `voronoidiagram` |

## Color System

```python
from scomp_link.utils.colors import PRIMARY, LIGHT, MEDIUM, DARK, MAIN, MAIN_LIGHT, MAIN_DARK

# PRIMARY = 10 distinct colors for categorical data
# ["#6E37FA", "#32BBB9", "#FF9408", "#F40953", "#FA32A0",
#  "#B30095", "#FFD500", "#AAF564", "#50E6AA", "#2765F0"]
```

Use these when creating custom Plotly figures for visual consistency with scomp-link reports.

# Workflow Patterns

## Pattern 1: CSV → Trained Model → Deploy

```bash
# Understand the data
scomp-link describe --data customers.csv --format table

# Engineer features
scomp-link engineer --data customers.csv --target churn --interactions --log-transform --output features.csv

# Tune and save best model
scomp-link tune --data features.csv --target churn --task classification --method optuna --n-trials 100 --save-artifact churn_model.scomp

# Validate
scomp-link validate --artifact churn_model.scomp --data test.csv --target churn --report validation.html

# Deploy
scomp-link serve --artifact churn_model.scomp --port 8080
```

**Prediction from deployed model:**
```bash
curl -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [{"age": 35, "income": 50000, "tenure": 24}]}'
```

## Pattern 2: Monitor Production Data

```bash
# Save reference data during training
cp train.csv reference_data.csv

# Weekly monitoring job
scomp-link monitor --reference reference_data.csv --current this_week.csv \
  --artifact model.scomp --target y --output weekly_report.html

# Quick drift check (no model needed)
scomp-link drift --reference reference_data.csv --current this_week.csv --plot drift.html

# Anomaly scan on incoming data
scomp-link anomaly --data this_week.csv --methods iforest,lof --contamination 0.03 --output anomalies.csv
```

## Pattern 3: Create Analytical Dashboard (Python)

```python
from scomp_link.utils.report_html import ScompLinkHTMLReport
from scomp_link.utils.plotly_utils import histogram, barchart, linechart, area_chart
from scomp_link.utils.highcharts import streamgraphs, calendar_heatmap
from scomp_link.utils.rawgraphs import treemap, sankey_diagram, sunburst
import pandas as pd

df = pd.read_csv('sales_data.csv')

report = ScompLinkHTMLReport(title='Sales Dashboard Q4 2025')

# Section 1: KPIs
report.open_section("Key Metrics")
kpi_df = pd.DataFrame([{
    'Total Revenue': f"${df['revenue'].sum():,.0f}",
    'Avg Order': f"${df['revenue'].mean():,.2f}",
    'Customers': df['customer_id'].nunique(),
}])
report.add_dataframe(kpi_df, "KPIs")
report.close_section()

# Section 2: Trends (Highcharts streamgraph)
report.open_section("Revenue by Category Over Time")
monthly = df.groupby(['month', 'category'])['revenue'].sum().unstack(fill_value=0)
dates = monthly.index.tolist()
series = {col: monthly[col].tolist() for col in monthly.columns}
html_stream = streamgraphs("Revenue by Category", dates, series, area=True)
report.html_report += html_stream
report.close_section()

# Section 3: Distribution (Plotly)
report.open_section("Order Value Distribution")
fig = histogram(df['revenue'].values, "Order Value ($)")
report.add_graph_to_report(fig, "Revenue Distribution")
report.close_section()

# Section 4: Hierarchy (RAWGraphs)
report.open_section("Revenue Breakdown")
cat_rev = df.groupby('category')['revenue'].sum()
svg = treemap(cat_rev.index.tolist(), [''] * len(cat_rev), cat_rev.values.tolist(), "Revenue Treemap")
report.add_rawgraphs_to_report(svg, "Treemap")
report.close_section()

# Section 5: Flow (RAWGraphs)
report.open_section("Customer Journey")
flows = [
    {'source': 'Homepage', 'target': 'Product', 'value': 1000},
    {'source': 'Product', 'target': 'Cart', 'value': 400},
    {'source': 'Cart', 'target': 'Checkout', 'value': 300},
    {'source': 'Checkout', 'target': 'Purchase', 'value': 250},
]
svg_sankey = sankey_diagram(
    ['Homepage', 'Product', 'Cart', 'Checkout', 'Purchase'],
    [{'source': f['source'], 'target': f['target'], 'value': f['value']} for f in flows],
    "Customer Flow"
)
report.add_rawgraphs_to_report(svg_sankey, "Sankey")
report.close_section()

report.save_html('sales_dashboard.html')
```

## Pattern 4: Compare Multiple Approaches

```bash
# Approach 1: Simple model
scomp-link run --data train.csv --target y --task regression --save-artifact simple.scomp --silent

# Approach 2: With feature engineering
scomp-link engineer --data train.csv --target y --interactions --log-transform --output eng.csv
scomp-link run --data eng.csv --target y --task regression --save-artifact engineered.scomp --silent

# Approach 3: Tuned
scomp-link tune --data eng.csv --target y --task regression --method optuna --n-trials 50 --save-artifact tuned.scomp --silent

# Compare all three
scomp-link compare --artifacts simple.scomp engineered.scomp tuned.scomp --plot comparison.html
```

## Pattern 5: Text Classification Pipeline

```bash
# Profile text data
scomp-link describe --data tickets.csv

# Quick TF-IDF approach (fast)
scomp-link text --data tickets.csv --text-col message --target category --method tfidf --save-artifact tfidf_model.scomp

# Validate
scomp-link validate --artifact tfidf_model.scomp --data test_tickets.csv --target category --format table
```

## Pattern 6: YAML-Driven Pipeline

```yaml
# pipeline.yaml — full automated workflow
data: data/train.csv
target: price
task: regression
name: house_price_predictor

engineer:
  interactions: true
  log_transform: true
  date_features: true
  target_encode: true

tune:
  method: optuna
  n_trials: 100

validate:
  test_data: data/test.csv
  report: reports/validation.html

save: models/house_price_v2.scomp
```

```bash
scomp-link pipeline --config pipeline.yaml
```

## Troubleshooting

### Model performs poorly
1. Check data quality: `scomp-link quality --data train.csv`
2. Look for drift: `scomp-link drift --reference train.csv --current test.csv`
3. Try feature engineering: `scomp-link engineer --data train.csv --target y --interactions --log-transform`
4. Tune more: increase `--n-trials`
5. Check fairness: `scomp-link fairness --data preds.csv --target y_true --predicted y_pred --sensitive group`

### Command fails with error
- `unsupported file format` → use .csv, .tsv, or .parquet files
- `target column not found` → check column name with `scomp-link describe --data file.csv`
- `artifact not found` → use absolute path or check working directory
- `could not convert string to float` → categoricals need encoding, add `--engineer` flag
- `Memory error` → reduce dataset size or use `--methods iforest,lof` (skip deep learning methods)

### Report is empty or broken
- Check that `open_section()` has matching `close_section()`
- Highcharts content must be appended with `report.html_report += html_string`
- Plotly figures must be passed to `add_graph_to_report(fig, title)`
- RAWGraphs SVGs go to `add_rawgraphs_to_report(svg, title)`