backtesting
Use when testing a trading strategy on historical data. Covers the biases that make a backtest lie, realistic costs, walk-forward validation, and the statistics that distinguish an edge from a coincidence. Trigger on "backtest this strategy", "my backtest looks too good to be tru
Install
npx skills add https://github.com/nimadorostkar/Claude-Skills-collection/tree/main/skills/finance/backtesting
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install nimadorostkar-claude-skills-collection@llmmart
git clone https://github.com/nimadorostkar/Claude-Skills-collection.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole nimadorostkar/claude-skills-collection collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Backtesting
Purpose
Test a strategy honestly. The default outcome of a backtest is a beautiful equity curve that describes the past and predicts nothing — and every bias in the process pushes in that direction.
When to Use
- Testing a strategy before risking capital.
- Validating that an apparent edge is real.
- Diagnosing why a live strategy underperforms its backtest.
- Reviewing someone else's backtest.
Capabilities
- Bias elimination: survivorship, look-ahead, selection.
- Realistic cost modeling: commissions, spread, slippage, borrow.
- Walk-forward and out-of-sample validation.
- Statistical significance and multiple-testing correction.
- Robustness testing.
Inputs
- The strategy rules, stated precisely enough to be executed mechanically.
- Point-in-time historical data, including delisted instruments.
- Realistic assumptions about execution.
Outputs
- Performance metrics with costs included.
- Out-of-sample results, reported honestly.
- A judgment on whether the edge is real, with the evidence.
Workflow
- Eliminate the biases before you run anything — Survivorship (delisted names must be in the universe), look-ahead (data must be used only when it was available), and selection (the strategy must be specified before it is tested).
- Model the costs honestly — Commission, the bid-ask spread, slippage that scales with size and inversely with liquidity, and borrow costs on shorts. Most published edges are smaller than realistic costs.
- Split the data before you look at it — In-sample for development, out-of-sample for validation, and a final hold-out you touch exactly once.
- Walk forward — Re-fit on a rolling window and test on the period immediately after. A single in-sample fit tells you nothing about a strategy that will be run forward in time.
- Correct for multiple testing — If you tested forty variants, the best one looks good by chance. A p-value that has not been adjusted for the number of things you tried is not a p-value.
- Stress it — Perturb the parameters. A strategy that only works with a 14-day lookback and fails with 13 or 15 is fitted to noise.
Best Practices
- The most common backtesting error is look-ahead bias, and it is usually accidental: using a closing price to make a decision that would have been made during the day, or using a fundamental figure on the fiscal date rather than the filing date.
- Transaction costs kill more strategies than any other factor. A strategy trading daily with a 0.05% edge per trade is a losing strategy after a 0.10% round-trip cost, and it will have looked excellent before costs.
- A Sharpe ratio above 2 on a simple strategy in a liquid market is a bug, not a discovery. Look for the bias.
- Parameter sensitivity is the honesty test. If performance collapses when a parameter moves by one unit, the strategy has fitted noise. Plot the surface.
- The out-of-sample period must be genuinely untouched. If you look at it, adjust, and look again, it is now in-sample and you have no validation left.
- Report the strategy's worst period, not just its average. The maximum drawdown and its duration determine whether it is survivable in practice.
Examples
A backtest that does not lie to itself:
def backtest(strategy: Strategy, start: date, end: date) -> BacktestResult:
equity, trades = [], []
for day in trading_days(start, end):
# Point-in-time universe: includes everything that was tradable THAT DAY,
# including companies that were later delisted or went to zero.
universe = universe_as_of(day)
# Point-in-time fundamentals: as reported, on or before this date. Not
# as later restated, and never before the filing date.
data = fundamentals_as_of(day, universe)
# Signals are computed on data available at the CLOSE of the prior day.
# Trades execute at the NEXT day's open. Using today's close to decide
# and today's close to fill is the classic look-ahead error.
signals = strategy.signals(data.shift(1))
for signal in signals:
fill = next_open(signal.symbol, day)
cost = transaction_cost(
price=fill,
shares=signal.shares,
adv=data.loc[signal.symbol, "adv_20d"],
)
trades.append(Trade(day, signal.symbol, fill, cost))
equity.append(mark_to_market(day))
return BacktestResult(equity=equity, trades=trades)
def transaction_cost(price: float, shares: int, adv: float) -> float:
"""Commission + half the spread + slippage that scales with participation."""
notional = price * shares
commission = max(1.0, shares * 0.005)
spread_cost = notional * 0.0005 # half-spread, liquid names
# Market impact grows with the fraction of daily volume you consume.
participation = (shares * price) / max(adv, 1)
impact = notional * 0.10 * math.sqrt(participation) # square-root impact model
return commission + spread_cost + impact
The result, reported honestly:
Strategy: momentum + quality, 25 names, monthly rebalance.
In-sample Out-of-sample Difference
2010-2018 2019-2024
CAGR (gross) 18.4% 11.2%
CAGR (net of costs) 14.1% 6.8% <- costs take 4-5 points
Sharpe (net) 1.12 0.54 <- halved out of sample
Max drawdown -22.4% -31.7%
Longest drawdown 8 months 19 months <- the number that matters
Multiple testing: 34 variants were tested. The reported result is the best of
34. Adjusting for that (Bonferroni), the out-of-sample Sharpe of 0.54 is not
distinguishable from zero at the 5% level.
Parameter sensitivity: performance is stable for lookbacks of 100-150 days
(Sharpe 0.48-0.57) and collapses below 80 days. That stability is mildly
reassuring; the multiple-testing problem is not.
Judgment: the edge, if it exists, is much smaller than the in-sample result
suggests and is not statistically established. A 19-month drawdown is longer
than most people can actually sit through. Not deployable at meaningful size
on this evidence.
Notes
- The gap between in-sample and out-of-sample performance in the example — a Sharpe of 1.12 falling to 0.54 — is entirely typical, and it is the reason out-of-sample testing is not optional.
- The square-root market-impact model is a standard approximation: impact scales with the square root of participation rate. It matters enormously for any strategy trading size, and it is usually omitted from backtests that look excellent.
- This is educational material about backtesting methodology, not financial advice, and no backtest result constitutes a recommendation.
Files (claude-skills-collection)
-
SKILL.md 7.3 KB
--- name: backtesting description: Use when testing a trading strategy on historical data. Covers the biases that make a backtest lie, realistic costs, walk-forward validation, and the statistics that distinguish an edge from a coincidence. Trigger on "backtest this strategy", "my backtest looks too good to be true", "why does live trading underperform my backtest". Do not use for testing whether a single signal is statistically significant — see the quantitative-analysis skill. metadata: category: finance version: 1.0.0 tags: [backtesting, validation, overfitting, bias, statistics] --- # Backtesting ## Purpose Test a strategy honestly. The default outcome of a backtest is a beautiful equity curve that describes the past and predicts nothing — and every bias in the process pushes in that direction. ## When to Use - Testing a strategy before risking capital. - Validating that an apparent edge is real. - Diagnosing why a live strategy underperforms its backtest. - Reviewing someone else's backtest. ## Capabilities - Bias elimination: survivorship, look-ahead, selection. - Realistic cost modeling: commissions, spread, slippage, borrow. - Walk-forward and out-of-sample validation. - Statistical significance and multiple-testing correction. - Robustness testing. ## Inputs - The strategy rules, stated precisely enough to be executed mechanically. - Point-in-time historical data, including delisted instruments. - Realistic assumptions about execution. ## Outputs - Performance metrics with costs included. - Out-of-sample results, reported honestly. - A judgment on whether the edge is real, with the evidence. ## Workflow 1. **Eliminate the biases before you run anything** — Survivorship (delisted names must be in the universe), look-ahead (data must be used only when it was available), and selection (the strategy must be specified before it is tested). 2. **Model the costs honestly** — Commission, the bid-ask spread, slippage that scales with size and inversely with liquidity, and borrow costs on shorts. Most published edges are smaller than realistic costs. 3. **Split the data before you look at it** — In-sample for development, out-of-sample for validation, and a final hold-out you touch exactly once. 4. **Walk forward** — Re-fit on a rolling window and test on the period immediately after. A single in-sample fit tells you nothing about a strategy that will be run forward in time. 5. **Correct for multiple testing** — If you tested forty variants, the best one looks good by chance. A p-value that has not been adjusted for the number of things you tried is not a p-value. 6. **Stress it** — Perturb the parameters. A strategy that only works with a 14-day lookback and fails with 13 or 15 is fitted to noise. ## Best Practices - The most common backtesting error is look-ahead bias, and it is usually accidental: using a closing price to make a decision that would have been made during the day, or using a fundamental figure on the fiscal date rather than the filing date. - Transaction costs kill more strategies than any other factor. A strategy trading daily with a 0.05% edge per trade is a losing strategy after a 0.10% round-trip cost, and it will have looked excellent before costs. - A Sharpe ratio above 2 on a simple strategy in a liquid market is a bug, not a discovery. Look for the bias. - Parameter sensitivity is the honesty test. If performance collapses when a parameter moves by one unit, the strategy has fitted noise. Plot the surface. - The out-of-sample period must be genuinely untouched. If you look at it, adjust, and look again, it is now in-sample and you have no validation left. - Report the strategy's worst period, not just its average. The maximum drawdown and its duration determine whether it is survivable in practice. ## Examples **A backtest that does not lie to itself:** ```python def backtest(strategy: Strategy, start: date, end: date) -> BacktestResult: equity, trades = [], [] for day in trading_days(start, end): # Point-in-time universe: includes everything that was tradable THAT DAY, # including companies that were later delisted or went to zero. universe = universe_as_of(day) # Point-in-time fundamentals: as reported, on or before this date. Not # as later restated, and never before the filing date. data = fundamentals_as_of(day, universe) # Signals are computed on data available at the CLOSE of the prior day. # Trades execute at the NEXT day's open. Using today's close to decide # and today's close to fill is the classic look-ahead error. signals = strategy.signals(data.shift(1)) for signal in signals: fill = next_open(signal.symbol, day) cost = transaction_cost( price=fill, shares=signal.shares, adv=data.loc[signal.symbol, "adv_20d"], ) trades.append(Trade(day, signal.symbol, fill, cost)) equity.append(mark_to_market(day)) return BacktestResult(equity=equity, trades=trades) def transaction_cost(price: float, shares: int, adv: float) -> float: """Commission + half the spread + slippage that scales with participation.""" notional = price * shares commission = max(1.0, shares * 0.005) spread_cost = notional * 0.0005 # half-spread, liquid names # Market impact grows with the fraction of daily volume you consume. participation = (shares * price) / max(adv, 1) impact = notional * 0.10 * math.sqrt(participation) # square-root impact model return commission + spread_cost + impact ``` **The result, reported honestly:** ```text Strategy: momentum + quality, 25 names, monthly rebalance. In-sample Out-of-sample Difference 2010-2018 2019-2024 CAGR (gross) 18.4% 11.2% CAGR (net of costs) 14.1% 6.8% <- costs take 4-5 points Sharpe (net) 1.12 0.54 <- halved out of sample Max drawdown -22.4% -31.7% Longest drawdown 8 months 19 months <- the number that matters Multiple testing: 34 variants were tested. The reported result is the best of 34. Adjusting for that (Bonferroni), the out-of-sample Sharpe of 0.54 is not distinguishable from zero at the 5% level. Parameter sensitivity: performance is stable for lookbacks of 100-150 days (Sharpe 0.48-0.57) and collapses below 80 days. That stability is mildly reassuring; the multiple-testing problem is not. Judgment: the edge, if it exists, is much smaller than the in-sample result suggests and is not statistically established. A 19-month drawdown is longer than most people can actually sit through. Not deployable at meaningful size on this evidence. ``` ## Notes - The gap between in-sample and out-of-sample performance in the example — a Sharpe of 1.12 falling to 0.54 — is entirely typical, and it is the reason out-of-sample testing is not optional. - The square-root market-impact model is a standard approximation: impact scales with the square root of participation rate. It matters enormously for any strategy trading size, and it is usually omitted from backtests that look excellent. - This is educational material about backtesting methodology, not financial advice, and no backtest result constitutes a recommendation.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.