{"slug":"backtesting","title":"backtesting","summary":"Use when testing a trading strategy on historical data. Covers the biases that make a backtest lie, realistic costs, walk-forward validation, and the statistics that distinguish an edge from a coincidence. Trigger on \"backtest this strategy\", \"my backtest looks too good to be tru","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-11-09T16:53:17.111155Z","repo":{"url":"https://github.com/nimadorostkar/Claude-Skills-collection","stars":34,"forks":4,"license":"MIT","updatedAt":"2026-11-08T15:41:45Z"},"bodyHtml":"<hr>\n<h2>name: backtesting\ndescription: Use when testing a trading strategy on historical data. Covers the biases that make a backtest lie, realistic costs, walk-forward validation, and the statistics that distinguish an edge from a coincidence. Trigger on \"backtest this strategy\", \"my backtest looks too good to be true\", \"why does live trading underperform my backtest\". Do not use for testing whether a single signal is statistically significant — see the quantitative-analysis skill.\nmetadata:\ncategory: finance\nversion: 1.0.0\ntags: [backtesting, validation, overfitting, bias, statistics]</h2>\n<h1>Backtesting</h1>\n<h2>Purpose</h2>\n<p>Test a strategy honestly. The default outcome of a backtest is a beautiful equity curve that describes the past and predicts nothing — and every bias in the process pushes in that direction.</p>\n<h2>When to Use</h2>\n<ul>\n<li>Testing a strategy before risking capital.</li>\n<li>Validating that an apparent edge is real.</li>\n<li>Diagnosing why a live strategy underperforms its backtest.</li>\n<li>Reviewing someone else's backtest.</li>\n</ul>\n<h2>Capabilities</h2>\n<ul>\n<li>Bias elimination: survivorship, look-ahead, selection.</li>\n<li>Realistic cost modeling: commissions, spread, slippage, borrow.</li>\n<li>Walk-forward and out-of-sample validation.</li>\n<li>Statistical significance and multiple-testing correction.</li>\n<li>Robustness testing.</li>\n</ul>\n<h2>Inputs</h2>\n<ul>\n<li>The strategy rules, stated precisely enough to be executed mechanically.</li>\n<li>Point-in-time historical data, including delisted instruments.</li>\n<li>Realistic assumptions about execution.</li>\n</ul>\n<h2>Outputs</h2>\n<ul>\n<li>Performance metrics with costs included.</li>\n<li>Out-of-sample results, reported honestly.</li>\n<li>A judgment on whether the edge is real, with the evidence.</li>\n</ul>\n<h2>Workflow</h2>\n<ol>\n<li><strong>Eliminate the biases before you run anything</strong> — Survivorship (delisted names must be in the universe), look-ahead (data must be used only when it was available), and selection (the strategy must be specified before it is tested).</li>\n<li><strong>Model the costs honestly</strong> — Commission, the bid-ask spread, slippage that scales with size and inversely with liquidity, and borrow costs on shorts. Most published edges are smaller than realistic costs.</li>\n<li><strong>Split the data before you look at it</strong> — In-sample for development, out-of-sample for validation, and a final hold-out you touch exactly once.</li>\n<li><strong>Walk forward</strong> — Re-fit on a rolling window and test on the period immediately after. A single in-sample fit tells you nothing about a strategy that will be run forward in time.</li>\n<li><strong>Correct for multiple testing</strong> — If you tested forty variants, the best one looks good by chance. A p-value that has not been adjusted for the number of things you tried is not a p-value.</li>\n<li><strong>Stress it</strong> — Perturb the parameters. A strategy that only works with a 14-day lookback and fails with 13 or 15 is fitted to noise.</li>\n</ol>\n<h2>Best Practices</h2>\n<ul>\n<li>The most common backtesting error is look-ahead bias, and it is usually accidental: using a closing price to make a decision that would have been made during the day, or using a fundamental figure on the fiscal date rather than the filing date.</li>\n<li>Transaction costs kill more strategies than any other factor. A strategy trading daily with a 0.05% edge per trade is a losing strategy after a 0.10% round-trip cost, and it will have looked excellent before costs.</li>\n<li>A Sharpe ratio above 2 on a simple strategy in a liquid market is a bug, not a discovery. Look for the bias.</li>\n<li>Parameter sensitivity is the honesty test. If performance collapses when a parameter moves by one unit, the strategy has fitted noise. Plot the surface.</li>\n<li>The out-of-sample period must be genuinely untouched. If you look at it, adjust, and look again, it is now in-sample and you have no validation left.</li>\n<li>Report the strategy's worst period, not just its average. The maximum drawdown and its duration determine whether it is survivable in practice.</li>\n</ul>\n<h2>Examples</h2>\n<p><strong>A backtest that does not lie to itself:</strong></p>\n<pre><code>def backtest(strategy: Strategy, start: date, end: date) -&gt; BacktestResult:\n    equity, trades = [], []\n\n    for day in trading_days(start, end):\n        # Point-in-time universe: includes everything that was tradable THAT DAY,\n        # including companies that were later delisted or went to zero.\n        universe = universe_as_of(day)\n\n        # Point-in-time fundamentals: as reported, on or before this date. Not\n        # as later restated, and never before the filing date.\n        data = fundamentals_as_of(day, universe)\n\n        # Signals are computed on data available at the CLOSE of the prior day.\n        # Trades execute at the NEXT day's open. Using today's close to decide\n        # and today's close to fill is the classic look-ahead error.\n        signals = strategy.signals(data.shift(1))\n\n        for signal in signals:\n            fill = next_open(signal.symbol, day)\n            cost = transaction_cost(\n                price=fill,\n                shares=signal.shares,\n                adv=data.loc[signal.symbol, \"adv_20d\"],\n            )\n            trades.append(Trade(day, signal.symbol, fill, cost))\n\n        equity.append(mark_to_market(day))\n\n    return BacktestResult(equity=equity, trades=trades)\n\n\ndef transaction_cost(price: float, shares: int, adv: float) -&gt; float:\n    \"\"\"Commission + half the spread + slippage that scales with participation.\"\"\"\n    notional = price * shares\n    commission = max(1.0, shares * 0.005)\n    spread_cost = notional * 0.0005                       # half-spread, liquid names\n\n    # Market impact grows with the fraction of daily volume you consume.\n    participation = (shares * price) / max(adv, 1)\n    impact = notional * 0.10 * math.sqrt(participation)   # square-root impact model\n\n    return commission + spread_cost + impact\n</code></pre>\n<p><strong>The result, reported honestly:</strong></p>\n<pre><code>Strategy: momentum + quality, 25 names, monthly rebalance.\n\n                        In-sample    Out-of-sample   Difference\n                        2010-2018    2019-2024\nCAGR (gross)              18.4%          11.2%\nCAGR (net of costs)       14.1%           6.8%        &lt;- costs take 4-5 points\nSharpe (net)               1.12           0.54        &lt;- halved out of sample\nMax drawdown             -22.4%         -31.7%\nLongest drawdown          8 months      19 months     &lt;- the number that matters\n\nMultiple testing: 34 variants were tested. The reported result is the best of\n34. Adjusting for that (Bonferroni), the out-of-sample Sharpe of 0.54 is not\ndistinguishable from zero at the 5% level.\n\nParameter sensitivity: performance is stable for lookbacks of 100-150 days\n(Sharpe 0.48-0.57) and collapses below 80 days. That stability is mildly\nreassuring; the multiple-testing problem is not.\n\nJudgment: the edge, if it exists, is much smaller than the in-sample result\nsuggests and is not statistically established. A 19-month drawdown is longer\nthan most people can actually sit through. Not deployable at meaningful size\non this evidence.\n</code></pre>\n<h2>Notes</h2>\n<ul>\n<li>The gap between in-sample and out-of-sample performance in the example — a Sharpe of 1.12 falling to 0.54 — is entirely typical, and it is the reason out-of-sample testing is not optional.</li>\n<li>The square-root market-impact model is a standard approximation: impact scales with the square root of participation rate. It matters enormously for any strategy trading size, and it is usually omitted from backtests that look excellent.</li>\n<li>This is educational material about backtesting methodology, not financial advice, and no backtest result constitutes a recommendation.</li>\n</ul>\n","files":[{"path":"SKILL.md","sizeBytes":7482,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-11-09T16:55:14.71669Z","sha256":"5C576A48D1A01E5DA5B96506C6F5D48D9F5E3699E68FF597523D8070E86EEE2A","sizeBytes":3363},"review":null,"source":{"repositoryUrl":"https://github.com/nimadorostkar/Claude-Skills-collection","path":"skills/finance/backtesting","license":"MIT","commit":"1691932edc17063d712a85d02bcc7a1cd3791057","subtreeSha":"91A11E10E353DD9DBFC79F4A913CE4A25EEEB1B8B5A12434AA46D93D9538551E","lastSyncedAt":"2026-11-09T16:52:45.174974Z"},"reviewedAt":"2026-11-09T16:59:34.293357Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/nimadorostkar/Claude-Skills-collection/tree/main/skills/finance/backtesting"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install nimadorostkar-claude-skills-collection@llmmart"},{"target":"git","command":"git clone https://github.com/nimadorostkar/Claude-Skills-collection.git"}]}