Statistical Inference
Multiple Testing in Strategy Search
- Published
- 14 SEPT 2026
- Reading
- 1 MIN
- Status
- DEMONSTRATION ENTRY
Why the best backtest in a large search is biased upward, and how to reason about it before looking at a single equity curve.
DEMONSTRATION ENTRY — placeholder text written to show the layout. It contains no verified research results.
The problem
When many candidate strategies are evaluated on the same data, the maximum observed performance statistic grows with the number of candidates even when none of them has any edge. The selected winner is a draw from the right tail of a null distribution.
A framing
Let each candidate's measured Sharpe ratio under the null be approximately normal. The expected maximum of N independent draws grows slowly but without bound. The distribution below is synthetic; drag the parameters to see how the tail moves.
What to record
- The number of configurations tried, including the ones discarded early.
- The dependence structure between candidates.
- The sample length and any structural breaks.
Open questions
This entry is a stub. Replace it with your own derivation, references and code.
- P(X > k | null)
- 2.28%
- E[max of N null draws] ≈
- 2.06
- Best-of-N vs. true μ
- Selection bias exceeds signal
