Skip to content
All research

Statistical Inference

Multiple Testing in Strategy Search

Published
14 SEPT 2026
Reading
1 MIN
Status
DEMONSTRATION ENTRY

Why the best backtest in a large search is biased upward, and how to reason about it before looking at a single equity curve.

DEMONSTRATION ENTRY — placeholder text written to show the layout. It contains no verified research results.

The problem

When many candidate strategies are evaluated on the same data, the maximum observed performance statistic grows with the number of candidates even when none of them has any edge. The selected winner is a draw from the right tail of a null distribution.

A framing

Let each candidate's measured Sharpe ratio under the null be approximately normal. The expected maximum of N independent draws grows slowly but without bound. The distribution below is synthetic; drag the parameters to see how the tail moves.

What to record

  • The number of configurations tried, including the ones discarded early.
  • The dependence structure between candidates.
  • The sample length and any structural breaks.

Open questions

This entry is a stub. Replace it with your own derivation, references and code.

Fig. 06.2 — Probability explorerSynthetic · Gaussian null vs. alternative
-4-20246kE[max]NULL N(0, σ²)ALT. N(μ, σ²)
P(X > k | null)
2.28%
E[max of N null draws] ≈
2.06
Best-of-N vs. true μ
Selection bias exceeds signal
NextPath Dependence and Drawdown Uncertainty →

For research and educational purposes only. Nothing on this site is investment advice or an offer of any security. Visualizations use synthetic data. Demonstration entries are placeholders and contain no verified results.

© 2026 Q / Research