What backtesting is (and what it is not)
Backtesting is the process of applying a predefined trading rule or decision framework to historical market data to see how it might have performed. The key word is predefined: a backtest is only meaningful when the logic, inputs, and calculation method are set before you run it.
A backtest is not a guarantee of future results. Historical relationships can change, and your results are only as realistic as your assumptions about execution and trading conditions. If your goal is learning, treat backtesting as a measurement exercise: you test a specific hypothesis under specific modeled conditions.
Common mistakes and why they matter
-
Not separating assumptions from the outcome A frequent misunderstanding is to treat the backtest outcome as the market’s truth. In practice, the outcome depends on your assumptions: how fills are modeled, how spreads and commissions are handled, and how orders are executed. If you do not state these assumptions, two people can run “the same” idea and get different results.
-
Overfitting and parameter hunting Overfitting happens when the rule is tuned so tightly to past data that it captures noise rather than repeatable behavior. One sign is improving results on the same dataset repeatedly while losing performance elsewhere. Even without “cheating,” iterative refinement guided by the backtest can turn the process into curve-fitting.
-
Cherry-picking time periods Another mistake is testing only on favorable history. Sometimes this is accidental (for example, using a period that happens to match a specific regime). The consequence is a backtest that may look persuasive, but it reflects a selection effect rather than a broadly applicable effect.
-
Using unrealistic execution models Many backtests simplify fills: they may assume perfect execution at an ideal price, ignore latency, or treat slippage as constant when it is not. This can make results look stronger than they would be under real constraints. Even if the logic is sound, the performance measurement can be off.
-
Ignoring costs and practical constraints Costs include commissions, spreads, and any other consistent trading expenses. Constraints include limits on order placement and whether the rule can actually generate positions at the times the backtest assumes. If these are omitted or approximated loosely, returns may be overstated.
-
Confusing correlation-like outcomes with decision quality Backtesting can produce results that look attractive because of the dataset rather than the rule. For example, a rule that triggers in a trending regime may perform well in sample but fail when conditions shift. A common failure mode is mistaking “a good run” for “a stable decision process.”
Limitations and risks you should expect
The biggest limitation is that backtesting is a model of reality, not reality itself. Execution, liquidity, and market microstructure can differ across time and environments. Additionally, historical data may be incomplete or inconsistent (for example, missing certain intraday behaviors), which can distort measurements.
A second limitation is that performance metrics depend on the test design. Changing the data frequency, the calculation of returns, the order of operations, or the way trades are aligned can alter outcomes. Therefore, even if your backtest code is correct, the conclusion may not be robust.
Finally, results can vary with the specific market conditions and with how costs and execution are represented. Because of this, a single backtest should rarely be treated as a final answer.
Verification checks (neutral and self-contained)
Use neutral checks to confirm whether a backtest result is robust rather than accidental.
- Specify assumptions explicitly: document how you model spreads, slippage, commissions, and order execution.
- Keep parameters fixed for evaluation: only tune on a separate phase, then evaluate without further changes.
- Test across multiple time segments: compare performance in different historical regimes.
- Run stress tests on cost and execution: vary slippage assumptions within plausible ranges to see whether the conclusion changes.
- Use out-of-sample or holdout evaluation: confirm that improvement is not only present in the data used to craft the rule.
If these checks show large instability, that is a valuable outcome: it suggests the strategy logic may be sensitive to modeling choices or market conditions. The “failure” is information about robustness, not a dead end.