Definition: what algorithm testing means
Algorithm testing is the process of evaluating an automated rules-based system using defined data and defined metrics to understand how it would have behaved under specific assumptions. In a forex context, an “algorithm” typically means a deterministic set of rules for decisions (for example, when to enter/exit and how position size is set). The key point is that testing is not the same as forecasting: it describes past or simulated behavior given a particular setup.
Worked scenario example (with explicit assumptions)
Below is a worked example using simplified numbers. It is intentionally not based on real-time market data.
Goal and test period
Assume you want to test an algorithm that opens and closes trades using these fixed mechanics:
- Only one position can be open at a time.
- When a trade is opened, it closes after exactly 5 price steps.
- Profit/loss is calculated from a single entry price and a single exit price.
- Position size is constant: 10 units per trade.
Assumptions (state them explicitly):
- Test data: 3 independent “trades” from historical-like samples, with known entry and exit prices.
- Spread/slippage handling: you model transaction costs using a constant per-trade cost of 0.50 (in your chosen P/L units).
- No financing, rollover, or taxes are included.
- All prices are treated as perfectly known at decision time (no look-ahead).
Input data
Assume the three trades have the following entry and exit prices (price movement determines raw P/L):
- Trade A: entry 100.0 → exit 101.0 (movement +1.0)
- Trade B: entry 200.0 → exit 198.5 (movement −1.5)
- Trade C: entry 50.0 → exit 50.2 (movement +0.2)
To convert movement into raw P/L, assume a simple linear conversion:
- Raw P/L = movement × position size
- Position size = 10 units
So:
- Trade A raw P/L = (+1.0) × 10 = +10.0
- Trade B raw P/L = (−1.5) × 10 = −15.0
- Trade C raw P/L = (+0.2) × 10 = +2.0
Apply transaction costs
Assume a fixed transaction cost per closed trade of 0.50.
- Net P/L per trade = raw P/L − 0.50
Results:
- Trade A net = 10.0 − 0.50 = 9.5
- Trade B net = −15.0 − 0.50 = −15.5
- Trade C net = 2.0 − 0.50 = 1.5
Total net P/L over the 3-trade test = 9.5 + (−15.5) + 1.5 = −4.5
Metrics you can verify
From the same assumptions, you can compute:
- Win rate = 2 winning trades out of 3 = 66.7%
- Average net P/L per trade = (−4.5) / 3 = −1.5
- Gross profit (sum of winners) = 9.5 + 1.5 = 11.0
- Gross loss (absolute sum of losers) = 15.5
- Profit factor = gross profit / gross loss = 11.0 / 15.5 ≈ 0.71
This is a “worked” example because someone else can reproduce every number by using the same inputs and formulas. If they cannot, the testing setup is unclear.
How the mechanics differ from market/provider variability
In real use, several items typically vary and can change outcomes even if the algorithm rules stay the same:
- Market behavior: volatility regimes, correlations, and trend changes.
- Execution details: order execution quality, timing, and realized costs.
- Data quality: whether the historical-like prices reflect what could have been observed without look-ahead.
The example above separated fixed mechanics (5-step holding, constant size, cost model) from variable parts (the price movements). That separation is central to good testing because it tells you what is determined by your algorithm versus by your assumptions.
Material limitations and failure modes
At least one major limitation is that testing can produce misleading conclusions if assumptions do not match reality. Common failure modes include:
- Overfitting: tuning algorithm rules or parameters until they match a particular historical sample, reducing performance on new data.
- Execution mismatch: simulated fills and cost models may be too optimistic compared with real execution.
- Hidden costs and conditions: real trading can involve effects not represented in the simplified cost model (for example, varying spreads or other fees).
- Data leakage: using information in testing that would not have been available at decision time inflates results.