What “responsible backtesting” means for Frama
Backtesting is a way to study how an indicator-based rule might have behaved on historical price data. “Responsible” backtesting does not claim future performance. Instead, it builds a transparent, repeatable test that (1) defines the indicator inputs and settings, (2) models costs and execution realistically, (3) limits bias and overfitting, and (4) checks whether results survive new data outside the period used to design the test.
For Frama specifically, you should first understand it as a moving average whose behavior depends on market structure captured from price variation. Any backtest should therefore document how you compute the moving average inputs and how you turn the indicator value(s) into a rule—without assuming that an indicator value automatically equals a trade signal.
Mechanics: define data, assumptions, and how Frama is used in the test
Start by writing down every assumption that affects the numbers:
- Data definition: what instrument (generic FX pair), what timeframe (for example, hourly bars), what time zone, and whether the series is adjusted for corporate actions (usually less relevant for FX than equities, but the principle is the same).
- Indicator settings: the Frama parameters and any lookback window(s). Backtests should keep these fixed unless you explicitly run a parameter search.
- Computation rules: whether the indicator is computed using the same bar close you later evaluate, and how you handle the first bars where the indicator needs warm-up history.
- Rule definition: if you evaluate “signal quality,” define the exact decision logic (for example, whether the test evaluates crossing behavior, thresholding, or regime filters). Do not treat “indicator direction” as a universal standalone signal.
Next, model costs and execution in a way that matches your assumptions. Even without real-time spreads, you can include friction terms as constant placeholders (for example, a fixed transaction cost per entry and exit) and state that these are assumptions.
Costs to consider in the same framework:
- Transaction costs (commissions, fees).
- Bid/ask spread or slippage model (how you approximate the difference between bar-based prices and trade prices).
- Order timing (whether you assume execution at the next bar open, at close, or with a fixed delay).
Finally, choose metrics that reflect the question you are asking. Examples include risk-adjusted performance proxies, maximum drawdown, hit rate, or average adverse excursion. The key is to define metrics before you run tests.
Evidence and example: how to design checks that reduce bias
A common failure mode in indicator backtesting is overfitting: the test finds a configuration that matches one historical period but does not generalize.
Two practical bias-control approaches:
-
Walk-forward (rolling) validation
- Split history into multiple consecutive segments.
- Use an earlier segment to select or confirm settings, then evaluate on the next segment.
- Repeat and aggregate results.
-
Strict out-of-sample holdout
- Reserve a final period that is never used for selecting parameters or crafting rules.
- Only after the rule is fixed do you evaluate on the holdout.
If you run parameter exploration (for example, varying Frama lookback choices), separate the act of “finding” from the act of “measuring.” A responsible workflow treats the measurement step as a final test of generalization.
Also test stability:
- Check whether performance depends heavily on one regime.
- Repeat the evaluation across multiple time periods and, when feasible, multiple instruments.
- Compare results to a simple benchmark (such as a passive baseline) to interpret whether any improvement is meaningfully different from what a basic approach can achieve.
Limitations and failure modes to treat as first-class information
Historical backtest results are uncertain. Important limitations include:
- Market regime dependence: an indicator’s structure may work only when volatility and trend characteristics match the period used in the test.
- Cost and execution mismatch: small changes in assumed transaction costs, slippage, or timing can reverse conclusions, especially for strategies with frequent changes.
- Data artifacts: differences in sampling, missing bars, different data vendors, or inconsistent bar construction can change outcomes.
- Indicator warm-up and edge effects: early-bar initialization can bias results if not handled consistently.
- Multiple comparisons and researcher degrees of freedom: if you try many parameter settings and look for the “best” one, you increase the chance of finding patterns by chance.