Worked Example of Out-of-Sample Testing

Direct answer

Out-of-sample testing is a way to check how a decision rule (for example, a forecasting model or strategy rule) performs on data it did not see during fitting or tuning. A worked example makes the logic clear: you create a model using a first portion of historical data, then you evaluate it on a separate “holdout” portion. The goal is to estimate performance without the optimistic bias that comes from adapting to the same data.

A worked example should explicitly state assumptions such as how the split is made, what calculations are repeated, and how performance is computed. It should also note limitations: out-of-sample results can still fail when the market changes, when costs and execution are unrealistic, or when information leaks from the evaluation period into the fitting steps.

Mechanism and definition

Here is the core workflow, stated generally:

Choose a model family and a fitting procedure. This is the part that can “learn” from data.
Split the available dataset into two disjoint sets: an in-sample (training) period and an out-of-sample (test) period.
Fit or tune only on the in-sample set (including selecting hyperparameters or thresholds).
Freeze the model and evaluate it once on the out-of-sample set.

Important terms in this context:

“In-sample” means the data used for fitting/tuning.
“Out-of-sample” means the data reserved for evaluation.
“Holdout” means the test period is not used to make further tuning decisions.

Why this helps: when you tune on the same data you evaluate, you can accidentally capture noise. Out-of-sample testing reduces that particular bias by forcing evaluation on unseen data.

Evidence or worked numerical example (with explicit assumptions)

Assume you want to predict the next-day direction (Up or Down) based on one feature, X, computed from each day’s information available before the next day starts.

Assumptions for the example:

You have 20 sequential days of data.
Days 1–12 are in-sample; days 13–20 are out-of-sample.
The model is a simple rule: predict “Up” if X > T, otherwise predict “Down.”
X values and realized next-day outcomes are fixed (no new data arrives).
During tuning, you may choose T using only Days 1–12.
You compute accuracy as the fraction of correct day-ahead direction calls.

Step A: Tune T on in-sample (Days 1–12) Suppose that when you try candidate thresholds, the best threshold on Days 1–12 is T = 0.50.

Accuracy on in-sample using T=0.50: 9 correct out of 12 = 75%.

Step B: Freeze and evaluate on out-of-sample (Days 13–20) Now you apply the frozen rule (Up if X>0.50, else Down) to Days 13–20. Suppose the rule produces:

3 correct out of 8 days.
Out-of-sample accuracy = 3/8 = 37.5%.

How to interpret this numeric result

The in-sample score (75%) suggests the rule fits something in the training period.
The out-of-sample drop to 37.5% indicates the apparent in-sample relationship did not generalize well to the later period.
This does not prove the rule is useless forever; it shows that, under the stated split and assumptions, the evidence from the evaluation period is weaker than the evidence from training.

Limitations and material failure modes

Out-of-sample testing reduces one form of bias, but it does not guarantee reliable future performance. Material limitations include:

Multiple comparisons and test-set overuse If you repeatedly adjust T (or other choices) based on the test period’s results, the test becomes partially in-sample. That reintroduces optimistic bias.
Data leakage If any part of the fitting process accidentally uses information unavailable at the time of prediction (for example, using future-derived inputs), then the out-of-sample estimate can be invalid even with a clean split.
Non-stationarity (regime change) Financial time series can change behavior over time. A model that works in one period may fail in another because the underlying relationship between X and the outcome has shifted.
Missing realism: costs and execution Even if the direction predictions are “correct,” real outcomes depend on transaction costs, spreads, and execution timing. Simplified evaluation metrics may overstate practical results.
Small sample effects With limited out-of-sample data, the estimate can have high variability. A low or high number may reflect randomness rather than true generalization.