Direct answer
A worked example of overfitting shows how a model can look accurate on the data it was trained on, while performing poorly on new data. Overfitting is not guaranteed; it is a risk that increases when a model becomes more complex relative to the amount and variability of the available data.
Mechanism or definition
Overfitting means the learning process fits idiosyncrasies in the training dataset—random noise, quirks, or accidental correlations—rather than learning stable relationships that also hold for future data.
A simple way to think about it:
- Training (in-sample) fit: how well the model matches the historical data it saw while being trained.
- Test (out-of-sample) fit: how well the same fixed model matches new data it did not use for training.
If training fit improves as model complexity increases, but test fit stays the same or gets worse, that pattern is consistent with overfitting.
Evidence or example (with every assumption)
Consider a situation where you must predict a target value y from one input x using a very small dataset.
Assumptions
- We have 4 labeled observations for training: (x, y): (1,2), (2,4), (3,6), (4,8).
- We will compare two models:
- Model A: a straight line forced to pass through the origin, y = m x.
- Model B: a polynomial of degree 3 that exactly matches all 4 training points.
- We define error as mean squared error (MSE) on a separate test set.
- We assume test points are: (x, y): (5,10) and (6,12).
- We assume Model B keeps its fitted polynomial form unchanged when evaluated on test points.
- We add no transaction costs, slippage, or execution constraints (to keep the example purely mathematical).
Step 1: Fit Model A (linear through origin)
The training data follow y = 2x exactly, so the best m for Model A is m = 2.
- Training error for Model A: 0 MSE.
On the test set:
- For x=5, predicted y = 2·5 = 10 (matches actual)
- For x=6, predicted y = 2·6 = 12 (matches actual) So test error for Model A is also 0 MSE.
Step 2: Fit Model B (degree-3 polynomial that matches training exactly)
Because Model B is flexible enough, it can match the 4 training points perfectly, so:
- Training error for Model B: 0 MSE.
However, this model does not “know” the underlying rule. With only 4 training points, there are infinitely many behaviors outside those points; many degree-3 fits will match training while differing on test points. In this worked scenario, assume the learned degree-3 polynomial implies predictions on the test set of:
- x=5 → predicted y = 9
- x=6 → predicted y = 13 (These values are consistent with the idea that extra flexibility can distort extrapolation.)
Then the test MSE for Model B is:
- Errors: (9−10) = −1 and (13−12) = +1
- Squared errors: 1 and 1
- MSE = (1+1)/2 = 1
What this shows
- Both models have zero training error.
- Model A generalizes in this scenario.
- Model B can generalize poorly even with zero training error.
The conceptual lesson is why overfitting is plausible: a more complex model can match training patterns exactly, but that does not ensure the same patterns exist in future data.
Limitations and risks
- No guarantee from historical fit: perfect training fit does not imply future performance. The relationship may be accidental or regime-specific.
- Small datasets amplify overfitting: with fewer data points, many different complex functions can fit training while disagreeing elsewhere.
- Data leakage risk: if information from the test period accidentally influences training (even indirectly), test results can be overly optimistic.
- Regime changes and noise: if the data-generating process changes, the “stable” relationship may disappear.
- Evaluation choices matter: different train/test splits, metrics, and error definitions can change whether you observe overfitting.
Verification or next question
To independently verify whether overfitting is occurring, compare in-sample and out-of-sample errors while controlling complexity:
- Start with a simpler model class.
- Increase flexibility gradually.
- Observe whether training error decreases but test error increases.
A practical next question is: *How sensitive are the results to the specific train/test split and the evaluation metric?