Linear regression in one minute
Linear regression models an outcome as a linear combination of input variables. In its simplest form, it estimates an equation like: y = a + b·x, where a is the intercept and b is the slope. “Fit” typically means the method chooses parameters (a and b) to reduce an objective such as the average squared difference between predicted values and observed values.
When you apply the idea in any setting (including finance-related feature engineering), the key limitation is not the arithmetic of the fit—it is whether the assumptions behind the fit resemble the real data-generating process.
Mechanism and where assumptions enter
A basic limitation is that linear regression is built around modeling choices and assumptions:
- Linearity: it assumes the expected change in y per unit change in x is constant (after accounting for other included variables). If the true relationship is nonlinear, the fitted line can systematically miss the pattern.
- Independence and stable error behavior: many standard interpretations assume errors are not strongly correlated and have roughly constant variance. If errors cluster over time or variance changes, uncertainty estimates can be wrong.
- Exogeneity of inputs (in causal terms): if x is influenced by unobserved factors that also affect y, the fitted relationship can reflect those hidden factors rather than a stable link.
Separating “mechanics” from “market/provider conditions” matters because the regression fitting procedure is deterministic given data, but the real-world environment that generates new data may differ.
Evidence and concrete failure modes (with assumptions stated)
Consider a toy example where you fit y = a + b·x using historical (x, y) pairs and evaluate the fit using the same historical data. A good historical fit can happen even if the relationship is not reliable.
Common failure modes include:
- Overfitting to noise: with too many variables, or with patterns that only exist in a limited time window, the model can match past randomness.
- Regime shifts: if the relationship between x and y changes after the training period, the earlier coefficients no longer describe the new regime.
- Outliers and heavy tails: squared-error objectives can be dominated by extreme observations, leading to parameter estimates that reflect outliers more than the typical data.
- Non-stationarity: in many time-ordered datasets, distributions can drift. A regression trained on one period may not represent later periods.
In finance-related contexts, it is also common for the “same” variable to behave differently across conditions (for example, during volatility changes). Linear regression does not automatically detect those shifts; it simply extrapolates a learned relationship until the data stops matching its assumptions.
Limitations and risks
The main limitations are about uncertainty and how easily the interpretation breaks:
- Historical relationships do not establish future results. Even if the model minimizes squared error in the past, there is no guarantee the future data-generating process matches the training conditions.
- Estimates and intervals depend on assumptions. If error variance changes or observations are correlated, standard uncertainty statements can be too optimistic or miscalibrated.
- Sensitivity to data quality: missing values, inconsistent preprocessing, or shifting measurement can change results.
- “Model misspecification”: even when the method runs, the linear form may be the wrong functional form.
Separately, outcomes in real decision settings can vary due to costs, execution differences, and jurisdiction-specific constraints. Those factors are not captured by the regression fit alone, so an apparently strong relationship may not translate into usable real-world performance.
How to independently verify what is (and isn’t) supported
To verify limitations in a way that does not assume future success, you can check whether the underlying assumptions appear plausible:
- Use a time-respecting evaluation: test on later data that was not used to fit parameters.
- Check residual behavior: look for systematic patterns in errors and signs of changing variance.
- Test stability: fit models on multiple windows and see whether coefficients vary dramatically.
- Compare functional forms: if a simple linear form repeatedly fails while nonlinear alternatives reduce systematic error, the limitation is likely misspecification.
If you need to explain the limitations clearly, focus on what would have to be true for the regression to be meaningful in your context, and then assess where that could fail.