Definition: what linear regression is
Linear regression is a statistical method that models the relationship between a dependent variable (y) and one independent variable (x) using a straight-line equation:
[ \hat{y} = a + b x ]
- (\hat{y}) is the predicted value of (y).
- (a) is the intercept.
- (b) is the slope.
The key idea is that the fitted line is chosen so it makes the prediction errors as small as possible in a specific way: it minimizes the sum of squared residuals (\sum (y-\hat{y})^2) over the training data. Residuals are the differences between observed (y) values and predicted (\hat{y}) values.
Mechanism: what a worked example actually computes
A “worked example” means you pick a small dataset, state every assumption (including how (x) and (y) are defined), compute (a) and (b), then use the fitted line to produce fitted values and at least one new prediction.
Assumptions for the worked numeric calculations below:
- You have paired observations ((x_i, y_i)).
- The model form is correct for the data window: a linear relationship between (x) and (y).
- The coefficients are computed with ordinary least squares (OLS): minimizing squared residuals.
- We treat the input (x) values as given (no additional uncertainty model for (x)).
- The example is purely illustrative; it does not use real-time market data.
Evidence: a fully numerical worked example
Suppose you observe 4 points:
- (x: 1, 2, 3, 4)
- (y: 2, 3, 5, 4)
Step 1: Compute means. [ \bar{x} = (1+2+3+4)/4 = 2.5 ] [ \bar{y} = (2+3+5+4)/4 = 3.5 ]
Step 2: Compute the slope (b). [ b = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} ] Compute the table values:
- For (x=1): (x-\bar{x}=-1.5), (y-\bar{y}=-1.5), product = 2.25
- For (x=2): (x-\bar{x}=-0.5), (y-\bar{y}=-0.5), product = 0.25
- For (x=3): (x-\bar{x}=0.5), (y-\bar{y}=1.5), product = 0.75
- For (x=4): (x-\bar{x}=1.5), (y-\bar{y}=0.5), product = 0.75
Sum of products = (2.25+0.25+0.75+0.75 = 4.0).
Denominator (\sum (x_i-\bar{x})^2):
- ((-1.5)^2=2.25), ((-0.5)^2=0.25), ((0.5)^2=0.25), ((1.5)^2=2.25)
Sum = (2.25+0.25+0.25+2.25 = 5.0).
So: [ b = 4.0/5.0 = 0.8 ]
Step 3: Compute intercept (a). [ a = \bar{y} - b\bar{x} = 3.5 - 0.8\cdot 2.5 = 3.5 - 2.0 = 1.5 ]
Step 4: Write the fitted line. [ \hat{y} = 1.5 + 0.8x ]
Step 5: Compute fitted values for the observed (x).
- If (x=1), (\hat{y}=2.3)
- If (x=2), (\hat{y}=3.1)
- If (x=3), (\hat{y}=3.9)
- If (x=4), (\hat{y}=4.7)
Step 6 (one simple prediction): For a new (x=5), [ \hat{y}=1.5+0.8\cdot 5=5.5 ]
Every number above can be independently verified by recomputing the sums and the OLS formulas.
Limitations and risks: what can make linear regression fail
Linear regression is not a guarantee of accurate predictions. Material limitations include:
- Linearity assumption may be wrong. If the true relationship between (x) and (y) bends, a straight line can systematically miss the pattern.
- Sensitivity to outliers. A single extreme (y) (or an influential (x) far from the mean) can pull the slope and intercept.
- Historical relationships may not hold. Even if the fitted line matches the training data, the future could behave differently because the data-generating process can change.
- Omitted variables. If (y) depends on other factors not represented by (x), the errors can be structured rather than random.
- Noisy data and model error. Minimizing squared residuals improves one notion of fit but does not ensure good performance under different conditions.
These issues are general to statistical modeling: outcomes vary with data properties and how the data was collected and processed.