Linear regression, in plain terms
Linear regression is a method for modeling a relationship between two quantities:
- x: the input (independent variable)
- y: the output (dependent variable)
The model assumes that, on average, y changes approximately in a straight-line pattern as x changes. “Approximately” is important: the data almost never follows an exact line, so the model includes an error term.
The most common version is ordinary least squares (OLS), which chooses the straight line that makes the overall prediction errors as small as possible.
The calculation: model and formula
The linear regression model is written as:
[ \hat{y} = a + b x ]
Where:
- (\hat{y}) is the predicted value of y
- (a) is the intercept (the predicted y when x = 0)
- (b) is the slope (how much (\hat{y}) changes when x increases by 1)
To calculate a and b, OLS uses the idea of minimizing sum of squared residuals.
Residuals and the least-squares objective
For data points ((x_i, y_i)) with (i=1,2,\dots,n):
- Residual: (e_i = y_i - \hat{y}_i)
- Sum of squared residuals: (\sum_{i=1}^{n} e_i^2)
OLS picks (a) and (b) to minimize that sum.
Slope (b)
A widely used computational form for the slope is:
[ b = \frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n}(x_i-\bar{x})^2} ]
Here:
- (\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i)
- (\bar{y} = \frac{1}{n}\sum_{i=1}^{n} y_i)
Intuition: the numerator is the co-movement of x and y (how x deviations line up with y deviations). The denominator is the spread of x around its mean.
Intercept (a)
Once (b) is known, the intercept follows from the requirement that the fitted line passes through the point ((\bar{x},\bar{y})):
[ a = \bar{y} - b\bar{x} ]
What data you need (and what must be true)
To calculate a simple linear regression you need:
- A set of paired observations ((x_i, y_i)) for (i=1\ldots n).
- Enough variation in x, meaning (\sum (x_i-\bar{x})^2 \neq 0). If all x values are the same, the slope is undefined.
- A clear meaning of x and y for your use case. The model always produces predictions; it does not automatically tell you that the relationship is causal.
Assumptions behind the “straight-line” fit
OLS calculations come with assumptions that affect interpretation:
- The line form (a + bx) is an appropriate approximation for the data pattern.
- Errors are typically treated as unrelated to x (more precisely, the expected residual given x is often assumed to be 0).
- Residuals should not show systematic structure; if they do, a straight line may be a poor choice.
These are assumptions about how the data behaves, not about the future.
A worked check with a small example (no special software)
Suppose you have 3 points:
((x_1,y_1),(x_2,y_2),(x_3,y_3)). To compute the regression line:
- Compute (\bar{x}) and (\bar{y}).
- Compute the numerator:
(\sum (x_i-\bar{x})(y_i-\bar{y})). - Compute the denominator:
(\sum (x_i-\bar{x})^2). - Compute (b) as numerator ÷ denominator.
- Compute (a = \bar{y} - b\bar{x}).
To verify, calculate each fitted value (\hat{y}_i = a + b x_i), then compute residuals (e_i = y_i-\hat{y}_i). If residuals are large or show a curve pattern rather than being random, that indicates the simple linear form may not match the data well.
Limitations and failure modes
Even though the arithmetic is straightforward, linear regression can be misleading when the data violates the idea behind the model.
1) Nonlinear relationships
If the true relationship is curved (for example, it rises then falls), a straight line will average across that behavior. The output may still minimize squared errors, but it can hide important structure.
2) Outliers can dominate
Because errors are squared in the objective, extreme points can strongly influence (a) and (b). A single unusual observation can shift the fitted line considerably.
3) x must vary
If x has no spread (all x values identical), the denominator in the slope formula becomes zero. In that case, you cannot compute a meaningful slope.
4) “Good fit” does not imply future usefulness
Historical patterns can change. A regression fitted on past data describes how the data behaved in that dataset; it does not prove that the same pattern will persist.
Verification: how to independently check your result
To independently verify a computed linear regression, you can:
- Recompute (\bar{x}) and (\bar{y}).
- Recompute the slope (b) using the deviation form.
- Recompute the intercept (a = \bar{y}-b\bar{x}).
- Generate fitted values (\hat{y}_i) and residuals (e_i).
- Inspect residuals:
- Are they mostly small?
- Do they look randomly scattered, or do they show systematic patterns?
If any computed value does not match your re-calculation, the likely issue is an input mismatch (different x/y pairing, unit mix-up, or wrong ordering), not the regression formulas.
Next question to clarify
Before applying or interpreting a linear regression, it helps to clarify what x represents and whether a straight-line approximation is reasonable for that data. If you want to go further, comparing linear regression to related ideas such as correlation or different types of regression (e.g., models with more predictors) can clarify what changes in the calculation and what remains the same.