Topic 3: Linear Regression

Introduction to Business Analytics — HKUST

Prof. Xuhu Wan

Why Predict?

Why it matters

Prediction = data-driven decisions.

Instead of guessing, we let the data tell us what to expect.

Today’s finance questions:

“Is NVDA aggressive or defensive?”
“Which forces drive NVDA returns?”
“Does NVDA earn alpha?”

Visual idea: CAPM fits a line through NVDA returns vs. market returns — slope = beta, intercept = alpha.

Today’s Roadmap

Visualizing Association
SLR: CAPM (NVDA & Market)
Inference: $t$-tests & CIs for $\beta$
CI for $\mu_Y$ vs. Prediction Interval
Train/Test & Evaluation
Assumptions & Residual Analysis
MLR: Fama-French 5-Factor
Variable Selection
Applications

1. Visualizing Association

Learning Goals

Compute and interpret the correlation coefficient
Create heatmaps and scatter plots in Python
Understand that correlation measures only linear association

Correlation Coefficient

The Pearson correlation $r$ measures the strength of linear association:

\[ r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2 \cdot \sum_{i=1}^{n}(y_i - \bar{y})^2}} \]

$r$ value	Interpretation
$-1$	Perfect negative linear
$-0.5$	Strong negative
$0$	No linear association
$+0.5$	Strong positive
$+1$	Perfect positive linear

Key takeaway

$r$ ranges from $-1$ to $+1$. Close to $\pm 1$ = strong linear relationship. Close to $0$ = weak or no linear relationship.

Download Data & Compute Correlation

In the real world you’d use yfinance to pull live prices. For these slides we pre-fetched real NVDA and SPY daily closes for 2023–2024 and ship them as a CSV — so the analysis runs in your browser, with real data, instantly.

What just happened

The browser fetched a small CSV (~25 KB) from this site, parsed it into a pandas DataFrame, and computed daily returns + correlation — all in your browser. No yfinance, no Colab, no Binder, no tab switching.

For Topic 3 demos that need live data

The same CSV-pre-fetch pattern is used throughout this topic. Production code (for an assignment, your own analysis) should still use yfinance — run it in Colab for live prices.

Alternative: Binder (live Python, embedded)

For comparison — Binder lets you run a real Jupyter kernel inline with any package (yfinance, isom2600, etc.) but takes 30s–2min to cold-start.

How Binder works:

Reads a requirements.txt from a public GitHub repo
Spins up a fresh Docker container in the cloud
Embeds a JupyterLab UI inline in this slide
First load is slow (cold start), then runs fast

Best for: occasional one-off demos that need packages Pyodide doesn’t have.

Not for: every-day teaching — the wait kills lecture flow.

First load ~30s–2min. Or open in new tab.

Simulated NVDA vs SPY in Pyodide

Scatter Plot: NVDA vs. SPY

Key takeaway

Slope $>1$: NVDA moves more than the market.

2. Simple Linear Regression: CAPM

Learning Goals

Understand the regression model $y = \alpha + \beta x + \epsilon$
Apply CAPM: relate NVDA’s excess return to the market
Interpret $\alpha$ (Jensen’s alpha) and $\beta$ (market sensitivity)
Build and read a regression output in Python

Simple Linear Regression Model

\[ y = \alpha + \beta x + \epsilon \]

Geometric view: the regression line passes through the data; $\alpha$ is its intercept, $\beta$ is its slope, and $\epsilon_i$ is the vertical distance from each point to the line.

$\alpha$ = intercept — predicted $y$ when $x=0$
$\beta$ = slope — change in $y$ per unit $x$
$\epsilon$ = error — what the model cannot explain

The Capital Asset Pricing Model (CAPM)

CAPM is a simple linear regression of a stock’s excess return on the market:

\[ Y = \alpha + \beta\, X + \epsilon \]

where $Y = R_{\text{NVDA}} - R_f$ (NVDA excess return) and $X = R_m - R_f$ (market excess return).

$\alpha$ (Jensen’s alpha) = return above what CAPM predicts
- $\alpha > 0$: NVDA outperforms the market
- $\alpha = 0$: no extra reward (efficient market)
$\beta$ (market beta) = sensitivity to market
- $\beta > 1$: aggressive — amplifies market
- $\beta < 1$: defensive — dampens market

Prepare the Data

import yfinance as yf
import pandas_datareader.data as web
tickers = ["NVDA", "SPY"]
px = yf.download(tickers, start="2023-01-01", end="2024-12-31",
                 auto_adjust=True, progress=False)["Close"]
ret = px.pct_change().dropna()
ff = web.DataReader("F-F_Research_Data_5_Factors_2x3_daily",
    "famafrench", start="2023-01-01", end="2024-12-31")
rf = ff[0]["RF"] / 100  # percent -> decimal
data = ret.join(rf, how="inner")
data["NVDA_excess"] = data["NVDA"] - data["RF"]
data["Mkt_excess"]  = data["SPY"]  - data["RF"]

Why it matters

We subtract $R_f$ because CAPM measures the premium over T-bills.

Fit the CAPM with `statsmodels`

Key takeaway

Three steps: (1) sm.add_constant(X) adds $\alpha$, (2) sm.OLS(y, X) defines the model, (3) .fit() estimates.

Why it matters

.fit() uses Ordinary Least Squares (OLS) — the same method from ISOM 2500. It finds $\hat{\alpha}, \hat{\beta}$ that minimise $\sum (y_i - \hat{y}_i)^2$.

Reading the CAPM Output

	coef	std err	t	P>\|t\|
`const` ($\alpha$)	0.0036	0.001	2.71	0.007
`Mkt_excess` ($\beta$)	2.332	0.166	14.04	0.000
$R^2$	0.331

$\beta = 2.33$: NVDA moves 2.33% per 1% market move
$p(\beta) = 0.000$: market is significant
$p(\alpha) = 0.007$: $\alpha$ is significant

Why it matters

$R^2 = 0.331$: market explains 33.1% of NVDA variation. Remaining 66.9% = idiosyncratic risk.

Variation Decomposition

In any linear regression, the total variation in $Y$ decomposes as:

\[ \underbrace{SST}_{\sum(y_i - \bar{y})^2} \;=\; \underbrace{SSR}_{\sum(\hat{y}_i - \bar{y})^2} \;+\; \underbrace{SSE}_{\sum(y_i - \hat{y}_i)^2} \qquad R^2 = \frac{SSR}{SST} \]

SST = total variation in $Y$
SSR = variation explained by $X$
SSE = variation not explained (residual)

Why it matters

In CAPM: SSR = systematic risk (market), SSE = idiosyncratic risk (NVDA-specific).

3. Inference on Regression Coefficients

Learning Goals

Understand the sampling distributions of $\hat{\alpha}$ and $\hat{\beta}$
Conduct $t$-tests for slope and intercept
Build and interpret 95% confidence intervals for $\alpha$ and $\beta$
Read all of this directly from the statsmodels summary

LINE Assumptions for Inference

For $t$-tests and confidence intervals to be valid, the regression must satisfy:

L Linearity — $E[Y|X] = \alpha + \beta X$
I Independence — $\epsilon_i$ are independent
N Normality — $\epsilon_i \sim N(0, \sigma^2)$
E Equal variance — $\text{Var}(\epsilon_i) = \sigma^2$

Important

If LINE holds: $\hat{\alpha}$ and $\hat{\beta}$ follow $t$-distributions, $p$-values are trustworthy, CIs have correct coverage. If violated → $p$-values and CIs may be misleading.

Sampling Distributions of $\hat{\alpha}$ and $\hat{\beta}$

The OLS estimates are random variables — different samples give different estimates.

Let $s^2 = \dfrac{SSE}{n-2}$ (residual variance) and $S_{xx} = \sum_{i}(x_i - \bar{x})^2$.

Standard errors:

\[ SE(\hat{\beta}) = \frac{s}{\sqrt{S_{xx}}} \qquad SE(\hat{\alpha}) = s\,\sqrt{\dfrac{1}{n} + \dfrac{\bar{x}^2}{S_{xx}}} \]

Why it matters

$SE(\hat{\beta})$ shrinks when $S_{xx}$ is large (wide spread in $x$), $s$ is small (tight residuals), or $n$ is large. More data spread $\Rightarrow$ sharper slope estimate.

Why Do Standard Errors Matter?

Coefficients tell you what the relationship is. Standard errors tell you how much to trust it.

1. They power every $t$-test:

\[ t_{\hat{\alpha}} = \frac{\hat{\alpha}}{SE(\hat{\alpha})} \qquad t_{\hat{\beta}} = \frac{\hat{\beta}}{SE(\hat{\beta})} \]

2. They determine CI width:

\[ \hat{\alpha} \pm t_{0.025}\!\cdot\! SE(\hat{\alpha}) \quad \hat{\beta} \pm t_{0.025}\!\cdot\! SE(\hat{\beta}) \]

Key takeaway

Coefficient = point estimate. SE = margin of error.

`model.summary()` Output

	coef	std err	t	P>\|t\|	[0.025, 0.975]
const ($\alpha$)	0.0036	0.001	2.706	0.007	0.001, 0.006
Mkt_excess ($\beta$)	2.3320	0.166	14.038	0.000	2.005, 2.659

$H_0: \alpha = 0$ vs $H_a: \alpha \ne 0$ (two-tailed)
$H_0: \beta = 0$ vs $H_a: \beta \ne 0$ (two-tailed)

Key takeaway

Each row tests whether the coefficient is significantly different from zero. The P>|t| column gives the two-tailed $p$-value.

$t$-Tests for Slope and Intercept

Under LINE, each estimate follows a $t$-distribution:

Slope ($\beta$):

\[ t_{\hat{\beta}} = \frac{\hat{\beta}}{SE(\hat{\beta})} \;\sim\; t_{n-2} \qquad H_0: \beta = 0 \]

$p$-value $< 0.05$ $\Rightarrow$ $x$ is a significant predictor.

Intercept ($\alpha$):

\[ t_{\hat{\alpha}} = \frac{\hat{\alpha}}{SE(\hat{\alpha})} \;\sim\; t_{n-2} \qquad H_0: \alpha = 0 \]

Important

In CAPM, the slope test ($H_0:\beta=0$) answers: “Does the market explain NVDA at all?” The intercept test ($H_0:\alpha=0$) answers: “Does NVDA earn a free lunch above market risk?”

What Is a $p$-Value? (Two-Tailed)

Why it matters

The $p$-value is the probability of seeing a result as extreme or more, assuming $H_0$ is true.

Decision rule:

$p < 0.05$ $\Rightarrow$ reject $H_0$ (significant)
$p \ge 0.05$ $\Rightarrow$ fail to reject $H_0$

Example: $p = 0.003$ for $\hat{\beta}$ $\Rightarrow$ only 0.3% chance of this slope if $\beta = 0$ $\Rightarrow$ $X$ matters.

Key takeaway

Small $p$ $\Rightarrow$ unlikely under $H_0$ $\Rightarrow$ reject. $p$ is not $P(H_0 \text{ is true})$.

One-Tailed $p$-Value

Sometimes we test a direction, not just “different from zero”.

Right-tailed: $H_0: \beta \le 0$ vs $H_a: \beta > 0$

If $t > 0$: $p_{\text{one}} = p_{\text{two}} / 2$
If $t < 0$: $p_{\text{one}} = 1 - p_{\text{two}} / 2$

Left-tailed: $H_0: \beta \ge 0$ vs $H_a: \beta < 0$

If $t < 0$: $p_{\text{one}} = p_{\text{two}} / 2$
If $t > 0$: $p_{\text{one}} = 1 - p_{\text{two}} / 2$

Key takeaway

statsmodels reports two-tailed $p$. For one-tailed: divide by 2 only if the sign of $t$ matches $H_a$.

Testing $\alpha > 0$ and $\beta > 1$ in CAPM

Does NVDA earn positive alpha?

$H_0: \alpha \le 0$ vs $H_a: \alpha > 0$

Is NVDA more aggressive than the market?

$H_0: \beta \le 1$ vs $H_a: \beta > 1$

Key takeaway

Default test: $H_0: \beta = 0$ (does $x$ matter?). To test a threshold like $\beta = 1$, shift the null and recompute $t = (\hat{\beta} - 1)/SE(\hat{\beta})$.

Critical Value Approach

Instead of comparing $p$ to $0.05$, compare $|t|$ to the critical value $t_{\text{crit}}$.

Two-tailed ($H_0: \beta = 0$): Reject if $|t| > t_{0.025,\,n-2}$

One-tailed right ($H_0: \beta \le 0$): Reject if $t > t_{0.05,\,n-2}$

Key takeaway

Two equivalent approaches: (1) $p$-value $< 0.05$, or (2) $|t| > t_{\text{crit}}$. Both give the same conclusion. For large $n$, $t_{\text{crit}} \approx 1.96$ (two-tailed) or $1.645$ (one-tailed).

Confidence Intervals for $\alpha$ and $\beta$

A 95% confidence interval captures the true parameter with 95% probability:

\[ \text{CI for } \alpha:\quad \hat{\alpha} \;\pm\; t_{0.025,\,n-2} \cdot SE(\hat{\alpha}) \]

\[ \text{CI for } \beta:\quad \hat{\beta} \;\pm\; t_{0.025,\,n-2} \cdot SE(\hat{\beta}) \]

Key takeaway

CI for $\beta$: $[1.66, 2.58]$ — we are 95% confident NVDA’s market sensitivity is between 1.66 and 2.58. CI for $\alpha$: $[0.0008, 0.0085]$ — does not contain 0 $\Rightarrow$ significant alpha.

Reading the Statsmodels CI Output

model.summary() already reports everything:

	coef	std err	t	P>\|t\|	[0.025	0.975]
const ($\alpha$)	0.0036	0.001	2.71	0.007	0.0008	0.0085
Mkt_excess ($\beta$)	2.332	0.166	14.04	0.000	1.662	2.577

coef: $\hat{\alpha}$ or $\hat{\beta}$
std err: $SE$
t: coef / std err
P>|t|: two-sided p-value
[0.025, 0.975]: 95% CI

Why it matters

CI contains $0$ $\Leftrightarrow$ $p$-value $> 0.05$. They carry the same information — always check both.

Try It! — Inference on CAPM

Try it!

Using the CAPM model on NVDA data:

What is the 95% CI for $\beta$ (market sensitivity)?
Can you reject $H_0: \beta = 1$ at 5% significance? Hint: $t = (\hat{\beta} - 1) / SE(\hat{\beta})$. Compare to $t_{0.025, n-2} \approx 1.96$.
What does the CI for $\alpha$ tell you about NVDA’s risk-adjusted performance?
Re-run for TSLA. Compare $\beta_{\text{TSLA}}$ vs $\beta_{\text{NVDA}}$ — which is more aggressive?

4. CI for $\mu_Y$ vs. Prediction Interval

Learning Goals

Distinguish between two types of interval: CI for $\mu_Y$ and prediction interval (PI) for $Y$
Derive and compare the two formulas
Compute both with statsmodels .get_prediction()
Know when to use each in a business context

Two Questions, Two Intervals

Given a new predictor value $x^*$ (e.g., market return $= +1\%$), we ask two different questions:

CI for $\mu_Y \mid x^*$

Question: What is the average NVDA return on all days when the market returns $x^*$?
Target: the mean $\mu_Y = \alpha + \beta x^*$
Use when: estimating a population average.

Prediction Interval for $Y$

Question: What will NVDA return tomorrow, given the market returns $x^*$?
Target: a single future observation $Y = \mu_Y + \epsilon$
Use when: forecasting a specific future value.

The Formulas — One Extra Term

Both intervals are centered at $\hat{y}^* = \hat{\alpha} + \hat{\beta} x^*$.

CI for $\mu_Y$:

\[ \hat{y}^* \pm t_{\alpha/2,\,n-2} \cdot s\sqrt{\dfrac{1}{n} + \dfrac{(x^*-\bar{x})^2}{S_{xx}}} \]

Prediction Interval:

\[ \hat{y}^* \pm t_{\alpha/2,\,n-2} \cdot s\sqrt{\color{red}{1} + \dfrac{1}{n} + \dfrac{(x^*-\bar{x})^2}{S_{xx}}} \]

Important

The “1” in the PI formula accounts for the individual error $\epsilon$ in the new observation.

A single observation varies around the mean — even if we knew $\mu_Y$ exactly, the PI could not shrink to zero.

PI is always wider than CI.

Why PI Is Always Wider

The CI band hugs the regression line tightly; the PI band is much wider because it has to cover individual scatter around the line.

Key takeaway

As $n \to \infty$, the CI for $\mu_Y$ narrows to a line (we learn $\mu_Y$ exactly). The PI never collapses — individual variation $\sigma^2$ is irreducible.

Python: `get_prediction()` for Both Intervals

Key takeaway

mean_ci: CI for $\mu_Y$ — NVDA’s expected average on +1% market days: $[+1.7\%, +2.1\%]$ obs_ci: PI for individual $Y$ — tomorrow’s NVDA return: $[-1.1\%, +4.9\%]$

The PI is 10× wider — individual returns are noisy even when the mean is precise.

Plot Both Bands Together

Business Interpretation

Use CI for $\mu_Y$ when…

Estimating the average effect
Benchmarking: “On average, when the market is up 1%, where should NVDA trade?”
Policy evaluation (expected impact)

Use PI when…

Forecasting a specific future value
Risk sizing: “What is the worst-case NVDA loss tomorrow?”
Inventory/demand planning for one period

Try it!

If the market drops 2% tomorrow, use get_prediction() to find: (a) the 95% CI for NVDA’s expected return, and (b) the 95% PI for NVDA’s actual return. Which interval would a risk manager care about?

5. Train/Test Split & Model Evaluation

Learning Goals

Understand why we split data into train and test sets
Compute R$^2$ and RMSE to evaluate model performance
Detect overfitting by comparing train vs. test metrics

Why Split the Data?

Why it matters

Train to learn, test to verify. If you test on training data, you are grading your own homework!

Diagram: full dataset → split into 80% Train (estimate $\alpha, \beta$) and 20% Test (predict).

Key rule: Time series — train on earlier dates, test on later dates. Never shuffle!

R$^2$ and RMSE

Two key metrics:

R$^2$ (R-squared):

\[ R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2} \]

$R^2 = 1$: perfect prediction
$R^2 = 0$: no better than mean

RMSE:

\[ \text{RMSE} = \sqrt{\frac{1}{n-2}\sum (y_i - \hat{y}_i)^2} \]

Typical: $\approx 0.5\%$ for daily returns
Lower is better

Key takeaway

R$^2$ = proportion explained. RMSE = typical prediction error. Always report both.

Compute Train and Test Metrics

Key takeaway

If train and test R$^2$ are close, the model generalises well — the CAPM beta estimated on past data is stable.

Overfitting vs. Underfitting

	Underfitting	Overfitting
Cause	Model too simple	Model too complex
Train R$^2$	Low	High
Test R$^2$	Also low	Much lower than Train
Fix	Add predictors / nonlinear terms	Remove predictors / regularise

Three regimes:

Underfitting (too simple): flat line misses the trend
Good fit (just right): gentle smooth trend
Overfitting (too complex): wiggly curve chasing noise

Detecting Overfitting: Train vs. Test Gap

As model complexity grows, training error keeps falling but test error follows a U-shape — falling, then rising. The “sweet spot” is the minimum test error.

Important

Warning signs of overfitting:

Train R$^2$ much higher than Test R$^2$
Adding more variables helps Train but hurts Test
Many insignificant predictors in the model

Key takeaway

The goal is the sweet spot: complex enough to capture real patterns, simple enough to generalise. Adjusted R$^2$ and cross-validation help find it.

6. Assumptions & Residual Analysis

Learning Goals

Know the 4 assumptions of linear regression
Use residual plots and Q-Q plots to check assumptions visually
Understand what assumption violations mean for CAPM

The LINE Assumptions

Regression is trustworthy when residuals ($e_i = y_i - \hat{y}_i$) satisfy:

L Linearity — $E[\epsilon \mid x] = 0$
I Independence — residuals uncorrelated
N Normality — $\epsilon \sim N(0, \sigma^2)$
E Equal variance — $\text{Var}(\epsilon_i) = \sigma^2$ for all $i$

Important

For stock returns: I often violated (volatility clustering). N: daily returns have fat tails.

Assumption L: Linearity — Residuals vs. Fitted

Good	Bad
Random scatter around 0	Curved (U-shape) pattern

Key takeaway

How to read: Plot residuals ($e$) against fitted values ($\hat{y}$). Want a random cloud around 0. A curve means the linear model misses a nonlinear pattern — consider adding polynomial or transformed terms.

Assumption I: Independence — Residuals Over Time

Good	Bad
Random fluctuations	Volatility clustering (bunches of large residuals)

Key takeaway

How to read: Plot residuals against time. Want no patterns or clusters. Bunches of large residuals = volatility clustering (very common in daily stock returns). Runs of same sign = autocorrelation. Both violate independence $\Rightarrow$ standard errors are unreliable.

Assumption N: Normality — Q-Q Plot & Histogram

Good	Bad
Points on the diagonal line	S-shape = fat tails

Key takeaway

How to read: In a Q-Q plot, each point compares a sample quantile to the corresponding normal quantile. Points on the diagonal = normal. Deviations at the tails = heavy/fat tails (common in daily stock returns). Matters most for small samples; large $n$ benefits from the CLT.

Assumption E: Equal Variance — Scale-Location Plot

Good (Homoscedastic)	Bad (Heteroscedastic)
Constant band width	Fan shape — $\sigma$ grows

Key takeaway

How to read: Residuals should have the same spread across all fitted values. A funnel/fan shape means variance depends on $x$ (heteroscedasticity) $\Rightarrow$ OLS standard errors are biased $\Rightarrow$ $p$-values and CIs are unreliable. Fix: use robust (HC) standard errors or transform $y$.

Checking Assumptions with Python

LINE Diagnostics — CAPM Output

Four-panel diagnostic plot: residuals vs fitted (top-left), residuals over time (top-right), Q-Q plot (bottom-left), scale-location (bottom-right).

What Happens When Assumptions Fail?

Violated	Effect on Inference	Consequence
L Linearity	$\hat{\beta}$ is biased; model misses pattern	Predictions systematically wrong; tests test the wrong model
I Independence	SEs are wrong (usually too small)	$p$-values too optimistic; CIs too narrow
N Normality	$t$/$F$-tests are approximate	Unreliable in small samples; OK for large $n$ (CLT)
E Equal var.	OLS inefficient; SEs biased	Wrong CI width; $p$-values unreliable

Important

L and I are the most damaging: they make the estimates themselves wrong or the uncertainty estimates wrong. Always check these first.

Key takeaway

N and E are less severe for large $n$ (CLT helps). Always check L and I first — they are the hardest to fix.

Assumptions Hold vs. Fail: The Big Picture

Assumptions Hold	Assumptions Fail
$\hat{\beta}$ is unbiased	L fails $\Rightarrow$ $\hat{\beta}$ biased
SEs are correct	I or E fails $\Rightarrow$ SEs wrong
$t$-tests & $F$-tests are exact	N fails $\Rightarrow$ tests approximate
CIs have correct coverage	CIs may under/over-cover
Predictions are optimal	OLS no longer best estimator

Key takeaway

OLS always gives numbers — but those numbers are only trustworthy if the assumptions approximately hold. Diagnostic plots let you check before trusting the output.

7. Multiple Linear Regression: Fama-French

Why Do We Need Multiple Regression?

CAPM uses one factor (market return). But NVDA’s return may also depend on:

Firm size — small vs large cap
Value vs growth orientation
Profitability and investment

CAPM $R^2 \approx 33\%$ — the market alone leaves 67% unexplained. Can we do better?

Important

Omitted variable bias: if a missing variable correlates with both $X$ and $Y$, $\hat{\beta}$ is biased.

Key takeaway

MLR controls for multiple factors, giving each $\beta_j$ a “holding all else constant” interpretation. CAPM → FF5.

Learning Goals

Understand the multiple linear regression model $y = \alpha + \beta_1 x_1 + \cdots + \beta_k x_k + \epsilon$
Interpret each coefficient as “holding other variables constant”
Know why and when to use Adjusted R$^2$ instead of R$^2$
Apply MLR to the Fama-French 5-Factor Model

Multiple Linear Regression (MLR)

Simple regression uses one predictor. In practice, outcomes depend on many variables:

\[ y = \alpha + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k + \epsilon \]

$\alpha$ = intercept: predicted $y$ when all $x_j = 0$
$\beta_j$ = partial slope: change in $y$ per unit change in $x_j$, holding all other $x$’s constant
$\epsilon$ = error (same LINE assumptions as SLR)

Conceptual diagram: $x_1, x_2, \ldots, x_k$ each carry a weight $\beta_j$ into the prediction $\hat{y}$.

Interpreting Coefficients: “Ceteris Paribus”

The key difference from SLR: each $\beta_j$ is a partial effect:

Why it matters

Example: Suppose we predict house price:

$\hat{y} = 50 + 0.1 \times \text{sqft} + 15 \times \text{bedrooms}$

$\beta_{\text{sqft}} = 0.1$: each extra sq ft adds $100, holding bedrooms constant.

$\beta_{\text{bed}} = 15$: each extra bedroom adds $15k, holding sqft constant.

Important

Without “holding constant”, $\beta_j$ in MLR is not the same as $\beta$ in SLR.

Adding correlated predictors changes all coefficients — this is not a bug, it is controlling for confounders.

Fitting MLR in Python

# General MLR: identical syntax -- just pass multiple columns
# y = b0 + b1*x1 + b2*x2 + ... + bk*xk

X = sm.add_constant(df[["x1", "x2", "x3"]])  # k predictors
model = sm.OLS(y, X).fit()
print(model.summary())

# Key outputs:
# model.params        -> coefficients (beta_0, ..., beta_k)
# model.pvalues       -> p-value for each beta_j
# model.rsquared_adj  -> Adjusted R-squared
# model.conf_int()    -> 95% CIs for all betas

Key takeaway

The code is identical to SLR — just pass a DataFrame with $k$ columns instead of one. OLS handles everything.

Live Demo: Predicting Pharmacy Profit

Business question: A pharmacy chain has 111 branches across US metro areas. What local demographics predict per-branch profit?

Predictors (6 features): Income, Disposable Income, Birth Rate, Soc Security, CV Death, % 65 or Older.

Target: Profit per branch.

Why this dataset?

Real business case, small enough to read by eye, large enough to need multiple regression. Profit depends on multiple local features — no single one tells the full story.

Fit the Pharmacy Multiple Regression

After running, look at:

R-squared — overall fit
Adj. R-squared — penalised for k
Coefficients under coef — direction & magnitude
P>|t| — which features are statistically significant
F-statistic — joint significance of all predictors

Interpreting the Pharmacy Model

Reading the coefficients

Positive coef = feature increases profit, holding others constant
Negative coef = feature decreases profit
Insignificant (p > 0.05) = not enough evidence this feature matters once others are accounted for

Try it!

Drop the insignificant features and refit. Does Adjusted R² go up or down? What does that tell you about model parsimony?

Predict Profit for a New Branch

Suppose a new branch opens in a city with these demographics. What does the model predict?

CI vs PI for the business

CI for mean — “If we opened many branches with these demographics, average profit would be in this range.” Used for portfolio decisions.

PI for individual — “This specific branch’s profit will be in this range.” Wider — more cautious for single-branch decisions.

R$^2$ vs. Adjusted R$^2$

Problem: R$^2$ always increases when you add variables — even useless ones!

R$^2$:

\[ R^2 = 1 - \frac{SSE}{SST} \]

Adding any variable can only decrease SSE, so R$^2$ never drops.

Adjusted R$^2$:

\[ \bar{R}^2 = 1 - \frac{(1-R^2)(n-1)}{n-k-1} \]

Penalises for $k$. Can decrease if a new variable does not help enough.

Key takeaway

Use $\bar{R}^2$ to compare models with different numbers of predictors.

Multicollinearity

Problem: When predictors are correlated with each other, OLS still works but coefficients become unstable.

Symptoms:

Large standard errors $\Rightarrow$ wide CIs
Coefficients flip sign or change wildly when a variable is added/removed
High R$^2$ but many insignificant $p$-values

Variance Inflation Factor:

\[ \text{VIF}_j = \frac{1}{1 - R_j^2} \]

where $R_j^2$ = R$^2$ from regressing $x_j$ on all other $x$’s.

VIF	Interpretation
$< 5$	OK
$5$–$10$	Moderate
$> 10$	Severe

Important

Multicollinearity does not bias predictions — it only inflates uncertainty around individual $\beta_j$’s.

Detecting Multicollinearity in Python

from statsmodels.stats.outliers_influence import (
    variance_inflation_factor)

# Compute VIF for each predictor
X_vif = sm.add_constant(df[["x1", "x2", "x3"]])
for i, col in enumerate(["x1", "x2", "x3"]):
    vif = variance_inflation_factor(X_vif.values, i + 1)
    print(f"{col}: VIF = {vif:.2f}")

Key takeaway

Rule of thumb: If any VIF $> 10$, consider dropping or combining the offending variable. VIF $< 5$ is generally safe.

Understanding $R_j^2$ in VIF — A Simple Example

Setup: 3 predictors — $x_1$ (age), $x_2$ (income), $x_3$ (credit score). To get VIF for $x_2$: regress $x_2$ on $x_1, x_3$, get $R_2^2$, then $\text{VIF}_2 = 1/(1 - R_2^2)$.

Predictor	$R_j^2$	VIF	Status
$x_1$ (age)	0.10	1.11	OK
$x_2$ (income)	0.85	6.67	Warn
$x_3$ (credit)	0.82	5.56	Warn

Why it matters

$R_j^2 = 0.85$: 85% of income explained by others $\Rightarrow$ redundant. $R_j^2 = 0.10$: age is independent of others.

Key takeaway

High $R_j^2$ $\Rightarrow$ $x_j$ well-predicted by others $\Rightarrow$ high VIF $\Rightarrow$ inflated $SE(\hat{\beta}_j)$. Fix: drop or combine correlated predictors.

Application: From CAPM to Fama-French

CAPM: one factor explains $$33%. What else matters?

Fama-French 5-Factor Model:

Mkt-RF: Market excess return
SMB: Small-minus-Big (size)
HML: High-minus-Low (value)
RMW: Robust-minus-Weak (profit)
CMA: Conservative-minus-Aggressive

\[ R - R_f = \alpha + \beta_1 \text{Mkt} + \beta_2 \text{SMB} + \beta_3 \text{HML} + \beta_4 \text{RMW} + \beta_5 \text{CMA} + \epsilon \]

Why it matters

Each $\beta_j$ captures a different type of systematic risk. Together they explain more than the market alone.

What Each Factor Means

Factor	Long / Short	Intuition
Mkt-RF	Broad market	Core market risk
SMB	Small / large caps	Small firms tend to outperform
HML	High B/M / low B/M	Value beats growth
RMW	High / low profit	Profitable firms earn more
CMA	Low / high investment	Conservative firms outperform

Key takeaway

For tech growth (NVDA): expect large $\beta_1$, negative $\beta_2$ (large cap), negative $\beta_3$ (growth $\ne$ value).

Load Fama-French Factors

import pandas_datareader.data as web

ff = web.DataReader("F-F_Research_Data_5_Factors_2x3_daily",
    "famafrench", start="2023-01-01", end="2024-12-31")
ff_factors = ff[0] / 100   # percent -> decimal

data_ff = ret[["NVDA"]].join(ff_factors, how="inner")
data_ff["NVDA_excess"] = data_ff["NVDA"] - data_ff["RF"]
print(data_ff.head(3))

Why it matters

pandas_datareader pulls the official Fama-French factors from Ken French’s data library.

Build the Fama-French Regression

Key takeaway

The code is identical to simple regression — just pass a list of five factor columns. statsmodels handles everything.

Interpreting the Factor Loadings

Factor	$\beta$	p-value	Meaning
const ($\alpha$)	0.0006	0.56	No significant alpha
Mkt-RF	1.84	0.000	High market sensitivity
SMB	$-0.32$	0.03	Behaves like large-cap
HML	$-0.65$	0.000	Strong growth tilt
RMW	$-0.48$	0.01	Lower vs. high-profit
CMA	$-0.22$	0.18	Not significant

Key takeaway

Each $\beta_j$: “holding other factors constant, a 1-unit increase in this factor changes NVDA excess return by $\beta_j$.”

Testing Factor Significance

Individual $t$-test (same as SLR):

\[ H_0: \beta_j = 0 \quad\text{vs}\quad H_a: \beta_j \ne 0 \qquad t_j = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)} \;\sim\; t_{n-k-1} \]

Overall $F$-test (are any factors useful?):

\[ H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0 \qquad F = \frac{SSR/k}{SSE/(n-k-1)} \;\sim\; F_{k,\,n-k-1} \]

Key takeaway

$F$-test significant $\Rightarrow$ at least one factor matters. Individual $t$-tests tell you which ones. CMA ($p = 0.18$): candidate for dropping.

Multicollinearity Check for FF5 Factors

Why it matters

FF5 factors are constructed to be approximately orthogonal, so VIFs are typically low. But HML and CMA can be correlated (both relate to firm investment).

Key takeaway

If VIF $> 10$: drop or combine. Here all VIFs are low $\Rightarrow$ multicollinearity is not a problem for NVDA’s factor model.

CAPM vs. FF5: Does MLR Help?

Key takeaway

Compare using Adjusted R$^2$ (not R$^2$, which always increases). If FF5 $\bar{R}^2 >$ CAPM $\bar{R}^2$, the extra factors genuinely add explanatory power beyond the market alone.

CI for $\mu_Y$ and Prediction Interval in MLR

The SLR formulas extend naturally to MLR. For a new observation $\mathbf{x}^* = (x_1^*, \ldots, x_k^*)$:

CI for $\mu_Y \mid \mathbf{x}^*$

Average NVDA return on all days with these factor values. Width depends on $SE(\hat{\mu})$ only — shrinks with more data.

Prediction Interval for $Y$

Tomorrow’s NVDA return given these factor values. Adds $\sigma^2$ for individual noise — always wider than CI.

8. Variable Selection

Learning Goals

Understand the bias-variance tradeoff: why adding variables is not always better
Apply the overall F-test to assess model significance
Perform best subset selection with AIC/BIC/Adj-R$^2$/C$_p$
Apply backward elimination as a practical greedy alternative

Overall F-Test: Is the Model Significant?

Test whether any predictor is useful:

\[ H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0 \quad\text{vs}\quad H_a: \text{at least one } \beta_j \neq 0 \]

\[ F = \frac{SSR/k}{SSE/(n-k-1)} = \frac{R^2/k}{(1-R^2)/(n-k-1)} \;\sim\; F_{k,\,n-k-1} \]

Key takeaway

The F-test is in every model.summary(). An insignificant F (p $>$ 0.05) means none of your variables explains the response — the model is useless.

Best Subset Selection: The Idea

Goal: Find the single best model of each size $k = 1, 2, \ldots, p$.

Algorithm:

For each subset size $k$, enumerate all $\binom{p}{k}$ models
Keep the best model of each size (lowest SSE) $\Rightarrow$ $p$ candidate models
Select among candidates using a criterion that penalises complexity: AIC, BIC, Adj-R$^2$, or Mallow’s $C_p$

Visual: plot every candidate model’s Adj $R^2$ vs $k$; an upper-envelope “frontier” gives the best at each size, and the optimal complexity is the peak of that frontier.

Best Subset with `isom2600.regression`

import isom2600

# Prepare data: X must be a DataFrame, Y a Series
X = train_ff[factors]     # 5 Fama-French factors
Y = train_ff["NVDA_excess"]

# Stage 1: best model of each size (by SSE)
df    = isom2600.regression.best_subset_init(X, Y)
df_s1 = isom2600.regression.best_subset_stage1(df)

# Stage 2: pick best overall by each criterion
best_adjR2 = isom2600.regression.best_subset_stage2(df_s1, "Adj_R2")
best_AIC   = isom2600.regression.best_subset_stage2(df_s1, "AIC")
best_BIC   = isom2600.regression.best_subset_stage2(df_s1, "BIC")

print("Adj R2 selects:", best_adjR2)
print("AIC    selects:", best_AIC)
print("BIC    selects:", best_BIC)

Why it matters

BIC penalises complexity more than AIC — it tends to pick a sparser model. They may agree (good) or disagree (check both).

Model Selection Criteria

Recall: $SSE = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2$ (sum of squared residuals), $k$ = number of predictors.

Criterion	Formula	Penalises	Select
Adj R$^2$	$1 - \tfrac{(1-R^2)(n-1)}{n-k-1}$	Weak	Maximise
AIC	$n\ln(\tfrac{SSE}{n}) + 2k$	Moderate	Minimise
BIC	$n\ln(\tfrac{SSE}{n}) + k\ln n$	Strong	Minimise
$C_p$	$\tfrac{SSE}{\hat\sigma^2} - n + 2k$	Moderate	$C_p \approx k$

Key takeaway

AIC optimises prediction accuracy. BIC optimises model identification (consistent). For small samples ($n < 40$) use BIC. For forecasting use AIC (or cross-validation).

Reading the Criteria Table

What does “Penalises” mean? Each criterion has a fit term (always improves with more $x$’s) and a penalty term that punishes complexity. “Penalises” rates how strongly that penalty pushes back.

Why “Select: $C_p \approx k$”?

Theorem: if the model is correctly specified, $E[C_p] \approx k$. So we look for $C_p$ close to its own $k$:

$C_p \gg k$ $\Rightarrow$ missing important variables (biased)
$C_p \approx k$ $\Rightarrow$ well-specified
$C_p < k$ $\Rightarrow$ possibly overfitting

What does “consistent” mean for BIC?

As $n \to \infty$, BIC picks the true model with probability $\to 1$. AIC does not.

Why? BIC’s penalty $k\ln n$ grows with $n$, eventually crushing any spurious variable. AIC’s $2k$ stays fixed.

Trade-off: AIC is efficient (best predictions); BIC is consistent (right model). No criterion is both.

Key takeaway

For forecasting NVDA returns → AIC. For which factors really drive returns (inference) → BIC.

Model Comparison: CAPM vs. FF4 vs. FF5

Model	$k$	Train Adj R$^2$	Test R$^2$	BIC rank
CAPM	1	0.543	0.521	3
FF4	4	0.582	0.563	1
FF5	5	0.581	0.558	2

FF4 wins on all criteria
Dropping CMA improves test R$^2$ (less overfitting)
CAPM is too simple (high bias)

Key takeaway

Best model = FF4. Extra factors add variance without reducing bias. Occam’s razor confirmed by data.

9. Business Applications

Learning Goals

Use the trained CAPM model to predict NVDA returns
Test whether NVDA earns significant alpha
Extend the workflow to any stock

Application 1: Predict NVDA Return on a New Day

Scenario: Market up 1.5% — what do we expect from NVDA?

Why it matters

With $\beta \approx 2.33$, on a $+1.5\%$ market day NVDA is predicted to gain $\approx +3.5\%$.

Application 2: Test for Significant Alpha

Question: Does NVDA consistently outperform the Fama-French model?

Key takeaway

A significant $\alpha$ means NVDA has genuine excess returns beyond its factor exposures. In an efficient market, this is rare.

Try It! — Analyse Your Own Stock

Try it!

Pick any stock (AAPL, TSLA, MSFT, AMD) and:

Download returns with yfinance
Load FF5 factors with pandas_datareader
Fit CAPM and FF5 models
Compare Adjusted R$^2$ — do extra factors help?
Is there significant alpha?
Aggressive ($\beta>1$) or defensive ($\beta<1$)?

Hint: Start with tickers = ["AAPL", "SPY"] and reuse the pipeline.

Takeaway — The Regression Pipeline

Pipeline: Explore → Visualize → Build → Evaluate → Select → Apply

Step	Tool
Explore	`.corr()`
Visualize	`regplot`
Build	`sm.OLS`
Evaluate	R$^2$, RMSE
Select	p-values, AIC/BIC
Apply	`.predict()`

Key takeaway

SLR = CAPM: one market factor, $\beta$ measures risk exposure.

MLR = Fama-French: five factors explain more variation in returns.

Variable selection: only keep factors that are statistically significant.

Next: Topic 4 — Clustering (K-Means & Hierarchical).

\(r\) value	Interpretation
\(-1\)	Perfect negative linear
\(-0.5\)	Strong negative
\(0\)	No linear association
\(+0.5\)	Strong positive
\(+1\)	Perfect positive linear

	coef	std err	t	P>\|t\|
`const` (\(\alpha\))	0.0036	0.001	2.71	0.007
`Mkt_excess` (\(\beta\))	2.332	0.166	14.04	0.000
\(R^2\)	0.331

	coef	std err	t	P>\|t\|	[0.025, 0.975]
const (\(\alpha\))	0.0036	0.001	2.706	0.007	0.001, 0.006
Mkt_excess (\(\beta\))	2.3320	0.166	14.038	0.000	2.005, 2.659

Violated	Effect on Inference	Consequence
L Linearity	\(\hat{\beta}\) is biased; model misses pattern	Predictions systematically wrong; tests test the wrong model
I Independence	SEs are wrong (usually too small)	\(p\)-values too optimistic; CIs too narrow
N Normality	\(t\)/\(F\)-tests are approximate	Unreliable in small samples; OK for large \(n\) (CLT)
E Equal var.	OLS inefficient; SEs biased	Wrong CI width; \(p\)-values unreliable

Assumptions Hold	Assumptions Fail
\(\hat{\beta}\) is unbiased	L fails \(\Rightarrow\) \(\hat{\beta}\) biased
SEs are correct	I or E fails \(\Rightarrow\) SEs wrong
\(t\)-tests & \(F\)-tests are exact	N fails \(\Rightarrow\) tests approximate
CIs have correct coverage	CIs may under/over-cover
Predictions are optimal	OLS no longer best estimator

Predictor	\(R_j^2\)	VIF	Status
\(x_1\) (age)	0.10	1.11	OK
\(x_2\) (income)	0.85	6.67	Warn
\(x_3\) (credit)	0.82	5.56	Warn

Criterion	Formula	Penalises	Select
Adj R\(^2\)	\(1 - \tfrac{(1-R^2)(n-1)}{n-k-1}\)	Weak	Maximise
AIC	\(n\ln(\tfrac{SSE}{n}) + 2k\)	Moderate	Minimise
BIC	\(n\ln(\tfrac{SSE}{n}) + k\ln n\)	Strong	Minimise
\(C_p\)	\(\tfrac{SSE}{\hat\sigma^2} - n + 2k\)	Moderate	\(C_p \approx k\)

VIF	Interpretation
\(< 5\)	OK
\(5\)–\(10\)	Moderate
\(> 10\)	Severe

Topic 3: Linear Regression

Why Predict?

Today’s Roadmap

1. Visualizing Association

Learning Goals

Correlation Coefficient

Download Data & Compute Correlation

Alternative: Binder (live Python, embedded)

Simulated NVDA vs SPY in Pyodide

Scatter Plot: NVDA vs. SPY

2. Simple Linear Regression: CAPM

Learning Goals

Simple Linear Regression Model

The Capital Asset Pricing Model (CAPM)

Prepare the Data

Fit the CAPM with statsmodels

Reading the CAPM Output

Variation Decomposition

3. Inference on Regression Coefficients

Learning Goals

LINE Assumptions for Inference

Sampling Distributions of \(\hat{\alpha}\) and \(\hat{\beta}\)

Why Do Standard Errors Matter?

model.summary() Output

\(t\)-Tests for Slope and Intercept

What Is a \(p\)-Value? (Two-Tailed)

One-Tailed \(p\)-Value

Testing \(\alpha > 0\) and \(\beta > 1\) in CAPM

Critical Value Approach

Confidence Intervals for \(\alpha\) and \(\beta\)

Reading the Statsmodels CI Output

Try It! — Inference on CAPM

4. CI for \(\mu_Y\) vs. Prediction Interval

Learning Goals

Two Questions, Two Intervals

The Formulas — One Extra Term

Why PI Is Always Wider

Python: get_prediction() for Both Intervals

Plot Both Bands Together

Business Interpretation

5. Train/Test Split & Model Evaluation

Learning Goals

Why Split the Data?

R\(^2\) and RMSE

Compute Train and Test Metrics

Overfitting vs. Underfitting

Detecting Overfitting: Train vs. Test Gap

6. Assumptions & Residual Analysis

Learning Goals

The LINE Assumptions

Assumption L: Linearity — Residuals vs. Fitted

Assumption I: Independence — Residuals Over Time

Assumption N: Normality — Q-Q Plot & Histogram

Assumption E: Equal Variance — Scale-Location Plot

Checking Assumptions with Python

LINE Diagnostics — CAPM Output

What Happens When Assumptions Fail?

Assumptions Hold vs. Fail: The Big Picture

7. Multiple Linear Regression: Fama-French

Why Do We Need Multiple Regression?

Learning Goals

Multiple Linear Regression (MLR)

Interpreting Coefficients: “Ceteris Paribus”

Fitting MLR in Python

Live Demo: Predicting Pharmacy Profit

Fit the Pharmacy Multiple Regression

Interpreting the Pharmacy Model

Predict Profit for a New Branch

R\(^2\) vs. Adjusted R\(^2\)

Multicollinearity

Detecting Multicollinearity in Python

Understanding \(R_j^2\) in VIF — A Simple Example

Application: From CAPM to Fama-French

What Each Factor Means

Load Fama-French Factors

Build the Fama-French Regression

Interpreting the Factor Loadings

Testing Factor Significance

Multicollinearity Check for FF5 Factors

CAPM vs. FF5: Does MLR Help?

Fit the CAPM with `statsmodels`

`model.summary()` Output

Python: `get_prediction()` for Both Intervals

Best Subset with `isom2600.regression`