2.3 Linear Regression

Learning objectives

You will be able to

• Students can implement a Linear Regression in Python
• Students can interpret coefficients and p-values of an regression model
• Students describe the accuracy of an model using $R^2$

Motivation

• Linear Regression is supervised and parametric
• Interpretation:
  • Is there a relationship between advertising budget and sales?
  • How strong is the relationship between advertising budget and sales?

• Interpretation:
  • Which media contribute to sales?
  • How accurately can we estimate the effect of each medium on sales?
  • How accurately can we predict future sales?
  • Is the relationship linear?
  • Is there synergy among the advertising media?

Simple Linear Regression

• coefficients or parameters:

  • $β_0$ ... intercept
  • $β_1$ ... slope

• We estimate the parameters based on training data

Estimating the Coefficients

• model: $\hat{y}_j = \hat{β}_0 + \hat{β}_1 \cdot x_{1,j}$
• prediction error: $e_j = y_j−\hat{y}_j$
• residual sum of squares (RSS) over all observations
  $RSS = e^2_1 + e^2_2 + · · · + e^2_n$
• best model: $\min(RSS)$

What is the relationship between $MSE$ and $RSS$?

• Mean Squared Error
  $MSE=\frac{1}{n}\sum_{j=1}^n(y_j−\hat{y}_j)^2$
• Residual Sum of Squares
  $RSS= \sum_{j=1}^n(y_j−\hat{y}_j)^2$

There is an analytical solution

• to find the parameters that have the minimal $RSS$ given the training data ($X,Y$)

• $\bar{x} = \frac{1}{n}\sum_{j=1}^n x_{1,j}$
• $\bar{y} = \frac{1}{n}\sum_{j=1}^n y_{j}$

You can find the proof in any textbook or wikipedia. Later, we will discuss alternative algorithm to find the solution

Same Data, different Solutions

• red: true relationship of the simulated data
• grey: random sample drawn from the true relationship
• blue: estimated regressions lines based on different samples (training sets)

• How sure can we be about the values of $\beta_0$ and $\beta_1$?

Standard Error $SE(\beta), \sigma(\beta)$

• is a measure for the average amount that an estimate (e.g., $\hat{\beta}_0$) differs from the actual value of $\beta_0$ and depends on the variance $σ^2$ in the data and the number of observations $n$ on the sample
• where $σ$ is the standard deviation of each of the realizations $y_j$ of $Y$
• there is approximately a 95% chance that the interval
  will contain the true value of $β_i$. (given a large enough sample size $n$ and normal distributed errors)
• This is used as a confidence interval.

Normal Distribution

Example TV advertising data

• $95 \%$ confidence interval for $\hat{β}_0$ $[6.130, 7.935]$

• $95 \%$ confidence interval for $\hat{β}_1$ $[0.042, 0.053]$

• without any advertising, sales will, on average, fall between $6{,}130$ and $7{,}940$ units.

• for each $1{,}000$ $ increase in TV advertising, there will be an average increase in sales of between $42$ and $53$ units.

Hypothesis Tests in on Parameters

Is there really a influence of advertising on sales?

• $H_0$: There is no relationship between $X$ and $Y$
  $H_0: β_1 = 0$
• $H_a$: There is some relationship between $X$ and $Y$
  $H_a: β_1 \ne 0$

Is $\hat{β}_1$ far enough from $0$ that we can reject $H_0$?

• From the data we know / can calculate:
  • $\hat{β}_1 = 0.0475$
  • $SE(\hat{\beta}_1)=0.0027$
  • So, we are pretty sure that the real $\beta_1$ is in $[0.042,0.053]$
• how sure are we?
  • based on the $SE$ more then $95\%$
  • We can use a t-Test to calculate, how likely it is have a $\beta_1$ of $0$ given our data:

https://www.geogebra.org/m/b85v7zww, $df = n-p-1$

• the probability of observing a $t>17.6$ or $t<17.6$, given the real $\beta_1=0$ is very low
• we call this probability the $p$-value
• a small p-value indicates that it is unlikely to observe such a substantial association between the predictor and response due to chance
• We reject the null hypothesis $H_0: β_1 = 0$ — that is, we declare a relationship to exist between $X$ and $Y$ — if the $p$-value is small enough (usually $5\%$)

Regression Tables

After fitting the model, the models parameters can be found in a regression table.

• $β_0=7.0325$, $SE(\hat{\beta}_0)=0.4578$
• $β_1=0.0475$, $SE(\hat{\beta}_1)=0.0027$
• as $p<0.05$, we say: TV adds have a significant positive correlation with sales

Real Regression Tables

• what is the predicted variable?
• how large is the intercept?
• is there a significant influence of femur length on the predicted variable?

OLS - Ordinary least squares = Regular Linear Regression

Assessing the Accuracy of the Model

Does the model fit the data?

Residual Standard Error

• There will always be an error $\epsilon$, even if we know $β_0$ and $β_1$ perfectly
• Residual Standard Error ($RSE$) is an estimate of the standard deviation of $\epsilon$
• average amount that the response will deviate from the true regression line

• $RSE=0$ ... perfect fit

even if the model was correct and the true values of the unknown coefficients $β_0$ and $β_1$ were known exactly, any prediction of sales on the basis of TV advertising would still be off by about $3.260$ units on average.

$R^2$ Statistic

• $RSE$ provides an absolute measure of the fit (e.g., $3.260$ units)
• $R^2$ provides a proportion between $0$ and $1$

• where $TSS=\sum{(y_j-\bar{y})^2}$ is the total sum of squares
• where $\bar{y}$ is the sample mean

Interpretation of $R^2$

https://www.datasciencecentral.com/wp-content/uploads/2021/10/2742052271.jpg

Be careful!

• no good measure for prediction accuracy!
• $RSS$ or $MSE$ is identical, still $R^2$ is very different
• Especially as it is calculated on the test set

https://www.enmanreg.org/r2/r2_vs_cvrmse/

Case Study

https://ocean.si.edu/ocean-life/seabirds/penguins

• Is there any relationship between two variables (for any species and penguin), where You do not expect a linear correlation?

• If so (or not), is the model a good predictor?

3 Linear Regression

35 minutes