2.5 Model Development

2.5.1 Model Development

• Data Preparation:
• Feature Engineering: New binary variables (isFemale)
• Model Development: Linear Regression comparing different predictors
• Model Evaluation / Prediction: on test-set

Data Preparation

• Load the data
• Clean the data
• Make sure what question You want to answer
• Make plots to get an understanding of the data

Feature Engineering

• Create binary variables from categories
• Normalize and scale variables is necessary
• Create new features

Model Development

• Split the data in training and test or decide for another resampling approach
• Decide what models could work
• Decide how You want to measure if a model is good
• Train models and compare their accuracy on the validation-set

Model Evaluation / Prediction

• Evaluate the models on a unseen test-set
• Evaluate strength and weaknesses of the best models

2.5.2 Resampling Methods

Learning objectives

You will be able to

• what questions to evaluate based on training, validation and test-sets
• perform LOO and k-fold cross-validation to create models that do not over-fit the training data

Model & Feature Selection

• we want to find the best model for the data
• models can differ in
  • predictors / features that go into the model
  • model type (linear regression, decision tree, etc...)
  • models power to over-fit
  • hyper parameters (parameters, that tweak the model)

• first, we focus in the the predictors / features

Finding the best predictors / features

• We have $6$ possible predictor variables in the data set to predict body_mass_g:
  • species, island, bill_length_mm, bill_depth_mm, flipper_length_mm, sex

• Encoding the categorical variables binary we have $p=8$ possible predictors
  • isAdelie, isGentoo , fromTorgersen, fromBiscoe, bill_length_mm, bill_depth_mm, flipper_length_mm, isFemale
• from that we can build ${2^p} = {2^8}= 256$ possible linear models:
  • $\text{body mass g}=\beta_0$
  • $\text{body mass g}=\beta_0 + \beta_1 \times \text{isAdelie}$
  • $\text{body mass g}=\beta_0+ \beta_2 \times \text{isGentoo}$
  • ...

What we found

• It is hard to predict what is the best model in advance
• We compared models $M_i$ in different ways
  • Fit on the training set ($R^2$, $RSS$)
  • Prediction accuracy
    • on the training
    • and test-set ($MSE$)
  • Evaluation only on the training set can lead to over fitting

Over-fitting the Training Data

• What we really want to know: What will be the best model on new unseen data?
  • We need to select models on a set that is not the training data
  • After selecting a model, we still want to test how well it performs on unseen data
• validation-set
  • A set on which we compare trained models to find
  • the best features and ML Algorithms for the task

We must divide the data into three parts

https://www.sxcco.com/?category_id=4128701

How do we split the data?

• Often, we do not have much data, but ...
• if we choose a
• small training set
  • We only use a small proportion of the data to learn
  • If we have powerful models on sparse data, they tend to over-fit
• small validation / test-set
  • set can be very special by chance
  • misleading results

https://www.sxcco.com/?category_id=4128701

The Validation-Set Approach

• split the data once
• $70\%$ / $15\%$ / $15\%$ are common proportions

• note, that we can shuffle the folds randomly - train is not the first three!

Problems with the Validation-Set Approach

• test-set $MSE$ vs polynomial complexity of the model
• Left: test-set $MSE$ one split, Right: test-set $MSE$ with ten different splits

• Results depend on how we split the data

Leave-One-Out Cross-Validation (LOO)

• only put one observation in the test or validation-set
• repeat this for all observations

Averaging the Cross-Validation results

• With $n$ observations in the original data set
• We get $n$ models $M_l$ with different prediction errors on the validation-sets
• We can still calculate the $MSE$ the same way
• $MSE^{CV}=\frac{1}{n}\sum_{l=1}^n MSE_l$

k-fold Cross-Validation

• instead of $n$ folds with length $1$, the data is split in $k$ folds of equal length

https://medium.com/coders-mojo/quick-recap-most-important-projects-data-science-machine-learning-programming-tricks-and-c7d99a7a2391

Averaging the Cross-Validation results

• With $k$ folds
• we get $k$ different $MSE_l$s
• $MSE^{kCV}=\frac{1}{k}\sum_{l=1}^k{MSE_l}$

What to use?

• Leave-One-Out (LOO) is a special case of n-folds ($n=k$)
• LOO is computationally more expensive
• LOO has a smaller bias (we use more data)
• LOO has a higher variance (the models we train are all
  very similar, while the test-set very different)
• using kFolds with $k = 5$ or $k = 10$ has been shown empirically
  to yield good results

A practical approach to Cross-Validation

https://scikit-learn.org/stable/modules/cross_validation.html

• note that the data should be shuffled before the split
• there are some cases, where this will not work (e.g. temporal data)

2.5.3 Feature Selection

• given we have ${2^p} = {2^8}= 256$ possible linear models:
  • $\text{body mass g}=\beta_0$
  • $\text{body mass g}=\beta_0 + \beta_1 \cdot \text{isAdelie}$
  • $\text{body mass g}=\beta_0+ \beta_2 \cdot \text{isGentoo}$
  • ...
• how can we reliably find the best one?
• we use 5-fold cross-validation to find the best features (predictors) and parameters

Option 1: Best Subset Selection

• brute force: train all possible models on the training set
• compare the models performance on the validation-set
• select the best model (e.g, based on validation $MSE$)
• With 5-fold cross-validation You will train $2^8 \cdot 5$ different models

Option 2: Forward Selection

• start he the null model ($\text{body mass g}=\beta_0$)
• calculate the best model with only one predictor $X_a$
• calculate the best model containing $X_a$ and one further predictor
• ...

https://medium.com/codex/what-are-three-approaches-for-variable-selection-and-when-to-use-which-54de12f32464

Option 3: Backward Selection

• inversion of the forward selection
• start with all predictors
• drop the predictor with the smallest decrease in model accuracy

More Options:

• Automated Feature Selection
• Regularization can shrink unimportant parameters

Case Study

Try to find the best possible linear model for predicting the penguin weight using

• best subset selection and
• cross-validation

https://www.scoopnest.com/user/AFP/1035147372572102656-do-you-know-your-gentoo-from-your-adelie-penguins-infographic-on-10-of-the-world39s-species-after

5.1 Cross-Validation and Model Selection - Resampling

40 minutes

Learning Summary

Now, You can

• what questions to evaluate based on training, validation and test-sets
• perform LOO and k-fold cross-validation to create models that do not over-fit the training data