2.7 Classification

2.7.1 Classification - Logit-Regression

Learning objectives

You will be able to

• describe what odds are
• interpret the results of a logistic regression model

Classification: the response variable is qualitative

• Predicted variable $Y$: Will the debitor default
• Predictors:
  • $X_1$: What is their current account balance
  • $X_2$: What is their income

Examples of classifications

• Binary classification: Will the creditor default?
• Multi-class classification: Which species of penguin is it?
  • Any multi-class classification can be broken down into several binary classifications:
    • is it Gentoo?
    • is it Adelie?
    • is it Chinstrap?

https://www.scoopnest.com/user/AFP/1035147372572102656-do-you-know-your-gentoo-from-your-adelie-penguins-infographic-on-10-of-the-world39s-species-after

Examples of multi-class classifications

https://medium.com/@Suraj_Yadav/in-depth-knowledge-of-convolutional-neural-networks-b4bfff8145ab

Logistic Regression Model

Why Not Linear Regression?

• Probability that the creditor will default: $p(Y=1)$
• left: $p(Y=1)=\beta_0+\beta_1 X$
  • negative probability if low balance
  • high balances will result in probability greater than 0

• adaption of the models form
• right: $p(Y)=\frac{e^{\beta_0+\beta_1 X}}{1+e^{\beta_0+\beta_1 X}}$

Solving Logistic Regression

Odds

• are just another way to speaking of probabilities

• The odds of Bayern Munich winning are 2 to 1
  • $\frac{p(Y)}{1-p(Y)}=\frac{2}{1}$
  • It's twice as likely that they win, than tie or loosing
  • $\frac{p(Y)}{1-p(Y)}=\frac{0.66}{1-0.66}$

• The left-hand side is called the log-odds or logit. We see that the logistic regression model has a logit that is linear in $X$.
• the logarithm of the odds is a linear function that can be solved with
  • least squares approach
  • maximum likelihood method
  • gradient descent

Interpretation of the Regression Table

• the logarithm of the odds of having a default ($y=1$)
  is explained with a linear model

• balance ($\beta_1>0$) has a positive effect in the probability to default
• this effect is significant $p\text{-value}<0.05$
• the intercept is harder to interpret

Making a Prediction

• We can just plug in the values of predictors
  and get a probability value (between $0$ and $1$ as a prediction)
• Probability of a default with
  • a balance of $x_1=1000$
    $p(Y)=\frac{e^{\beta_0+\beta_1 X}}{1+e^{\beta_0+\beta_1 X}}=\frac{e^{-10.65+0.0055 \cdot 1000}}{1+e^{-10.65+0.0055 \cdot 1000}}=0.00576$

Task

Given the following Regression Model of the odds of having high blood pressure from Lavie et al. (BMJ, 2000) surveyed $2677$ adults referred to a sleep clinic

• Does the model have an intercept?
• Is the apnoea index significantly predictive of high blood pressure?
• Is sex a predictor of high blood pressure?

data is changed from the original paper, https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1b-statistical-methods/multiple-logistic-regression

• Does the model have an intercept?
  • no
• Is the apnoea index predictive of high blood pressure?
  • no significance as $p>0.05$
• Is sex a predictor of high blood pressure?
  • males have significantly higher odds of having high blood pressure?

2.7.2 Classification - Evaluating Classification Results

Learning objectives

You will be able to

• read a confusion matrix and calculate classification errors based on its results
• set classification thresholds to generate a ROC-curve of a classifier
• compare models using ROC and AUC

No prediction is perfect

• Predicted variable $Y$: Will the debitor default
• Predictors:
  • $X_1$: What is their current account balance

• Probability of a default for the second person with
  • a balance of $x_1=2000$
    $p(X)=\frac{e^{\beta_0+\beta_1 X}}{1+e^{\beta_0+\beta_1 X}}=\frac{e^{-10.65+0.0055 \cdot 2000}}{1+e^{-10.65+0.0055 \cdot 2000}}=0.587$
  • We can decide to predict default based on the threshold probability of $50\%$
  • Still the second person did not default

Confusion Matrix

• making a prediction, we will make errors
• with categorical data we cannot use the accuracy measures from regression

https://towardsdatascience.com/understanding-confusion-matrix-a9ad42dcfd62

• True Positive ($TP$):
  • You predicted positive and it’s true.

• True Negative ($TN$):
  • You predicted negative and it’s true.

• False Positive ($FP$): (Type 1 Error):
  • You predicted positive and it’s false.

• False Negative ($FN$): (Type 2 Error):
  • You predicted negative and it’s false.

Example: Corona-Test

• What error do we want to minimize, when everyone who is sick should stay at home?
  • We want to to find all positives ($P$)
  • False negatives ($FN$) are dangerous

• A threshold is a probability value we set to decide on the predicted classification based on the predicted probability

• There is one false negative that is not detected to have corona with this threshold

Task

• Select a threshold, so that no false negative remains
• Fill the confusion matrix
  15 minutes

• No, we find all positives
• However, we send one more person home as a false positive

Thresholds

• In most classification problem, we have to balance (at least) to different errors
  • False Positive ($FP$): (Type 1 Error )
  • False Negative ($FN$): (Type 2 Error )
• By moving the thresholds, we can calibrate a classifier (that predict a class probability) so that is suits the use case
• To compare classifiers, we can use a receiver operating characteristic

Receiver operating characteristic

• to create a line, we use the same model but change the threshold
• how much more $FP$ do we get, if we increase the $TP$ rate by changing the threshold?
• Sensitivity / Recall:
  $TPR=\frac{TP}{TP+FN} = \frac{TP}{P}$
• $FPR=\frac{FP}{FP+TN}=\frac{FP}{N}$
• the perfect classifier would achieve all $TP$ without any $FP$

Good Classifier

• red: distribution of the predictions for negatives
• green: distribution of the predictions for positives
• only few of the guesses land on the wrong side of the threshold

https://towardsdatascience.com/demystifying-roc-curves-df809474529a

Random Classifier

• the classifier has to power to differentiate between the classes
• no matter where we place the threshold, we will get the same number of $FP$ and $TP$

https://towardsdatascience.com/demystifying-roc-curves-df809474529a

AUC (Area Under the ROC Curve)

• We can measure the area under the ROC-curve to evaluate the skill of a model
• A perfect classifier has a AUC of $1$
• a random classifier has a AUC of $0.5$

https://towardsdatascience.com/understanding-the-roc-curve-and-auc-dd4f9a192ecb

Other Accuracy Measures for Classifiers

• for a model and a given threshold, we have the the confusion matrix to get a first impression of it's performance
• it is helpful to have a single metric to describe a models accuracy

https://en.wikipedia.org/wiki/Precision_and_recall#/media/File:Precisionrecall.svg

Accuracy

What proportion identifications were correct?
How often do we hit the target?

• You don't have to learn the formulas, but should be able to calculate the values if formulas and data are given

Precision

What proportion of positive identifications was actually correct?
Of everything we predict to be positive, how many are really positive?

Sensitivity / Recall

What proportion of actual positives was identified correctly?
How many of the positives do we find?

F1-Score

• The F1 score is the harmonic mean of the precision and recall.

Task

• Calculate accuracy, precision, recall and F1-Score for the following examples

10 minutes

Corona Test I

Corona Test II

Corona Test I

• $\text{accuracy} = \frac{TP + TN}{P + N}= \frac{2 + 2}{3 + 3}=\frac{2}{3}$
• $\text{precision} = \frac{TP}{TP + FP}= \frac{2}{2 + 1}=\frac{2}{3}$
• $\text{recall} = \frac{TP}{TP + FN}=\frac{2}{2 + 1}=\frac{2}{3}$
• $F_1 = 2 \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}=2 \frac{\frac{2}{3} \frac{2}{3}}{\frac{2}{3} + \frac{2}{3}}=\frac{2}{3}$

Corona Test II

• $\text{accuracy} = \frac{TP + TN}{P + N}= \frac{988 + 1}{998 + 2}=0.999$
• $\text{precision} = \frac{TP}{TP + FP}= \frac{1}{1 + 0}=1$
• $\text{recall} = \frac{TP}{TP + FN}=\frac{1}{1 + 1}=\frac{1}{2}$
• $F_1 = 2 \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}=2 \frac{1 \cdot \frac{1}{2}}{1 + \frac{1}{2}}=\frac{2}{3}$

Unbalanced Data Sets

Given this data. Is it hard to train an model with high accuracy (high share of correct predictions)?

• The simple model
  • $P(x=0)=1$
  • always predicts negative

• Datasets, where one group is much more common that the other are called unbalanced
• This can lead to a special case of over-fitting (as in this example)

Training on Unbalanced Data Sets

• a common way to solve this problem is to create a balanced training data set
• depending on the amount of data available
  • under-sampling: omit some instances of the majority class in the training
  • over-sampling: multiply some instances of the minority class in the training

https://medium.com/analytics-vidhya/undersampling-and-oversampling-an-old-and-a-new-approach-4f984a0e8392