Confusion Matrix

A confusion matrix, also known as an error matrix, is a table layout of measurement results that allows visualization of the performance of an algorithm under various conditions.

A common confusion matrix used for machine learning measures actual known classes against predicted classes:

It is often useful to visualize the confusion matrix as illustrated below:

Derived Values

A confusion matrix can be used to derive a number of measures. Common measures used for Machine Learning include:

$\\ACC = \frac{TP+TN}{TP+TN+FP+FN} \\\\\\TPR = \frac{TP}{TP+FN} \\\\\\TNR = \frac{TN}{TN+FP} \\\\\\FPR = \frac{FP}{TN+FP} \\\\\\PPV = \frac{TP}{TP+FP} \\\\\\NPV = \frac{TN}{TN+FN} \\\\\\\mathrm{\textup{where:}} \\\\TP\,\,\,\,\,\,\,\,\,\mathrm{\textup{True Positive}} \\TN\,\,\,\,\,\,\,\,\mathrm{\textup{True Negative}} \\FP\,\,\,\,\,\,\,\,\,\mathrm{\textup{False Positive (aka: type I error)}} \\FN\,\,\,\,\,\,\,\,\mathrm{\textup{False Negative (aka: type II error)}} \\ACC\,\,\,\,\mathrm{\textup{Accuracy}} \\TPR\,\,\,\,\,\mathrm{\textup{True Positive Rate (aka: sensitivity, recall, hit rate)}} \\FPR\,\,\,\,\,\mathrm{\textup{False Positive Rate (aka: false alarm rate)}} \\PPV\,\,\,\,\mathrm{\textup{Positive Predictive Value (aka: precision)}} \\NPV\,\,\,\mathrm{\textup{Negative Predictive Value}}$

Sensitivity/Recall

Sensitivity (aka: True Positive Rate, Recall, Hit Rate) is a measure of how many of the elements that should have been predicted as True actually were predicted as True.

Using a photograph image recognition example, if a model is trained to predict when a photo contains an automobile, sensitivity is what percent of all photos that do contain an automobile were predicted to contain an automobile.

Precision

Precision (aka: Positive Prediction Value) is a measure of how many of the elements that were predicted as True actually were True.

Using a photograph image recognition example, if a model is trained to predict when a photo contains an automobile, precision is what percent of photos that were predicted as containing an automobile actually did.

F1 Score

The F1 score is:

the harmonic mean of precision and recall
a weighted average of precision and recall
a measure of a test’s accuracy
a measure between 0 and 1
used in classification tests where True Negatives don’t matter much
calculated as:

$\\F1=2\cdot \frac{Precision\cdot Recall}{Precision+Recall}$

Receiver Operating Characteristic (ROC)

Receiver Operating Characteristic (ROC):

is a graphical plot
measures diagnostic ability
works for binary classifier models (classifiers with two classes)
shows the True Positive Rate (TPR) against the False Positive Rate (FPR) for various threshold settings
was originally developed to test military radar receivers, thus giving it its name

Area Under ROC Curve (AUC)

Area Under ROC Curve (AUC):

is measured as the integral value of the area under the ROC curve

gives the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one