In the first module, we describe how the classification performance of a machine learning model can be estimated using the receiver operating characteristic (ROC). It is explained how the ROC involves calculating model classification performance with multiple different decision thresholds, and how the ROC is a better measure of classification performance than simple classification accuracy or misclassification rate measures. Furthermore, the closely related concepts of an area under the curve (AUC) and the equivalent concordance index (C-index) values are discussed, which summarize the classifier model performance using ROC.

In this module, an interpretation of supervised machine learning methods simply as abstract mappings from a sample of data to a predictive hypothesis is presented. As an important special case that covers a surprisingly large portion learning algorithms, we consider methods that select an optimal hypothesis based on a given measure of how well hypotheses fit to a sample of data. The measure can be just a straightforward measure of prediction performance of a hypothesis on the sample, such as classification accuracy or regression error. However, it can also be something more complicated and seemingly more distant from the learning objective, such as a function measuring the distance of Voronoi partitions from the sample points as is the case with nearest neighbor methods we consider as example methods. Furthermore, resampling and cross-validation based model selection method considered in the third module are also examples of this framework. The law of large numbers concept is revisited and the so-called bounded differences conditions under which it holds for arbitrary performance measures on a sample of data are considered.

In this module, a case study on pain assessment from biosignal data is considered in which cross-validation based model performance estimation is conducted with non-independent data sample points. The independence assumption of data samples is violated when data set consists from repeated measurements from the same subject source. Because of these independence violations, the standard leave-one-out cross-validation can not be used, since it leads to biased performance estimation. Instead, with the repeated measurement data a leave-subject-out cross-validation method is utilized, which answers the statistical question on how well the model estimates the experienced pain of new patients not seen in the model training phase.