LOOCV (Leave One Out Cross-Validation) in R Programming

LOOCV (Leave-One-Out Cross-Validation) is a model evaluation technique used to assess the performance of a machine learning model on small datasets. In LOOCV, one observation is used as the test set while the rest form the training set. This process is repeated for each data point in the dataset, resulting in n training-testing cycles, where n is the number of observations. The overall accuracy is averaged across all iterations.

Mathematical Expression 

In Leave-One-Out Cross-Validation (LOOCV), each individual observation serves once as the validation set, while the remaining n-1 observations are used for training. Instead of refitting the model n times, LOOCV for linear models can be computed efficiently using the following formula:

LOOCV Error

Where:

• = Actual value of the observation
• = Predicted value of the observation using the full model
• = Leverage of the observation (diagonal element of the hat matrix

Implementation in R

We are going to perform a Leave-One-Out Cross Validation (LOOCV) on the Hedonic dataset to evaluate the performance of linear regression models with increasing polynomial degrees.

1. Importing the Dataset

We are loading the Ecdat package, which contains the Hedonic dataset with information on housing prices, and the boot package, which provides tools for resampling methods, including Leave-One-Out Cross Validation (LOOCV). We will then check the structure of the Hedonic dataset.

• install.packages(): Installs the Ecdat and boot packages.
• library(): Loads the Ecdat and boot packages.
• str(): Displays the structure of the Hedonic dataset.

Output:

2. Fitting a Linear Model and Performing LOOCV

We are fitting a linear regression model to predict age and performing LOOCV to evaluate its performance.

• glm(...): Fits a generalized linear model. In this case, it defaults to a linear regression because no family argument is specified.

Output:

3. Performing LOOCV and Extracting MSE

We are extracting the Mean Squared Error (MSE) from the LOOCV to evaluate model performance.

• cv.glm(Hedonic, age.glm): Performs LOOCV on the model.
• cv.mse$delta: Extracts the MSE value.

Output:

Mean Squared error for the model is: 250.2985 250.2856

4. Fitting Polynomial Models for Increasing Degrees and Performing LOOCV

We are fitting polynomial models with increasing degrees and performing LOOCV to evaluate their performance.

• rep(0, 5): Initializes a vector to store LOOCV errors for 5 models.
• poly(crim, i): Creates polynomial terms for the crim variable.
• glm(age ~ ...): Fits the polynomial regression model.
• cv.glm(Hedonic, age.loocv): Performs LOOCV for each polynomial model.

Output:

Mean Squared error for the model is: 250.2985 252.4706 254.7776 299.5546 455.6091

Advantages of LOOCV

• It has no randomness of using some observations for training vs. validation set. In the validation-set method, each observation is considered for both training and validation so it has less variability due to no randomness no matter how many times it runs.
• It has less bias than validation-set method as training-set is of n-1 size. On the entire data set. As a result, there is a reduced over-estimation of test-error as much compared to the validation-set method.

Disadvantage of LOOCV : Training the model N times leads to expensive computation time if the dataset is large.