Cross Validation in Machine Learning

Cross-validation is a technique used to check how well a machine learning model performs on unseen data while preventing overfitting. It works by:

• Splitting the dataset into several parts.
• Training the model on some parts and testing it on the remaining part.
• Repeating this resampling process multiple times by choosing different parts of the dataset.
• Averaging the results from each validation step to get the final performance.

Validation Techniques

1. Holdout Validation

In Holdout Validation the dataset is split into training and testing sets. Common splits include 70–30, 80–20 or 75–25 depending on the dataset size and problem. Making it simple and quick to apply.

2. LOOCV (Leave One Out Cross Validation)

In this method the model is trained on the entire dataset except for one data point which is used for testing. This process is repeated for each data point in the dataset.

• All data points are used for training, resulting in low bias.
• Testing on a single data point can cause high variance, especially if the point is an outlier.
• It can be very time-consuming for large datasets as it requires one iteration per data point.

3. Stratified Cross-Validation

It is a technique that ensures each fold of the cross-validation process has the same class distribution as the full dataset. This is useful for imbalanced datasets where some classes are underrepresented.

• The dataset is divided into k folds, keeping class proportions consistent in each fold.
• In each iteration, one fold is used for testing and the remaining folds for training.
• This process is repeated k times so that each fold is used once as the test set.
• It helps classification models generalize better by maintaining balanced class representation.

4. K-Fold Cross Validation

K-Fold Cross Validation splits the dataset into k equal-sized folds. The model is trained on k-1 folds and tested on the remaining fold. This process is repeated k times each time using a different fold for testing.

Note: A commonly used value of k is 10, but the choice depends on the dataset size and problem requirements.

5. Repeated K-Fold Cross Validation

This method repeats the K-Fold cross-validation process multiple times with different random splits. It helps reduce the effect of randomness in data splitting and provides a more robust performance estimate.

• The dataset is split into k folds multiple times.
• Each repetition uses a different random shuffle.
• Final performance is averaged across all repetitions.
• It is useful for improving reliability in model evaluation.

Example of K Fold Cross Validation

The diagram below shows an example of the training subsets and evaluation subsets generated in k-fold cross-validation. Here we have total 25 instances.

• Here we will take k as 5.
• 1st iteration: The first 20% of data [1–5] is used for testing and the remaining 80% [6–25] is used for training.
• 2nd iteration: The second 20% [6–10] is used for testing and the remaining data [1–5] and [11–25] is used for training.
• This process continues until each fold has been used once as the test set.

Each iteration uses different subsets for testing and training, ensuring that all data points are used for both training and testing.

Comparison between K-Fold Cross-Validation and Hold Out Method

K-Fold Cross-Validation and Hold Out Method are used technique and sometimes they are confusing so here is the quick comparison between them:

Python implementation for k fold cross-validation

Step 1: Importing necessary libraries

We will import essential modules from scikit-learn.

• cross_val_score helps evaluate model performance using cross-validation.
• KFold splits the data into defined folds.
• SVC is used for Support Vector Classification.
• load_iris loads the sample dataset.

Step 2: Loading the dataset

We will use the Iris dataset a built-in, multi-class dataset with 150 samples and 3 flower species (Setosa, Versicolor and Virginica).

Step 3: Creating SVM classifier

SVC() from scikit-learn is used to build the Support Vector Machine model. Here, we are using a linear kernel, suitable for linearly separable data.

Step 4: Defining the number of folds for cross-validation

We define 5 folds, meaning the dataset will be split into 5 parts. The model will train on 4 parts and test on 1, repeating this process 5 times for balanced evaluation.

Step 5: Performing k-fold cross-validation

We use cross_val_score() to automatically split data, train and evaluate the model across all folds. It returns the accuracy for each fold

Step 6: Evaluation metrics

We print individual fold accuracies and the mean accuracy across all folds to understand the model’s stability and generalization.

Output:

The output shows the accuracy scores from each of the 5 folds in the K-fold cross-validation process. The mean accuracy is the average of these individual scores which is approximately 97.33% indicating the model's overall performance across all the folds.

Advantages

1. Better performance estimate: Provides a more reliable evaluation than a single train-test split.
2. Reduces overfitting: Helps ensure the model generalizes well to unseen data.
3. Efficient use of data: All data points are used for both training and testing at different iterations.
4. Flexible: Works with different types of datasets and models.

Disadvantages

1. Computationally Expensive: It can be computationally expensive especially when the number of folds is large.
2. Time-consuming: Methods like LOOCV can take a long time for datasets with many data instances.
3. Bias-Variance Tradeoff: Few folds may result in high bias while too many folds may result in high variance.