Weighted K-NN

Weighted kNN is a modified version of k nearest neighbors. One of the many issues that affect the performance of the kNN algorithm is the choice of the hyperparameter k. If k is too small, the algorithm would be more sensitive to outliers. If k is too large, then the neighborhood may include too many points from other classes. Another issue is the approach to combining the class labels. The simplest method is to take the majority vote, but this can be a problem if the nearest neighbors vary widely in their distance and the closest neighbors more reliably indicate the class of the object. Intuition: Consider the following training set 👁 Image
The red labels indicate the class 0 points and the green labels indicate class 1 points. Consider the white point as the query point( the point whose class label has to be predicted) 👁 Image
If we give the above dataset to a kNN based classifier, then the classifier would declare the query point to belong to the class 0. But in the plot, it is clear that the point is more closer to the class 1 points compared to the class 0 points. To overcome this disadvantage, weighted kNN is used. In weighted kNN, the nearest k points are given a weight using a function called as the kernel function. The intuition behind weighted kNN, is to give more weight to the points which are nearby and less weight to the points which are farther away. Any function can be used as a kernel function for the weighted knn classifier whose value decreases as the distance increases. The simple function which is used is the inverse distance function. Algorithm:

• Let L = { ( x_i , y_i ) , i = 1, . . . ,n } be a training set of observations x_i with given class y_i and let x be a new observation(query point), whose class label y has to be predicted.
• Compute d(x_i, x) for i = 1, . . . ,n , the distance between the query point and every other point in the training set.
• Select D' ⊆ D, the set of k nearest training data points to the query points
• Predict the class of the query point, using distance-weighted voting. The v represents the class labels. Use the following formula
👁 Image

Implementation: Consider 0 as the label for class 0 and 1 as the label for class 1. Below is the implementation of weighted-kNN algorithm.