Regularized K-Nearest Neighbors Algorithm

The K-Nearest Neighbors algorithm (KNN) is a simple yet powerful machine learning algorithm, used in both classification and regressions tasks. For classification, for instance, it makes estimations for new observations by identifying the majority class within the k training data points that are the closest to the new data. Despite its effectiveness and interpretability, KNN has a few weaknesses. One weakness in particular comes from the fact that the distances between points usually do not take into account the importance of the features for predicting the target variable.
This code aims to present a method to deal with that weakness, by weighting the features according to their importance for the task in hand, and comparing the results to a non-regularized version.