{"id":15891,"date":"2023-01-06T22:19:05","date_gmt":"2023-01-07T03:19:05","guid":{"rendered":"https:\/\/panamahitek.com\/probando-el-clasificador-k-nearest-neighbors-con-el-dataset-mnist\/"},"modified":"2023-10-17T18:20:51","modified_gmt":"2023-10-17T23:20:51","slug":"testing-knn-algorithm-in-machine-learning","status":"publish","type":"post","link":"https:\/\/panamahitek.com\/en\/testing-knn-algorithm-in-machine-learning\/","title":{"rendered":"Testing KNN algorithm in Machine Learning"},"content":{"rendered":"

One of the most popular classification algorithms is the KNN algorithm in Machine Learning. The K-Nearest Neighbors<\/strong> (KNN) classifier is a simple and effective method for image and general data classification. In this article, we will explore the use of the KNN classifier on the well-known MNIST dataset of handwritten digits.<\/p>\n

We will use the scikit-learn library in Python to train and evaluate the classifier, and we will conduct a hyperparameter search to find the best combination of parameters for the KNN model<\/a>. By the end of this article, you will understand how to use the KNN classifier for image classification and how to optimize its performance through hyperparameter tuning.<\/p>\n

Features of the K-Nearest Neighbor classifier<\/strong><\/h5>\n

The K-Nearest Neighbors<\/strong> algorithm is a non-parametric classification method that assigns a sample a class based on the majority of the classes of its nearest neighbors in the feature space. In other words, a sample is classified according to the most common class of the K<\/strong> samples closest to it in the feature space. KNN is commonly used in classification and regression problems.<\/p>\n

The value of k<\/strong> is determined by the user and can affect the complexity and accuracy of the model. The parameter “k”<\/strong> is the number of nearest neighbors that are used to make a prediction. For example, if k=3<\/strong>, then the algorithm will consider the 3 nearest neighbors to a sample and make a prediction based on their classes. The value of k<\/strong> can be chosen by the user and can affect the complexity and accuracy of the model.<\/p>\n

A smaller value of k<\/strong> can lead to a model with lower bias but higher variance, while a larger value of k<\/strong> can lead to a model with higher bias but lower variance. In general, it is a good idea to test several values of k<\/strong> to see which gives the best performance on training and test data. It is also common to use odd values of k<\/strong> to avoid ties in class prediction.<\/p>\n

One of the benefits of the K-Nearest Neighbors algorithm is that it has a low training time, as it does not require a complex model to fit the training data. Instead, it simply stores the training data and makes predictions based on the K nearest neighbors to a sample when asked to classify new data. This means that the training time is essentially constant, regardless of the size of the training dataset.<\/p>\n

Despite its simplicity, KNN can still achieve high accuracy in many tasks, especially when the number of features is low and the classes are well separated. It is also a robust algorithm that is not sensitive to the scale of features or the presence of noise in the data. However, it can be sensitive to the value of K, and its performance may be compromised when the number of features is high or when the classes are not well separated.<\/p>\n

How does the KNN algorithm work?<\/strong><\/h5>\n

To illustrate how KNN works, I’m going to present some images with a hypothetical data classification scenario. Let’s assume we have the following dataset on some plane:<\/p>\n

<\/p>\n

Each “x” represents a data point in the spatial plane. Now let’s assume that I, as a human, am asked to classify the data into groups. I think we can all agree that there are two main data groups, roughly separated by the horizontal plane’s midpoint.<\/p>\n

\"K-Nearest
Class A and Class B<\/figcaption><\/figure>\n

As we can see in the image, we have separated the data into two groups. In Machine Learning, when we deal with datasets, we don’t talk about groups or sets but classes. And each class can represent any object in the real world. In my case, I’ll consider that objects of “class A” are squares and objects of “class B” are circles.<\/p>\n

\"K-Nearest<\/p>\n

The moment I, a human, decided to classify the dataset’s objects into two classes and assigned a class to each sample, I constructed a training dataset. With this dataset, we can train a Machine Learning classifier to classify new data we haven’t seen yet.<\/p>\n

\"K-Nearest
New sample to classify<\/figcaption><\/figure>\n

For example, let’s assume that after training our classification algorithm, we’re asked to classify a new data point:<\/p>\n

We, as humans, can identify that this new data point is a square. That is, it’s surrounded by other squares. To classify it as a circle, it should be further to the right of the plane, where the other circles are. It’s not that hard, right?<\/p>\n

But how do we make a computer program classify this data correctly? How do we make the computer “think” like us humans? Well, the truth is, we won’t make the computer think. But we can teach it to discriminate between squares and circles using a Machine Learning algorithm.<\/p>\n

Each algorithm available today solves this problem differently. In KNN’s case, we’ll identify the “k” nearest neighbors to the data point we want to classify. This way, the computer can determine the data point’s proximity to the two available classes and decide whether it belongs to one or the other.<\/p>\n

\"K-Nearest
Identification of the “k” nearest neighbors. In this example, k=3, so 3 neighbors are identified<\/figcaption><\/figure>\n

As we can see in the image, the 3 closest “neighbors” to the data point we want to classify are squares. Thus, it’s very likely that the “x” in the image represents a square. Note that I used the word “likely” because the Machine Learning paradigm is based more on probabilities than certainties.<\/p>\n

Since this method is based on checking a data point’s “neighbors,” it’s logical that the KNN algorithm is named as such. K-Nearest Neighbors means k-closest neighbors. I think KNN is the computational representation of the saying “Tell me who you’re with, and I’ll tell you who you are.”<\/p>\n

What happens when we need to classify data that’s located between the two classes? Let’s look at the following image:
\n\"K-Nearest<\/p>\n

For the data point shown in the image, it’s not easy to determine if it belongs to one class or the other. Even I, as a human, am not sure if it’s a square or a circle. For the computer, it will check the “k” nearest neighbors, calculating the distance between the neighbors’ centroids and the new data point to be classified. If we work with k=3<\/strong>, we’ll have something like this:<\/p>\n

\"K-Nearest<\/p>\n

Clearly, the data point is a square, as two out of its three closest neighbors are squares. If there’s a tie, where two or more classes have the same number of occurrences among the K nearest neighbors, then the sample can be assigned to the class with the lowest index value.<\/p>\n

This classification method is known as majority voting, as the sample’s class is determined based on the majority class among its neighbors. It’s a simple and effective method for classification and is easy to understand and implement.<\/p>\n

Broadly speaking, this is how the KNN algorithm works. There are other important hyperparameters to consider with this algorithm, besides the number of neighbors considered in classification:<\/p>\n