I am self learning data analysis using (mostly) R. I'm currently on K-nearest neighbour topic, and I went through the process of normalizing- splitting the dataset to train/test and fitting the model and evaluating it, and it all went well. But, there is one fundamental thing I just don't understand: what is the point in classifying a known data? How do I predict the value of NEW data? The outcome of knn() is a factor with labels, but after using it for evaluating my model (namely, the value of K I have chosen), what should come next? For example, in regression model I can use predict() to predict new data, but I do I predict (as opposed to just classify) using knn? Thank!


when you "predict" something in KNN-classify problems, you are classifying new information.

yah, KNN can be used for regression, but let's ignore that for now.

The root of your question is why bother handling known data, and how can we predict new data. Let's do KNN in R1, with two training examples. The first one will be 0 and it will be class A, the next one will be 100 and it will be class B.

So, KNN is what's known as a lazy classifier. You actually aren't training, any hyperparameters, just loading the training data. I've loaded two points, and now I want to classify a new point. Let's consider the point 0. Since we already have the point 0 labelled as class A, we know it's going to be class A. What about the point 4? It's closer to 0 than to 100, so it's class A. Similarly, the point 95 is closer to 100, so it's class B.

KNN is really just doing those comparisons with more points and/or more dimensions.

EDIT: user asked the following: how would I predict if it is yes or no, based on the classification I have done before?

The most basic implementations of KNN check the distance from your new case to all of the points in the training data. If you want to visualize it, it's possible with one, two, and maybe 3 dimensional data.

In python, you can do as follow (from scikit-learn.org):

X1 = np.random.normal(0,1,100) #values centered around 0
X2 = np.random.normal(6,.1,100) #values centered around 6
X = np.concatenate([X1,X2])
X = np.array(X).reshape(-1,1)
Y = [0 for y in range(100)]+[1 for y in range(100)] #2 classes
Y = np.array(Y).reshape(-1,1)

clf = neighbors.KNeighborsClassifier()
clf.fit(X, Y)

now, you can predict by calling predict


KNN visualized in 2D:

KNN visualized

from: https://www.analyticsvidhya.com/blog/2014/10/introduction-k-neighbours-algorithm-clustering/

  • $\begingroup$ Yes, I totally understand that theoretically. But practically, my original data already HAS the target variable (yes/no), so I don't have to predict if it's Yes or No because I have that information. Now, if I had a new observation in my dataset, that did not have a value in the that target variable, how would I predict if it is yes or no, based on the classification I have done before? In decision tree- you can visualize the tree and see the nodes and where your new observation belongs, in regression you use predict() to feed values and predict outcome, but here?.. $\endgroup$
    – SAR
    Apr 3 '17 at 18:28

You can use KNN for regression in your case. You can either use fuctions for knn regression like knn.reg or you can implement your own solution. Suppose you have decided to use K=1 then find distances of test point from all known points and put the target variable as closest value. average of three values for k=3 and so on. You can use different parametrics for distance like euclidean, hamming and standarised the dataset before applying operations.

Example using knn.reg:



train_index <- sample(1:150,120)

train <- iris[train_index,2:4]

y_train <- iris[train_index,1]

test <- iris[-train_index,2:4]

y_predict <- knn.reg(train,test,y_train,k=3)

y_actual <- iris[-train_index,1]

rmse <- sqrt(sum((y_actual - y_predict$pred)^2)/length(y_actual))

  • $\begingroup$ Thank you very much! I'll test it and see how it works, but the code make sense to me. $\endgroup$
    – SAR
    Apr 4 '17 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.