How to predict the value in KNN?

Question

I am trying to build the KNN algorithm for IRIS dataset. First, I've computed the distance and stored it in 1d array. However, I am really struggling to build the prediction function. Therefore two questions arise:

1. What does this code do? classes = y[np.argsort(dist)][:k]
2. How should I change the last loop of the function given certain requirements?

def KNN(k, X, y, x):
    from scipy.spatial.distance import cdist
    """K nearest neighbors
    k: number of nearest neighbors
    X: training input locations
    y: training labels
    x: test input
    """
    N, D = X.shape
    num_classes = len(np.unique(y))
    dist = np.zeros(X.shape[0])   # <-- EDIT THIS to compute the pairwise distance matrix
    for i in range(len(dist)):
        dist[i] = np.linalg.norm(X[i]-x)
    print(dist)

    # Next we make the predictions
    ypred = np.zeros(num_classes)
    classes = y[np.argsort(dist)][:k] # find the labels of the k nearest neighbors

    for c in np.unique(classes):
        ypred[c] = y[c]  # <-- EDIT THIS to compute the correct prediction
        print(ypred)
    return np.argmax(ypred)
```

Answer 1

Here is one example for you to consider.

from sklearn.neighbors import KNeighborsClassifier
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"

# Assign colum names to the dataset
names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'Class']

# Read dataset to pandas dataframe
dataset = pd.read_csv(url, names=names)

dataset.head()

X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, 4].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

classifier = KNeighborsClassifier(n_neighbors=3)
classifier.fit(X, y)
testSet = [[5.1, 3.5, 1.4, .2]]
test = pd.DataFrame(testSet)
y_pred = classifier.predict(testSet)
print(y_pred)

The very last line of code will print this: ['Iris-setosa']

Let's look at the very first row in the test set: testSet = [[5.1, 3.5, 1.4, .2]]

Look at: dataset.head()

Do you see how y_pred is the class, given [5.1, 3.5, 1.4, .2]?

Hope that makes sense! Now, let's change the testSet just a bit, and check the outcome.

classifier = KNeighborsClassifier(n_neighbors=3)
classifier.fit(X, y)
testSet = [[5.1, 3.5, 4.9, .2]]
test = pd.DataFrame(testSet)
y_pred = classifier.predict(testSet)
print(y_pred)

Result: ['Iris-versicolor']

The variable ('petal-length') changed from 1.4 to 4.9, and now we have a different 'Class'!! Does it make sense? If you look at the raw data, you will see that the testSet doesn't even exist, but nevertheless, the model can make a prediction about what the Class should be!

K-Nearest Neighbors Algorithm in Python and Scikit-Learn

Answer 2

The code you've mentioned sorts an array in ascending order and returns arguments (the labels) for the first k. As you want to predict one class, you need to evaluate, how each of classes is close to the considered point. Distance of j class could be computed as a sum of 1/distance(x_i)**2 for all points with the label j, which are laying amongst the k nearest points. Then you could return argmax of ypred. (it's a weighted approach which takes into account distances between points, on the other hand, you could just count number of points, which fall into k nearest) You should write ypred[c] = np.sum(np.where(classes==c, 1, 0))

Answer 3

What does this code do? classes = y[np.argsort(dist)][:k]

This code selects the first k examples of y, where y has been sorted by the distances found in dist.

Example

Let us assume dist = [2,0.5,10,5]. In this example this distances show that the closest two elements are in positions 1 and 0.

If we look at the output of argsort we can see that, indeed, elements 1 and 0 are the closest ones.

dist = [2,0.5,10,5]
np.argsort(dist)
array([1, 0, 3, 2])

Then let us assume k=2 and y is as defined as followsL

y = np.array([0,2,1,0])
k = 2

Then y[np.argsort(dist)][0:k] simply will pick elements 1 and 0 from y

y[np.argsort(dist)][0:k]
array([2, 0])

How should I change the last loop of the function given certain requirements?

You should give us more details. What requirements?