1
$\begingroup$

I am working on a multiclass classification problem with 3 (1, 2, 3) classes being perfectly distributed. (70 instances of each class resulting in (210, 8) dataframe). Now my data has all the 3 classes distributed in order i.e first 70 instances are class1, next 70 instances are class 2 and last 70 instances are class 3. I know that this kind of distribution will lead to good score on train set but poor score on test set as the test set has classes that the model has not seen. So I used stratify parameter in train_test_split. My code:-

# SPLITTING 
train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 
69, stratify = y)

cross_val_model = cross_val_score(pipe, train_x, train_y, cv = 5,
                              n_jobs = -1, scoring = 'f1_macro')
s_score = cross_val_model.mean()


def objective(trial):

    model__n_neighbors = trial.suggest_int('model__n_neighbors', 1, 20)
    model__metric = trial.suggest_categorical('model__metric', ['euclidean', 'manhattan', 
    'minkowski'])
    model__weights = trial.suggest_categorical('model__weights', ['uniform', 'distance'])

    params = {'model__n_neighbors' : model__n_neighbors, 
          'model__metric' : model__metric, 
          'model__weights' : model__weights}

    pipe.set_params(**params)

    return np.mean( cross_val_score(pipe, train_x, train_y, cv = 5, 
                                    n_jobs = -1, scoring = 'f1_macro'))

knn_study = optuna.create_study(direction = 'maximize')
knn_study.optimize(objective, n_trials = 10)

knn_study.best_params
optuna_gave_score = knn_study.best_value    

pipe.set_params(**knn_study.best_params)
pipe.fit(train_x, train_y)
pred = pipe.predict(test_x)
c_matrix = confusion_matrix(test_y, pred)
c_report = classification_report(test_y, pred)

Now the problem is that I am getting perfect scores on everything. The f1 macro score from performing cv is 0.898. Below are my confusion matrix and classification report:-

14  0   0 
0   14  0 
0   0   14

Classification Report:-

              precision    recall  f1-score   support

       1       1.00      1.00      1.00        14
       2       1.00      1.00      1.00        14
       3       1.00      1.00      1.00        14

accuracy                            1.00        42
macro avg       1.00      1.00      1.00        42
weighted avg    1.00      1.00      1.00        42

Am I overfitting or what?

$\endgroup$
10
  • 1
    $\begingroup$ Everything looks fine to me (just wondering what knn_study.best_params does, but I guess it is just printing the best parameters), overfitting is when your accuracy increases on train set, but decreases on test set. Here your accuracy is 1.0 on test set, so everything is perfect. It means that your classes are 'clustered' enough so the algorithm is able to give 42 correct predictions out of 42 test values. It is anyway not really possible to overfit on a KNN algorithm, it is an issue that mostly affects Neural Networks, which KNN is not. $\endgroup$
    – Ubikuity
    Aug 9 '21 at 18:46
  • $\begingroup$ @Ubikuity yes knn_study.best_params gives the best parameters. Are you sure that it is not overfitting? Because one rarely gets 90% score in real world models. Here I am getting 100% which makes me believe something is wrong! $\endgroup$
    – spectre
    Aug 9 '21 at 18:49
  • $\begingroup$ Also the fact that I have used stratify = y on an extremely balanced dataset. Maybe that might be the reason? I dunno what are your thoughts on that? $\endgroup$
    – spectre
    Aug 9 '21 at 18:50
  • $\begingroup$ i just checked the split function (from sklearn) and still cannot figure what stratify does, would you explain it to me ? $\endgroup$
    – Ubikuity
    Aug 9 '21 at 18:51
  • $\begingroup$ @Ubikuity This stratify parameter makes a split so that the proportion of values in the sample produced will be the same as the proportion of values provided to parameter stratify. For example, if variable y is a binary categorical variable with values 0 and 1 and there are 25% of zeros and 75% of ones, stratify=y will make sure that your random split has 25% of 0's and 75% of 1's. $\endgroup$
    – spectre
    Aug 9 '21 at 18:54
1
$\begingroup$

Sorry for the delay.

Having 100% accuracy means that the task is easy enough and the network has no trouble performing it.

Just a quick reminder about how K-NN algorithm works : During training, you put your data in it and it somehow remembers the data. Then when using it on new values (testing here), it just looks for the nearest neighbours and look at the classes of these ones. After finding that K Nearest Neighbours are from class X, it tells that the input was from class X.

KNN

Here for example, the algorithm finds out that most points (the 'most' depends on the K you choose) around your input are from class 2, so the input should belong to class 2 as well.

Now here is how i would try to analyse the data you send in the KNN :

import matplotlib.pyplot as plt
import numpy as np

data2 = np.random.randn(40, 2)  # Replace with your data
y = np.random.randint(0, 3, 40)  # Replace with your data
data2[:, 1] = data2[:, 1] + 4*y  # Replace with your data


for feature in range(data2.shape[1]):
    plt.figure()
    for sample in range(data2.shape[0]):
        plt.title('feature ' + str(feature))
        plt.scatter(data2[np.where(y == 0), feature].squeeze(), y[np.where(y == 0)], color='red')
        plt.scatter(data2[np.where(y == 1), feature].squeeze(), y[np.where(y == 1)], color='blue')
        plt.scatter(data2[np.where(y == 2), feature].squeeze(), y[np.where(y == 2)], color='green')
    plt.show()

In my example, I have 40 samples with 2 features each and 3 classes and here are the 2 different plots I get :

Plot1 Plot2

On these plots, you get each class values on a different line. So you can see if your classes are clustered or not. A feature which output something similar to feature 0 is nearly useless for the KNN classification as classes seem to have random values and are not forming clusters. A feature like feature 1 is a feature that provide much information to the classifier as we can clearly see the difference between classes.

Hope this helps, if you have any questions, feel free to ask

$\endgroup$
10
  • $\begingroup$ Sorry for the delay. Was facing some health issues. Regarding your answer, should I perform this after fitting my final model? Also what about learning curves to detect over/under fitting? $\endgroup$
    – spectre
    Aug 13 '21 at 6:43
  • $\begingroup$ @spectre You can perform the analysis whenever you want, we usually do it before applying KNN algorithm because this type of data analysis is an indicator that shows if KNN will actually work well or not. About the overfitting and learning curves, I though it would be clearer now, but these concepts do not apply to KNN algorithm, curves and overfitting is an issue that neural networks have to cope with. KNN is a much simpler algorithm and it does not really face the overfitting problem. $\endgroup$
    – Ubikuity
    Aug 13 '21 at 7:59
  • 1
    $\begingroup$ I'll try to explain briefly why KNN isn't affected by overfitting with an example. Let's have a NN (Neural Network) that tries to approximate the line x = 2y in 2D, we feed x in the network and it returns y. your samples may not be perfect, for example they may be (1, 2.3), (5, 9.5), (3, 5.4), ... So when you start your training, during each epoch it will go over your set of sample, predict and change the parameters of the network to give more accurate prediction. $\endgroup$
    – Ubikuity
    Aug 13 '21 at 8:05
  • $\begingroup$ The beginning of the training is when the NN learns the 'general relations', so it does the big part of the job, but not the perfect fit. So here it may find something like x = 1.9y. Overfitting is when after going over and over your set of samples multiples times, your network is super accurate on your training set (because he found some super complicated polynomial relation that perfetly describes the training set), but it is not able anymore to generalize the relation to other cases. So that is why we avoid this by checking the learning curves over the epochs. $\endgroup$
    – Ubikuity
    Aug 13 '21 at 8:07
  • 1
    $\begingroup$ Overall, I really feel like you underestimate the ability of your network to give the correct prediction, your task may not be so hard and since your dataset is not very massive, it is not such a big deal that your network gets 42 correct predictions in a row. A quick but not rigorous rule is to add 1 correct and 1 false prediction to your accuracy if your dataset is too small. So here it would turn your accuracy to 43/44 = 98% accuracy. $\endgroup$
    – Ubikuity
    Aug 13 '21 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.