0
$\begingroup$

My testing accuracy is way higher than my training accuracy. I have used feature selection and split the data into training, validation and test sets.

anova_filter = SelectKBest(f_classif, k=4)

rng = np.random.rand
X_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size = 0.40, shuffle = 
False, random_state = rng)
X_val, X_test, Y_val, Y_test = train_test_split(X_val, Y_val, test_size = 0.50, shuffle 
= False, random_state =rng)

#fitting the dataset
anova_svm.fit(X_train, Y_train)

#Predicting Values
Y_pred = anova_svm.predict(X_val)
X_train_pred = anova_svm.predict(X_train)

training_data_accuracy = accuracy_score(Y_train, X_train_pred)
testing_data_accuracy = accuracy_score(Y_val, Y_pred)

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Your testing dataset is strongly imbalanced. You have 82 samples in the positive class and only 3 classes in the negative class. By simply guessing "everything positive" your model would achieve 96.5% accuracy. This is a common problem in unbalanced datasets. I don't know what your data is exactly, so it is difficult to make a precise suggestion as to what you should change, but calculating the Balanced Accuracy which is the accuracy of the individual classes weighted equally instead of by their contribution, might be a good start. Evaluating your model's performance based on precision and recall might be a good option, too.
I might add however, that just 3 samples in the negative class is probably too little to make a good assumption about the performance of your model anyway.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi, thank you for your response. What should the percentage of positive to negative class for it to be a good performance? As shuffling the data also gives the ratio of positive to negative between 43-50%. Regarding the data: I have collected the features or data samples and the target values are obtained from fuzzy model. Is there any way I can balance my data or modify the test train split that the model takes equal values somehow to train and test. $\endgroup$
    – Akshita
    May 2 at 12:38
  • $\begingroup$ Generally speaking your test data should reflect real-world conditions as closely as possible. Say, you are trying to find rare birds on pictures, your occurence of "rare bird" vs "regular bird" would be much lower. Your model now has two learn that rare birds occur rarely, but they do occur. Making sure your model does not overfit to the more prominent class is a delicate task and as I said, evaluating recall vs precision in such a scenario might be better suited to evaluate the performance of the model. You can increase your test set by increasing its split size, oversampling or using smote $\endgroup$
    – the_strange
    May 2 at 14:40
  • $\begingroup$ Thank you. It worked $\endgroup$
    – Akshita
    May 3 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.