1
$\begingroup$

My dataset has about 100k entries, 6 features, and the label is simple binary classification (about 65% zeros, 35% ones).

When I train my dataset on different models: random forest, decision tree, extra trees, k-nearest neighbors, logistic regression, sgd, dense neural networks, etc, the evaluations differ GREATLY from model to model.

  • tree classifiers: about 80% for both accuracy and precision
  • k-nearest neighbors: 56% accuracy and 36% precision.
  • linear svm: 65% accuracy and 0 positives guessed
  • sgd : 63% accuracy and 2 true positives + 4 false positives

I don't understand the difference in such disparity. Can someone explain why that happens? Am I doing something wrong?

Also cannot find an answer to my question, so please link if someone asked it already

Would really appreciate the help!

Improve this question
$\endgroup$
2
  • $\begingroup$ Class inbalance is one thing but if one class is never chosen, there has to be a something wrong with the models. But it is impossible to say anything more without seeing the models. $\endgroup$
    – serali
    Commented Oct 7, 2021 at 20:01
  • $\begingroup$ @serali I am exaggerating a little bit, but there would only be about 10-20 positive cases chosen tops among the 35k possible. Could you say what could be wrong with the models? $\endgroup$
    – Egor
    Commented Oct 7, 2021 at 20:07

2 Answers 2

Reset to default
0
$\begingroup$

A few thoughts:

  • The first thing I would check is whether the other models overfit. You could check this by comparing the performance between the training set and the test set.
  • Also there's something a bit strange about k-NN always predicting the majority class. This would happen only if any instance is always closer to more majority instances than minority instances. In this case there's something wrong with either the features or the distance measure.
  • 100k instances looks like a large dataset but with only 6 features it's possible that the data contains many duplicates and/or near-duplicates which don't bring any information for the model. In general it's possible that the features are simply not good indicators, although in this case the decision tree models would fail as well.
  • The better performance of the tree models points to something discontinuous in the features (btw you didn't mention if they are numerical or categorical?). Decision trees and especially random forests can handle discontinuity but like logistic regression might have trouble with it.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Erwan, I highly appreciate your thoughtful answer!! Edited my question with the accurate data. My random forest doesn't seem to overfit. I ran a 10k fold validation, and all 10 accuracies, f1's, etc were consistently high. You are right about K-nn; among least performing models, this one was doing the best with a few thousand positives guessed. As for the dataset size, you might be right about the duplicates, but if they are there for both classes could they cancel out? Also, could you share your thoughts on max feature amount for 100k instances? All the features are numerical. $\endgroup$
    – Egor
    Commented Oct 8, 2021 at 3:59
  • $\begingroup$ @EgorIsakson in general it's best to have a large number of instances with a moderate number of features in order to avoid overfitting. With 100k instances and only 6 features you're really on the safe side, if you have other features available you could easily add them. However as usual this depends on the data, that's why I mentioned the point about possible duplicates: the data size might seem high but actually be equivalent to a small dataset if it's not "diverse enough". By themselves the duplicates are not a problem, except if the data is split randomly between training and test set: ... $\endgroup$
    – Erwan
    Commented Oct 8, 2021 at 12:31
  • $\begingroup$ ... This can cause a bias in the evaluation because the test set contains instances which have been seen during training (this would be a case of data leakage). $\endgroup$
    – Erwan
    Commented Oct 8, 2021 at 12:32
  • $\begingroup$ I spent last night testing exactly that, and that was the problem. I had duplicates in both sets. Ugh, I knew it was too good to be true haha. Thank you in any case; that was a good lesson for me $\endgroup$
    – Egor
    Commented Oct 8, 2021 at 14:14
0
$\begingroup$

One way to compare models is to look at the different decision boundaries the different models have learned. The different decision boundaries can impact the evaluation metrics.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.