5
$\begingroup$

I implemented different machine learning algorithms on a matrix with binary data to predict a univariate target with two classes.

  • random forest (accuracy = 62.01)
  • Neural Network(acc= 58.9)
  • svm-radial kernel (accuracy = 58.02)
  • linear discriminant analysis(accuracy = 57.9)
  • logistic regression(accuracy = 57.6).

My baseline accuracy is 52.55. But in case of Naive Bayes in same setting gives only 48.5 accuracy that identifies only one class in y. predict.

Is it possible for a machine learning model to behave worse than a random classification?

$\endgroup$
5
$\begingroup$

Yes, it's possible.

Just means the model is adjusting to noise, so it's valuing the "wrong" features.

As an analogy, if you were randomly guessing basketball game outcomes, you'd probably perform better than someone who thinks less points is better and is guessing based on previous games' scores

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Can I report NaiveBayes model result along with other models result? I have to report in my Masters thesis? $\endgroup$ – KHAN irfan Aug 11 '17 at 22:18
  • 3
    $\begingroup$ Well, you definitely CAN. This only means that a Naive Bayes model wasn't suited for your problem. Don't know if you should, though. That context is all you! $\endgroup$ – AMC Aug 11 '17 at 23:05
  • $\begingroup$ Naive Bayes is predicting y.pred as one class. This is something that indicates that the model is biased towards one class. I fail to understand this behavior. Although my classes are balanced. $\endgroup$ – KHAN irfan Aug 11 '17 at 23:07
  • $\begingroup$ Did you change any parameters for the Naive Bayes model? In what language did you write the script? $\endgroup$ – AMC Aug 11 '17 at 23:31
  • 1
    $\begingroup$ If you used the same data and there are no parameters, to me it just means that Naive Bayes was not suited for your data, not that there was anything wrong with your modeling technique or data. Especially considering you had 'normal' results with other models $\endgroup$ – AMC Aug 11 '17 at 23:53
2
$\begingroup$

Your model can be worse than random, for example, if some fundamental assumptions are violated, in an imbalanced setting when your using accuracy as your baseline or you have noisy data etc.

However, in a binary setting, if your classes are perfectly balanced and if your classifier is consistently making false predictions (not due to randomness), you can always adjust the model to be better than random by predicting the exact opposite of your model.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ My classes are balanced, I have not removed collinearity in features in every model, so I have kept the basic framework same for all models but for Naive Bayes I get worst results. Here is another reference stackoverflow.com/questions/32211157/… $\endgroup$ – KHAN irfan Aug 11 '17 at 22:24
2
$\begingroup$

Well, absolutely. I had the same issue recently (and ended up here). So I just simulated a dataset and an independant label (full article here)

Basically, when there is nothing to learn, your model will produce predictions uncorrelated to the label. In turn, these predictions (when repeated many times) show an interesting distribution of the error rate (evaluated on a validation set or using cross validation).

However, in your case, I am quite surprised that one specific model does not work correctly, when the others seem to show relevant predictions.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.