I am performing phishing URL classification, and I am comparing several ML classifiers on a balanced 2-class data-set (legitimate URL, phishy URL).

The ensemble and boosting classifiers such as Random Forest, Ada Boost, Extra Trees etc AND K-NN achieve an accuracy about 90%, and a False Positive Rate about 11-12%. (fig.)

On the other hand, classifiers such as SVM, Logistic Regression, Multinomial NB and Bernoulli NB seem to perform poorly with accuracies fluctuating between 70% - 80% and higher false positives.

Here is the thing. I also tried Gaussian NB and although it yields by far the worst accuracy 58.84% it has an incredibly low False Positive Rate 2.14% (and thus a high FNR)

  1. I have no idea why this is happening. Any ideas?
  2. Why some classifiers perform so poorly and others not?

I parametrized them all with Grid Search, they are used on the same dataset (about 30k records of each class) and I perform a 3-fold cross validation. It doesn't make any sense to me, especially for SVM. At last I use about 20 features.

P.S: I use python's sk-learn library



1 Answer 1

  1. I find the easiest way for people to understand this is to think of the confusion matrix. Accuracy score is just one measure of a confusion matrix, namely all the correct classifications over all the prediction data at large: $$\frac{True Positives + True Negatives}{True Positives + True Negatives + False Positives + False Negatives}$$ Your False Negative Rate is calculated by: $$\frac{False Negatives}{False Negatives + True Positives}$$ One model may turn out to have a worse accuracy, but a better False Negative Rate. For example, your model with worse accuracy may in fact have many False Positives but few False Negatives, leading to a lower False Negative Rate. You need to choose the model which produces the most value for your specific use case.

  2. Why do some classifier perform poorly? While an experience practitioner might surmise what could be a good modeling approach for a dataset, the truth is that for all datasets, there is no free lunch... also known as "The Lack of A Priori Distinctions Between Learning Algorithms" You don't know ahead of time if the best approach will be deep learning, gradient boosting, linear approaches, or any other number of models you could build.

  • $\begingroup$ Thank you for your answer, although I knew these. My question I guess is: Isn't the 2% FPR of the gaussian naive bayes a little bit too good to be true? No other algorithm goes below 10%! $\endgroup$
    – HelloWorld
    Commented May 12, 2020 at 23:18
  • $\begingroup$ "Too good to be true" is the wrong term. I could have a 0% FPR if I always predict the negative class, but you likely also care about making more correct predictions. The following confusion matrix has an accuracy of 58% and a FPR of 2%: TN: 98 FP: 2 FN: 82 TP: 18 $\endgroup$ Commented May 12, 2020 at 23:26
  • $\begingroup$ For a system where your goal is to minimize the FPR, it is a great result. I am bothered though about its validity. In confusion matrix terms, yes it makes sense... But what doesn't make sense to me is how Gaussian NB classifier can yield so much different results, from other classifiers (even from other NB classifiers!). For example Gaussian NB gave a FPR 2% and Multinomial NB FPR 27%. That's a HUGE difference, isnt it? $\endgroup$
    – HelloWorld
    Commented May 13, 2020 at 11:24
  • $\begingroup$ While the confusion matrix might be very different, the predictions might be less different. If your threshold is .5, then changing many predictions from .49 to .51 would greatly change the confusion matrix. $\endgroup$ Commented May 13, 2020 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.