0
$\begingroup$

I've used scikit-learn in Python to compare results of naive Bayes and SVM. I've found that naive Bayes is quicker than SVM. Could anyone shed some light on reasons for such finding?

$\endgroup$
  • 3
    $\begingroup$ You could give more relevant name to the question... like Models performance or smth... $\endgroup$ – IharS May 5 '15 at 8:52
  • 2
    $\begingroup$ Well, one of them has to be quicker. If it was the other way round, would you still be asking the question? Do you know what these algorithms do? $\endgroup$ – Spacedman May 5 '15 at 9:00
  • $\begingroup$ I know what the two algorithms do. I just want to know the reason behind Naive Bayes quicker performance. Thank you. $\endgroup$ – Vignesh Don May 5 '15 at 14:30
6
$\begingroup$

The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation.

Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates can be calculated independently and thus possibly very quickly.

In comparison, with SVM, every single record in the data base will require a computation of the distance function to determine the optimal decision boundary.

We can observe that for SVM, we have a complexity of $O(\text{#features} \times \text{#observations}^2)$ or more depending on the choice of SVM and kernel. On the other hand, for naive bayes we would have a complexity of $O(\text{#features} \times \text{#observations})$.

Hence in general, we can say that SVM will take longer than Naive Bayes.

| improve this answer | |
$\endgroup$
  • $\begingroup$ "In comparison, with SVM, every single record in the data base will require a computation of the distance function" - What do you mean with "distance function"? The kernel? $\endgroup$ – Martin Thoma Jan 26 '16 at 18:40
  • $\begingroup$ @moose yes, the distance function is basically the kernel applied on two instances, which usually is linear in number of features $\endgroup$ – rapaio Jan 27 '16 at 6:43
  • $\begingroup$ Additionally to the answer, we can mention the time for prediction which is given by the number of instances predicted x number of support vectors x kernel evaluation (which is usually the number of features). Prediction time is important because if for training one can employ some caching, for prediction the caching is useless $\endgroup$ – rapaio Jan 27 '16 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.