1
$\begingroup$

I am solving a questions binary classification problem and the training size for this is huge(291 billion). The data has bloated because of using tfidfvectorizerfor the questions column. Here, in the problem, I have to classify the questions.

I have used Logistic regression and have also kept MultinomialNB, Randomforest and svm for training. However, instead of doing such trial and hit method, Is there a logical explanation why one classification algorithm must perform better than others in this context.

Previously, I have tried randomforest and logistic regression for spam filtering and observed that the training error was less for logistic regression as compared to randomforest. I understand that it could be an overfitting solution. But Is there a way I can say for certain that 'this' is the classification algorithm you must use.

Note: I'm am yet to remove stopwords and do some dimensionality reduction.

$\endgroup$
1
$\begingroup$

I don't think there is a clear-cut criterion to decide on what method to use. From what I see, you have a good amount of data and the problem is "complex" (language). This are reasons to go for "deep" learning such as neural nets or boosting. The reason is that both can handle "non-linearity" very well.

Another thing that comes to my mind is that when you use for instance logit, you man have the problem that you have many features (words) and that only a certain share of the features might be relevant. So in this case you would try to get rid of features which are not helpful in making good predictions. With logit, a lasso approach (l1 penalty) would be a natural thing to do, to "shrink" features.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.