I am solving a questions binary classification problem and the training size for this is huge(291 billion). The data has bloated because of using tfidfvectorizer
for the questions column. Here, in the problem, I have to classify the questions.
I have used Logistic regression and have also kept MultinomialNB, Randomforest and svm for training. However, instead of doing such trial and hit method, Is there a logical explanation why one classification algorithm must perform better than others in this context.
Previously, I have tried randomforest and logistic regression for spam filtering and observed that the training error was less for logistic regression as compared to randomforest. I understand that it could be an overfitting solution. But Is there a way I can say for certain that 'this' is the classification algorithm you must use.
Note: I'm am yet to remove stopwords and do some dimensionality reduction.