Sentiment Analysis: Train Separate Models or Use One for All

Question

We have been tasked with building an in-house sentiment tool and we are going to use it on a multitude of data sources; survey responses, reviews, social listening etc..

This may be obvious to some but I would like to hear some thoughts. So my question is, would you train separate models for each data source since the documents from each source would be quite different? We would then apply the appropriate model depending on the data source.

Or is it best practice to train one model using a sample of all the data sources and apply that model to everything?

Answer 1

I would suggest that you train a single model. The features correlated with positive or negative sentiment in a review are likely the same as those in a survey response. That is, “good” means good in both, “bad” means bad in both.

In general you should combine models and datasets when you think that the features in the different situations have the same meanings/interpretations. If they have contradictory or different meanings/interpretations, then you should consider training separate models.

Answer 2

I think what @Kevin H has said is right.

There are couple of more thing which I would suggest you to spend more time on

1. Spend more time on understanding data, what does each word in each scenario means.
2. Generate a Stopword Dictionary to remove all the unnecessary words, Lemmatization of words.
3. Based on your Business Understanding make sure that all the important details are capture.
4. Using a Single Model helps in generalizing the results but before deciding that as the final model, make different models for different business problems and do ensemble of them and validate to know which model give better results(Ensemble or Single Model).

Main reason for suggesting Ensemble is Navie Bayes classifier works best with short sentence but for rest that is not the best model.

Regarding Sampling, I think to do that you need to understand data clearly as to extract sample you need to make sure that every dimension of the population is covered(Sample should exactly represent the population). So, you should be very careful when implementing Sampling techniques.

Choose the best model, you need to try all the methods apply them in all applicable conditions and based on your Business Application and results achieved, you need to decide but generally people implement Ensemble to achieve best results(in most of the cases).

If you need more on Ensemble or any other technique let me know, will help you!

Answer 3

since the documents from each source would be quite different

Well if this is true I would suggest to train different models.

Not that the opinions of the others are wrong, but if you have way different documents, for example tweets and newspapers articles, the accuracy of a model trained in both tweets and articles could not match the accuracy of models that are trained in only one kind of document.

Tweets have a lot of orthographic mistakes, lots of time the names are not capitalized, they also have emoji and slang.

On the other hand articles have most of the times a more "neutral" language regarding sentiments, they are way bigger in size than tweets and most of the times are well written.