3
$\begingroup$

I'm doing a beginning attempt at tweet sentiment analysis (positive, neutral, negative). So far I have cleaned the data and used a BoW to get some feeling of the data (>2.5k tweets). I also made bigrams to try to get clearer sentiment insight.

The data is severely skewed so I tried both upsampling and downsampling to view the difference.

I finally passed it all through a Random Forest Classifier and I get an accuracy of 0.7 for the upsampled data and 0.3 for the downsampled one.

I visualized this in a confusion matrix and I can see that the model sucks at actually labeling correctly. I retrieved Precision, Recall, and F1. I can see I have problems with the positive and negative sentiments above all (values are <0.45)

I have tried going back to cleaning the data but at this point, I can't think of anything else to do to it (I've run stemming, lemma, tokenize, stopwords and added stopwords that were left there, removed special characters (@, #, etc), and hyperlinks.

I also gave my countvectorizer a range of ngrams of 1,1; 2,2; 3,3; but no big change is detected.

This is my first time doing this, can anybody point me in the right direction here please?

$\endgroup$

1 Answer 1

1
$\begingroup$

Apparently what you've done so far is to apply various standard techniques by trial and error. I'd suggest you could investigate what happens with your models and your data by doing a manual error analysis. Take a random sample of instances which are incorrectly predicted and try to understand why this happens. Errors can can happen for various reasons:

  • The gold label is wrong or questionable, sentiment can be subjective. A tweet considered neutral by A could be negative for B. If the same kind of tweet has different labels, the model cannot find good patterns to predict the label. If this kind of problem is common then the gold data is low quality and there's not much to do with it.
  • The model doesn't catch some clues that are obvious to a human. Identify what are these clues and why the model fails to catch them. This way you can design features in a more informed way in order to improve performance (feature engineering).

But first you should probably avoid resampling, and check for overfitting: the performance on the test set should not be much lower than on the training set. When you use complex features like all the n-grams, it's likely that you cause the model to overfit.

$\endgroup$
2
  • $\begingroup$ I will check for over fitting first then. I haven't done that to be honest. I fo want to ask, what if I figure out the labels are questionable? The training data is super skewed and has tons of random characters which I believe to be on purpose $\endgroup$ Commented Nov 13, 2021 at 15:06
  • $\begingroup$ @noideawhatimdoing the imbalance is not a problem by itself as long as it accurately represents the "true population" (the target data). the first reason to check the quality of the data/labels is to find an explanation: if the problem is mostly due to the data, there's no point trying to improve the method. After that you could see whether the data can be improved (re-label it or remove bad instances), or completely switch to another dataset if that makes sense. $\endgroup$
    – Erwan
    Commented Nov 13, 2021 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.