1
$\begingroup$

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or negative (bearish). While the model is accurate when plugging in my own test data (F1 Score ~86%), the classification itself is not accurate. Tweets that are undoubtedly positive/bullish, and not classified as so. Perhaps this is because the language in the investment world is different than a movie review - which uses universally recognized positive/negative words and/or sentences.

The same is true when I take my tweet dataset and use Vader SentimentIntensityAnalyser to parse pos/neg tweets into separate folders.

So my question is... since the language that is used for telling whether a stock is bullish/bearish is uniquely different from that of an Amazon review, or movie review, would it be optimal for me to manually classify my dataset into positive (bullish) and negative (bearish) datasets?

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't understand, what do you mean when you say that the model is accurate on the test data but the classification itself is not accurate? Isn't that the same thing? Unless you mean that it's not accurate enough, that is you find too many errors? $\endgroup$
    – Erwan
    Commented Jun 8, 2019 at 21:36
  • $\begingroup$ Assuming that the classification of the data in the dataset is correct, the model will correctly predict whether positive or negative. However, this classification itself is incorrec. Example: "Apple's earnings were released today, I'm really feeling bearish now..." "this stock is going to tank... did you see the earnings?" The model will be "correct" in labeling these both positive (based on the trained dataset), however they're actually negative (bearish). $\endgroup$
    – BorangeOrange1337
    Commented Jun 8, 2019 at 22:06

1 Answer 1

Reset to default
1
$\begingroup$

There can be two distinct reasons to use instances annotated with the gold-standard class, i.e. the true answer for the target application:

  • In order to perform proper evaluation your test set must contain the gold-standard labels. The principle of evaluation is to measure by how much the predictions deviate from the truth, but without the truth the performance that you obtain on the test set is meaningless for the task that you are doing.
  • In order to train a supervised or semi-supervised model, the training set must contain the gold-standard labels. Semi-supervised methods offer some options to adapt a training set to a different task.

You can't rely on a model if you can't evaluate it at least on a small sample, so yes you probably need to manually annotate a subset of the data. It's only after that you can start thinking about how to improve performance.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.