1
$\begingroup$

I am trying to create a sentiment analysis model and I have a question.

After I preprocessed my tweets and created my vocabulary I've noticed that I have words that appear less than 5 times in my dataset (Also there are many of them that appear 1 time). Many of them are real words and not gibberish. My thinking is that if I keep those words then they will get wrong "sentimental" weights and gonna make my model worse. Is my thinking right or am I missing something?

My vocab size is around 40000 words and those that are "rare" are around 10k.Should I "sacrifice" them?

Improve this question
$\endgroup$
1
  • $\begingroup$ Often words with few counts are removed. Simply because they do not generalise. Just give it a try with removing words with low count (more or less agressively). Check back with your test error. $\endgroup$
    – Peter
    Feb 1, 2021 at 21:17

1 Answer 1

Reset to default
1
$\begingroup$

Instead of dropping rare words or incorporating them risking their scarcity in the training data leads to poor predictions, you can opt for a third alternative: using a subword vocabulary.

You can use approaches like byte-pair encoding (BPE) to extract a subword vocabulary, that removes the out-of-vocabulary word problem and reduces data sparsity in general. There is the canonical python implementation as well as the popular implementation by Google called sentencepiece.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.