New answers tagged

0

The baseline is not the word list by itself, the idea is to implement a simple classifier which works as follows: It receives as input a list of standard positive words P (e.g. "good", "great", "nice", ...) and a list of standard negative words N (e.g. "bad", "depressing", "annoying",...). For ...


0

Features of your naive bayes classifier is most likely a bag of words where each feature represents a single word. The naive bayes classifier also assumes that each feature (or word) contributes independently to the final prediction. So if your sentence is composed of a single word - you get an idea of how the model reacts to that word.


0

You need to do pip install sentencepiece for it to work. By the way, you can also give the tweets as a list to the tokenizer. You don't need to tokenize them one by one. tokenizer(tweets, max_length=max_len, padding='max_length', add_special_tokens=True)


0

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...


Top 50 recent answers are included