The baseline is not the word list by itself, the idea is to implement a simple classifier which works as follows:
It receives as input a list of standard positive words P (e.g. "good", "great", "nice", ...) and a list of standard negative words N (e.g. "bad", "depressing", "annoying",...).
Features of your naive bayes classifier is most likely a bag of words where each feature represents a single word.
The naive bayes classifier also assumes that each feature (or word) contributes independently to the final prediction.
So if your sentence is composed of a single word - you get an idea of how the model reacts to that word.
You need to do pip install sentencepiece for it to work.
By the way, you can also give the tweets as a list to the tokenizer. You don't need to tokenize them one by one.
tokenizer(tweets, max_length=max_len, padding='max_length', add_special_tokens=True)
(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...