Are there any potential issues on performing sentiment analysis using the first 100 words of a very large essay that is of 500 to 700 words. I am having to do this because since most transformer models have a upper limit of 500 words.
1 Answer
$\begingroup$
$\endgroup$
1
Are there any potential issues on performing sentiment analysis using the first 100 words?
No. You could do it and probably get away with it in certain instances. But if proper/accurate results are your aim, this might give you False results.
Why do you want to select only the first 100 words? As you mention, transformers token limit is 500 words. So why limit yourself to 100?
Also you can use more than 500 by some basic NLP preprocessing techniques. Just remove the stop words, punctuation marks, numbers, URL's and extra whitespaces.
All the above steps would have two fold advantage. It will increase your model accuracy and also decrease the total word length of your text.
