0
$\begingroup$

Are there any potential issues on performing sentiment analysis using the first 100 words of a very large essay that is of 500 to 700 words. I am having to do this because since most transformer models have a upper limit of 500 words.

$\endgroup$

1 Answer 1

1
$\begingroup$

Are there any potential issues on performing sentiment analysis using the first 100 words?

No. You could do it and probably get away with it in certain instances. But if proper/accurate results are your aim, this might give you False results.

Why do you want to select only the first 100 words? As you mention, transformers token limit is 500 words. So why limit yourself to 100?

Also you can use more than 500 by some basic NLP preprocessing techniques. Just remove the stop words, punctuation marks, numbers, URL's and extra whitespaces.

All the above steps would have two fold advantage. It will increase your model accuracy and also decrease the total word length of your text.

$\endgroup$
1
  • $\begingroup$ Thank you, appreciate your insights! $\endgroup$
    – Deepak
    Commented May 28, 2023 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.