0
$\begingroup$

I trained the LogisticRegression model with TF-IDF (both birgams and unigrams) and while predicting class it revealed that in longer texts (up to 3000 symbols)it works better that if I use short (+-100 symbols) texts. I assume that the reason is that bigram weights "work better" on longer texts, but it doesn't help me to understand why.

So any advice what to read to clarify this situation is welcome

Improve this question
$\endgroup$
6
  • $\begingroup$ What kind of differences are you seeing in the Evaluation Metrics? $\endgroup$ – Taylrl Nov 17 '20 at 16:50
  • $\begingroup$ @Taylrl the weighted average of precission and recall for longer texts is 0.70, 0.67 while for shorter 0.68 for both $\endgroup$ – jas_0n Nov 17 '20 at 17:33
  • $\begingroup$ Not that different then really. It might only take a couple of good correct answers on the smaller text to bring up the scores. Is this something you are seeing across a lot of different texts? $\endgroup$ – Taylrl Nov 17 '20 at 17:48
  • $\begingroup$ @Taylrl we one strange thing I revealed is, if binary pos-neg classification problem, in case I return not the class itself, but the probability of text to be classified pos or neg (predict_proba method in sklearn for instance), the longer texts are likely to be tended to a certain class. $\endgroup$ – jas_0n Nov 17 '20 at 18:03
  • $\begingroup$ @Taylrl I mean, if I put some long text that describes ones death and the sorrow felt by his relatives (basically the next with many "neg tokens" - tragic loss, found dead, the funeral ceremony etc) the probability scores will be like 0.95, 0.05 for neg, pos labels, while if I put some "general" sentence like "N found dead in his apartments yesterday about 7pm. We grieve with his family", the scores will be smth like 0.69, 0.31 respectively. $\endgroup$ – jas_0n Nov 17 '20 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.