0
$\begingroup$

I am using the text of comments on a forum to predict how many upvotes it will get. I want to be able to say, "Reviews with X, Y, Z words are more upvoted". So to do this, I want to use text features in a regression. In particular,

What model should I use to maximize interpretability of coefficients?

$\endgroup$
0
$\begingroup$

I suppose you have a binary outcome (upvote: yes/no). In this case you could use simple linear (ols) regression (with lasso penalty). Each word (in a bag of words) is a „dummy“ here. If you look at the predicted coefficients, you can directly interpret them as „marginal effects“. Higher value means higher chance of getting an upvote (if the word is present). You can also directly see the magnitude of exp. increase in upvote probability.

One problem is that OLS is unconstrained wrt to y. So you can end up with predicted probability of upvote > 1. Use logistic regression if this bothers you. Under logit you will have similar results. Positive/negative coefficients indicate if a word increases/decreases the probability of an upvote. But because logit uses a transformation to squeeze y into a interval from zero to one, the coefficients are the log-odds and you cannot directly infer marginal effects. You would need to calculate this separately.

I tend to say: use OLS and get quick and dirty results if you are not interested in super precise estimates but if you merely look for robust estimates for which words are important.

However, if you really want to do this in a sound way, you would also need to think about „interaction“ of words (positive or negative effect on y), which are „masked“ on approaches as described above.

$\endgroup$
4
  • $\begingroup$ Hi, upvotes are not binary -- it is a number. You can have like 5 or 6 upvotes. $\endgroup$ Jul 4 '19 at 10:41
  • $\begingroup$ Okay, so it is a count: in this case OLS would be the first thing to try. You can also opt for poisson regression or multinominal logit, but OLS might work well here. $\endgroup$
    – Peter
    Jul 4 '19 at 11:10
  • $\begingroup$ Wouldn't multicollinearity mean that OLS would be very bad? $\endgroup$ Jul 4 '19 at 11:15
  • 1
    $\begingroup$ True, this can be a problem. You need to check this baed on the data. I think using lasso you may be able to reduce this problem, i.e. if the coefficients of "many" words are shrunken to zero. Also drop stopwords and short words and/or do stemming and/or use n-grams. So you can "break up" some of the (possible) linear combinations in X. Multicolinearity affects causal interpretation of coefficients (not necessarily predictive power). You need to be aware that causal interpretation in the setup described by you is questionable anyway because you can't control the context of a expression. $\endgroup$
    – Peter
    Jul 4 '19 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.