I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one token). The texts greatly vary in terms of length, so I try to take length into consideration in my models. I first tried to divide the dictionary count in each text by text length (dictionary count/text length = k). Because I use a regression model, I then take the square root of k to normalize the data (which I use as a dependent variable). In a second model, I did not divide the dictionary count by text length, but I controlled for length as a predictor in a linear regression model (I still take the square root of the dictionary count). The results across these models are substantially different (Especially in terms of statistical significance). I am struggling to decide which model is better, as I could not locate the papers on the subject matter in my field (political science) or elsewhere. Any suggestions?
1 Answer
I'm not sure that I completely follow the first approach with the square root, but the normalization is likely to cause some bias anyway (if only because the number of zeros will be higher in a short text).
In the second option, my understanding is that you add the length of the text as an independent (input) variable for the linear regression, right? This might still have the same kind of bias, and since the variable doesn't have the same nature as others it will more likely be ignored by the model.
Between the two models I would lean towards the first one. But if I may offer a suggestion, a different solution would be to resample the texts as follows: decide the size of text N, typically selecting the shortest size (possibly a bit higher, in case there are a few very short documents). Then for each document larger than this, randomly pick N tokens as the resampled document. The counts are comparable across resampled documents, so the issue of the question disappears.
