I have a classification problem using emails as my dataset.
I would like to use scores from various readability formulas as features for the classification. However, most of them are defined for passages of text longer than 100 words, and the python library that I am using (py-readability-metrics) cannot work for text shorter than this.
Around 40% of my dataset consists of text shorter than 100 words, so if I want to use those features I can't simply drop those data points without losing a significant percentage of my dataset.
Also, in order to keep things reproducible, I wouldn't want to generalize the metrics to work for shorter texts too.
What would you recommend as an imputation value to assign as a score for those short texts?
- Intuitively, the first that came to my mind is to find the score that means "easiest to read" for each formula, but that could not always be the case (shorter does not always mean easier to read) and also this would mess up the distributions.
- Then I thought about the mean value, which would make more sense because I have observed that the number of words is not directly correlated to the readability score.
Is there another possible imputation value that I missed? Or is 40% missing values simply too much for these features to be worth it in your opinion?