Dealing with diverse text data

Question

I'm currently working with a dataset with a wide range of document lengths -- anywhere from a single word to a full page of text. In addition, the grammatical structure and use of punctuation varies wildly from document to document. The goal is to classify those documents into one of about 10-15 categories. I'm currently using ridge regression and logistic regression for the task, and CV for the alpha values of ridge. The feature vectors are tf-idf ngrams.

Recently I've noticed that longer documents are much less likely to be categorized. Why might this be the case, and how can one "normalize" for this kind of variation? As a more general question, how does one typically deal with diverse data sets? Should documents be grouped based off of metrics like document length, use of punctuation, grammatical rigor, etc. and then fed through different classifiers?

Answer 1

I'm not sure how you are applying a regression framework for document classification. The way I'd approach the problem is to apply a standard discriminative classification approach such as SVM.

In a discriminative classification approach the notion of similarity or inverse distance between data points (documents in this case) is pivotal. Fortunately for documents, there is a standard way of defining pairwise similarity. This is the standard cosine similarity measure which makes use of document length normalization to take different document lengths into account.

Thus, practically speaking, in cosine similarity you would work with relative term weights normalized by document lengths and hence document length diversity should not be a major issue in the similarity computation.

One also has to be careful when applying idf in term weights. If the number of documents is not significantly large the idf measure may be statistically imprecise thus adding noise to the term weights. It's also a standard practice to ignore stopwords and punctuations.