Correcting ALL CAPS for human and algorithmic consumption

Question

United States federal tax returns tend to be written in ALL CAPS to facilitate OCR. This practice has persisted even when returns are filed electronically. Thus, much of the text in the IRS 990 dataset is in all caps. This makes it hard to read, and limits the ability of algorithms such as Treebank to accurately tag part of speech.

I understand that the approach of the Stanford POS tagger may be more amenable to correction of capitalization, but in practice, I have not had much luck in using it to correct the text in the IRS 990 corpus, in which nearly every sentence contains one or more proper nouns.

Are there any "tricks of the trade" for improving the performance of an off-the-shelf POS tagger when using ALL CAPS text, and/or an algorithm that may do better at identifying the proper nouns therein?

Answer 1

If sensitivity to case is breaking your models you have two options:

1. Train or find a new model that's case-insensitive. This is probably the easiest thing to do. The Stanford parser has one.

2. Train a model to correct the case of your input, this is sometimes called truecasing. The Stanford Parser has this functionality too.