Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images. This works well, except when it doesn't. Many docs are old, scanned images and what Tika extracts is gibberish.
Using Spark on Hadoop and either ML or MLlib (haven't settled, though I like ML better).
So far getting best results from a pipeline using Naive Bayes that removes Stopwords, tokenizes and Countvectorizes features (no Tf-Idf). Total bag-of-words approach. Next best is using ML to tokenize, Tf with Idf and into LogisticregressionwithLBFGS.
Anyway, the thought occurred to me that the model uses many docs that are junk. Literally just strings of gibberish like "mmmmmmmm aaannnammmmrrr hdhhdhhhhhjjj..."
This isn't good, but since I'm operating at scale it's just what happened. Certainly I could pick through 10,000 training docs and remove the bad examples, but there has to be an easier way. Is there?
The title of this question belies my brainstorm that there might be a way to discount, downweight or outright ignore tokens that aren't discernable by a dictionary. Is there?
Open to any and all advice or approaches to get better precision out of this model.