I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word "hello" -- not "Hello", "hello", "hello.", and so on).
How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)? I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.