I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word "hello" -- not "Hello", "hello", "hello.", and so on).

How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)? I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.

  • 1
    $\begingroup$ Space is cheap; save both. Especially if it's merely text! $\endgroup$
    – Emre
    Commented Jun 2, 2017 at 16:32
  • $\begingroup$ @Emre there must be more to it than that, though. You're telling me Google (assuming they don't do anything else to solve this problem) should roughly double the number of data centers they have just to accommodate this configuration? $\endgroup$ Commented Jun 2, 2017 at 16:44
  • 1
    $\begingroup$ How much space do you think all the news articles in the world take? You can always compress the raw text. $\endgroup$
    – Emre
    Commented Jun 2, 2017 at 16:51
  • $\begingroup$ They keep them and you can even see it! Go do a Google Search. Next to a result, click the little green down arrow, and then "Cached". That's their copy. $\endgroup$
    – CalZ
    Commented Jun 2, 2017 at 19:56
  • $\begingroup$ @CalZ I'm sure Google is keeping everything. But my question is if this duplication of databases - one raw, one cleaned - is the best approach. I imagine it would be tough explaining to my boss that we need to spin up a new database to accommodate my recommender. $\endgroup$ Commented Jun 3, 2017 at 4:21

1 Answer 1


Think of this problem as a pipeline of steps to automate and re-run, and not just the ML step at the end:

  1. Read in raw documents.
  2. Stem words, remove stop words.
  3. Perform TF-IDF
  4. Train model on cleaned up data.

Now when it comes time to score:

  1. Load your saved model.
  2. Repeat steps #2 and #3 from above.
  3. Send the cleaned data into your model for a prediction/recommendation.

Here is an example in Python of something like that:



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.