I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word "hello" -- not "Hello", "hello", "hello.", and so on).

How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)? I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.

  • 1
    $\begingroup$ Space is cheap; save both. Especially if it's merely text! $\endgroup$
    – Emre
    Jun 2 '17 at 16:32
  • $\begingroup$ @Emre there must be more to it than that, though. You're telling me Google (assuming they don't do anything else to solve this problem) should roughly double the number of data centers they have just to accommodate this configuration? $\endgroup$
    – blacksite
    Jun 2 '17 at 16:44
  • 1
    $\begingroup$ How much space do you think all the news articles in the world take? You can always compress the raw text. $\endgroup$
    – Emre
    Jun 2 '17 at 16:51
  • $\begingroup$ They keep them and you can even see it! Go do a Google Search. Next to a result, click the little green down arrow, and then "Cached". That's their copy. $\endgroup$
    – CalZ
    Jun 2 '17 at 19:56
  • $\begingroup$ @CalZ I'm sure Google is keeping everything. But my question is if this duplication of databases - one raw, one cleaned - is the best approach. I imagine it would be tough explaining to my boss that we need to spin up a new database to accommodate my recommender. $\endgroup$
    – blacksite
    Jun 3 '17 at 4:21

Think of this problem as a pipeline of steps to automate and re-run, and not just the ML step at the end:

  1. Read in raw documents.
  2. Stem words, remove stop words.
  3. Perform TF-IDF
  4. Train model on cleaned up data.

Now when it comes time to score:

  1. Load your saved model.
  2. Repeat steps #2 and #3 from above.
  3. Send the cleaned data into your model for a prediction/recommendation.

Here is an example in Python of something like that:



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.