5
$\begingroup$

Which of the following is best (or widely used) for calculating item-item similarity measure in mahout and why ?

Pearson Correlation
Spearman Correlation
Euclidean Distance
Tanimoto Coefficient
LogLikelihood Similarity

Is there any thumb-rule to chose from these set of algorithm also how to differentiate each of them ?

$\endgroup$
1

1 Answer 1

3
$\begingroup$

For those not familiar, item-item recommenders calculate similarities between items, as opposed to user-user (or user-based) recommenders, which calculate similarities between users. Although some algorithms can be used for both, this question is in regard to item-item algorithms (thanks for being specific in your question).

Accuracy or effectiveness of recommenders is evaluated based on comparing recommendations to a previously collected data set (training set). For example, I have shopping cart data from the last six months; I'll use the first 5 months as training data, then run my various algorithms, and compare the quality against what really happened during the 6th month.

The reason Mahout ships with so many algorithms is because different algorithms are more or less effective in each data set you may work with. So, ideally, you do some testing as I described with many algorithms and compare the accuracy, then choose the winner.

Interestingly, you can also take other factors into account, such as the need to minimize the data set (for performance reasons), and run your tests only with a certain portion of the training data available. In such a case, one algorithm may work better with the smaller data set, but another may work with the complete set. Then, you get to weigh performance VS accuracy VS challenge of implementation (such as deploying on a Hadoop cluster).

Therefore, different algorithms are suited for different project. However, there are some general rules:

  1. All algorithms always do better with unreduced data sets (more data is better).
  2. More complex algorithms aren't necessarily better.

I suggest starting with a simple algorithm and ensuring you have high quality data. If you have additional time, you can implement more complex algorithms and create a comparison which is unique to your data set.

Most of my info comes from This study. You'll find lots of detail about implementation there.

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.