What kind of error measures do RMSE and nDCG give while evaluating a recommender system, and how do I know when to use one over the other? If you could give an example of when to use each, that would be great as well!
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
nDCG is used to evaluate a golden ranked list (typically human judged) against your output ranked list. The more is the correlation between the two ranked lists, i.e. the more similar are the ranks of the relevant items in the two lists, the closer is the value of nDCG to 1.
RMSE (Root Mean Squared Error) is typically used to evaluate regression problems where the output (a predicted scalar value) is compared with the true scalar value output for a given data point.
So, if you are simply recommending a score (such as recommending a movie rating), then use RMSE. Whereas, if you are recommending a list of items (such as a list of related movies), then use nDCG.
-
$\begingroup$ Correct me if I do not understand this properly. So if there is no "golden ranked list (typically human judged)", there is no point of using nDCG as evaluation, right? The problem would be simply a regression (saying we are a continuous target), and RMSE would suffice. Thanks $\endgroup$ Commented Jul 25, 2018 at 9:09
-
2$\begingroup$ yes... a standard use case requires the presence of a manually judged 'ground-truth' ideally ordered list... however, depending on your use-case you might construct such an ideal list automatically with partial human supervision... an example that i have in mind is that if u have a list of docs as D1(1) D2(1) D3(0), u might actually ask a human if D1 is better than D2 regardless of the query... in which case u may 'estimate' an ideal ranked list as D1(2) D2(1) D3(0) if D1 is judged more 'useful' than D2... $\endgroup$– DebasisCommented Jul 26, 2018 at 11:04
-
$\begingroup$ Thanks for your quick reply. That is a very idea. The challenge here is that in that in a particular domain there should be a reasonably good target that rank fairly good the orderings! $\endgroup$ Commented Jul 27, 2018 at 8:53
$\begingroup$
$\endgroup$
nDCG is a ranking metric and RMSE is not. In the context of recommender systems, you would use a ranking metric when your ratings are implicit (e.g., item skipped vs. item consumed) rather than explicit (the user provides an actual number, a la Netflix).
