There are mainly three ways to evaluate a recommender system: offline, online and user study. For most academic papers, offline evaluation is used to show the improvements:
- They split the offline dataset into training and testing sets
- They train models on the training dataset
- They evaluate them on the testing dataset.
However, it seems that for most nonsequential recommendation works, the dataset splitting is random, without considering the temporal sequential order of the records. I think this may cause a problem of overestimating.
For example, consider evaluating itemCF on MovieLens: if the dataset is random split, the order of the samples will be shuffled. It means that we may use future data to train a model and make a prediction about the past:
The training dataset contains that Bob bought a Harry Porter book in April 2019 (which is the second purchase of Harry Porter), and the testing dataset contains that Bob bought a Harry Porter book in March 2019 (which is the first purchase of Harry Porter). An item-based CF method may recommend Harry Porter for the first purchase due to the second purchase, which is against the law of causation with a result of overestimating the performance of item-based CF.
It is kind of cheating. And I fail to see this problem being discussed somewhere. So, I doubt the real performance of the offline evaluation for recommender system and would like to hear more voices about this.