On the offline evalution of recommender system

Question

There are mainly three ways to evaluate a recommender system: offline, online and user study. For most academic papers, offline evaluation is used to show the improvements:

1. They split the offline dataset into training and testing sets
2. They train models on the training dataset
3. They evaluate them on the testing dataset.

However, it seems that for most nonsequential recommendation works, the dataset splitting is random, without considering the temporal sequential order of the records. I think this may cause a problem of overestimating.

For example, consider evaluating itemCF on MovieLens: if the dataset is random split, the order of the samples will be shuffled. It means that we may use future data to train a model and make a prediction about the past:

The training dataset contains that Bob bought a Harry Porter book in April 2019 (which is the second purchase of Harry Porter), and the testing dataset contains that Bob bought a Harry Porter book in March 2019 (which is the first purchase of Harry Porter). An item-based CF method may recommend Harry Porter for the first purchase due to the second purchase, which is against the law of causation with a result of overestimating the performance of item-based CF.

It is kind of cheating. And I fail to see this problem being discussed somewhere. So, I doubt the real performance of the offline evaluation for recommender system and would like to hear more voices about this.

Answer 1

Sequential data (i.e. data that can be understood in sequences, such as time series) require apposite train-test splitting. When dealing with sequences, the split must be temporal as well.

It's the same principle that that must be followed when training language models, to make the most common example. This is well known in the literature.

I suggest you to find a timestep threshold in your data, and use that to split between train and test set.