I am an absolute beginner and am trying to implement collaborative filter for furniture ecommerce (think wayfair). I need some guidance about cross-validation strategy.
Situation: I am working on a fictitious dataset relating to customers' furniture purchases. Let's assume that the customers buy furniture and related items at a time. "At a time" because they mostly buy them either when furniture breaks or when they need to be replaced because the needs have changed (kids have grown up etc.). Hence, a typical buying cycle is somewhere between 6 and 9 years. Moreover, by "related items" I mean that they would buy related items such as mattresses, bed sheets along with bed.
When the customer visits the furniture store website for the first time (defined as "new customer"), they indicate their preferences in terms of things they are looking for in the furniture: size, handling etc. This was done to avoid "cold-start problem." in recommender system.
My goal is to recommend products for each new customer who visits the website, which happens every 6 to 9 years because of buying cycle explained above. Predicting right products for the new customer will help reduce sales cycle and possibly improve customer satisfaction.
The question is how should I split the dataset into test and validation?
Reading above sources, I believe that realistically, I have three options and here are my thoughts for each:
1) Separate recent k transactions for each customer for testing: This might suffer from data leakage problem i.e. for some customers I might have data from 2011 to 2020 in training set, but for others, I might have data from 2016 to 2019. I might already know that during Covid (2020), office desk was popular. So, I would automatically recommend this for users with training data until 2019. This will be considered cheating, I think.
2) Holdout: Shuffle and then separate last k transactions for testing: Timing is important. So, this might not work because I might use a future transaction to predict something in the past, which wouldn't make any sense.
3) Randomly shortlist k customers for testing, and then train using remaining customers: I am a big fan of this method because my goal is to predict product recommendations for new customers. I won't have cold-start problem because the customers would provide the preferences, and I will be able to use preferences from training customers to match (using collaborative filter) preferences of new customers. However, I am not too sure whether training customers' preferences will represent new customers.
Am I on the right track? I have spent a lot of time on this problem, and I would appreciate any guidance.