I have a multiclass problem (3 classes) that looks to predict if someone will buy a product, neutral or not. I have initial features of in-app activity data such as likes, share, bookmark, share, clicks, etc. (all of them were collected before the purchase date).
Do I still need to do Temporal Train-Validation-Test split or is a random one in scikit-learn enough? If yes, is Temporal splitting always the way to go for tabular data classification problems?
Thank you.
Temporal train-validation-test split is generally for time series problems or in cases where an external stimulus/intervention has led to a change in user behavior or some regime change in the dynamics of the system/the experiment. When you are certain the data points are not idependent and identically distrbuted,you cannot randomly shuffle them.
I am assuming based on reading the list of features in your question that neither of the above are true - you do not have a time component nor is there a reason to believe that some intervention has taken place. In this case a random split is reasonable.
Sounds like random split is the way to go, since you are using all previous data from a user to make one recommendation. You might need to fix the time dimension of in-app activity data considered for the prediction (e.g., three months), as I assume not all users will have the same activity length.
IMPORTANT:
If there are multiple data samples entries for each user (for example, multiple purchases from the same user), then I would recommend doing the split first on a PER-USER basis. At the end of the day, you want to predict user behavior, and you do that on a per-user basis. Otherwise you are corrupting your test set by including purchases by one user in both the test and the train set.
