I have a dataset of choices (between A,B and C) done by certain users, and I want to train a neural network to predict the choices. I divide in train and test sets.
An instance is formed by: [UserId, some features, choice]. Choice is the target variable. Same user can have multiple instances (with different features and choice).
My assumption is that the same user makes often the same choice, so I want a feature describing the "frequency" of a choice (e.g. User 1 choose A 60% of times). From which dataset can I compute this frequency?
- I cannot use the training set, otherwise is overfitting (with a frequency=0, the model is sure it was never chosen in the training-set).
- I do not have an additional dataset of choices in a different period (this would be ideal).
- Can I compute the frequency from the test-set? In that way, I'm actually training a model (on training set), based on the frequencies in a different period (the test set), which is what I want. Does it overfit the performances in the test-set? I assume not, since the model is trained on the training set. My results confirm similar performances in the two sets.