1
$\begingroup$

I have a dataset of choices (between A,B and C) done by certain users, and I want to train a neural network to predict the choices. I divide in train and test sets.

An instance is formed by: [UserId, some features, choice]. Choice is the target variable. Same user can have multiple instances (with different features and choice).

My assumption is that the same user makes often the same choice, so I want a feature describing the "frequency" of a choice (e.g. User 1 choose A 60% of times). From which dataset can I compute this frequency?

  • I cannot use the training set, otherwise is overfitting (with a frequency=0, the model is sure it was never chosen in the training-set).
  • I do not have an additional dataset of choices in a different period (this would be ideal).
  • Can I compute the frequency from the test-set? In that way, I'm actually training a model (on training set), based on the frequencies in a different period (the test set), which is what I want. Does it overfit the performances in the test-set? I assume not, since the model is trained on the training set. My results confirm similar performances in the two sets.
Improve this question
$\endgroup$
2
  • $\begingroup$ Can you please edit the question to clarify a few details: first, what would an instance represent in your design? Is it a single user, a group of users? If an instance represents a single user, I don't see what the frequency would represent. Also if the goal is to predict these choices then the choice is the target variable, right? normally this means frequent choices simply have more instances, i.e. the fact that many users choose X is reflected by the fact that the data contains many instances labelled X. $\endgroup$
    – Erwan
    Commented Jan 13, 2021 at 20:59
  • $\begingroup$ I clarified the question. An instance is a choice of a certain user. Each user can have multiple instances, so the frequency is how often that user made that choice (e.g. how often user1 choose the choice A) $\endgroup$
    – A M
    Commented Jan 13, 2021 at 23:33

1 Answer 1

Reset to default
1
$\begingroup$

I think that the issue depends on what you'd expect the model to learn:

  • If the model is supposed to "know" the users it has seen during training, i.e. exploit the user id in order to infer particular choices for a specific user, then I don't see the point in adding this kind of frequency feature: the model already "knows" what choices this user tends to make, and it's supposed to know it only from the training set. Disadvantage: for any new/unique user not seen during training, the model probably has to fall back to a generic prediction.
  • If the user id is not used (removed from the features), then I think it makes sense to add the frequency features. I would see this mostly as a matter of feature engineering: let's assume that there are 3 features which represent how many times so far this user has chosen option A, B and C. The counting is made from any previous data available for this user. Of course, this is assuming that the same features can be calculated at testing time (equivalently, in a production environment).

So in the second option the features exist before any question about what is the training set and what is the test set. However they introduce a potential risk of data leakage, so what I would do is to separate the training set and test set based on the users, i.e. make sure that users in the training set and test set don't overlap.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.