2
$\begingroup$

My question is focused around how to appropriately update an encoded feature set when a new category is introduced by the test data. I use the data in logistic regression and I know it is not a 'live' model (i.e. gradient descent is performed whenever new data is introduced) but do I have to retrain the model to account for added features or do I just add it to subsequent test set values.

To exemplify the problem consider a TV Show training set where each show has a 'networks' feature set that includes one or more of the following:

["abc","cbs","nbc"]

Then, in the testing set there is a TV Show with the feature set: 

["abc", "hulu"]

Would I have to add the new feature retroactively to the training data and retrain the model eventhough it will never occur? Wouldn't this introduce 'look-ahead-bias'?

How do I account for the added feature in the encoder going forward?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I think you have two options:

  • Automate your train/test pipeline so that one-hot encoding is part of it. If new categorical variables are introduced, they can be featured in the training dataset even if not very prevalent. This would introduce some bias if the nature of the TV show distribution has changed over time (e.g. 20 years ago there weren't as many options) but I don't necessarily think it is a show stopper.
  • If new possibilities are introduced over time but for whatever reason you can't retrain, then you should omit using that new value. This has its own disadvantages because in your example, it would be a TV show with no network.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the response. I was also thinking of adding an other category that would be included in training and only be assigned when new categories were introduced. But in the event that there is more than one new category there is no way to accomodate unless a series of weights were applied to the encoded values. Is this possible? $\endgroup$
    – Dan Temkin
    Commented May 18, 2017 at 1:32
  • $\begingroup$ I like the "other" category, not sure how you would handle the weights unless you inspected the new data and at that point you may as well explicitly train on them encoded separately. $\endgroup$
    – CalZ
    Commented May 18, 2017 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.