1
$\begingroup$

My question is focused around how to appropriately update an encoded feature set when a new category is introduced by the test data. I use the data in logistic regression and I know it is not a 'live' model (i.e. gradient descent is performed whenever new data is introduced) but do I have to retrain the model to account for added features or do I just add it to subsequent test set values.

To exemplify the problem consider a TV Show training set where each show has a 'networks' feature set that includes one or more of the following:

["abc","cbs","nbc"] 

Then, in the testing set there is a TV Show with the feature set:

["abc", "hulu"] 

Would I have to add the new feature retroactively to the training data and retrain the model eventhough it will never occur? Wouldn't this introduce 'look-ahead-bias'?

How do I account for the added feature in the encoder going forward?

$\endgroup$
2
$\begingroup$

I think you have two options:

  • Automate your train/test pipeline so that one-hot encoding is part of it. If new categorical variables are introduced, they can be featured in the training dataset even if not very prevalent. This would introduce some bias if the nature of the TV show distribution has changed over time (e.g. 20 years ago there weren't as many options) but I don't necessarily think it is a show stopper.
  • If new possibilities are introduced over time but for whatever reason you can't retrain, then you should omit using that new value. This has its own disadvantages because in your example, it would be a TV show with no network.
$\endgroup$
  • $\begingroup$ Thanks for the response. I was also thinking of adding an other category that would be included in training and only be assigned when new categories were introduced. But in the event that there is more than one new category there is no way to accomodate unless a series of weights were applied to the encoded values. Is this possible? $\endgroup$ – Dan Temkin May 18 '17 at 1:32
  • $\begingroup$ I like the "other" category, not sure how you would handle the weights unless you inspected the new data and at that point you may as well explicitly train on them encoded separately. $\endgroup$ – CalZ May 18 '17 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.