There are a lot of ml-algorithms which cannot directly deal with categorical variables. A very common solution is to apply binary (dummy-) coding to still properly handle the categorical nature of the data.

Very often e.g. in sk-learn or apache-spark the actual dummy-coder can only handle numeric values. So label-encoding needs to be performed beforehand.

In a real live ml-scenario, the fitted model will encounter new and formerly not known data. Usually, such a label-encoder (string-indexer) for spark has the option to either skip (ignore) a row of data which contains any unknown value or to throw an error. If multiple values require coding this can lead to a big loss of "new" data.

Are there any approaches which "tolerate" up to x new values per row and still properly evaluate the fitted pipeline?

An example for spark string-indexing + dummy-coding is shown below.

val df = spark.createDataFrame(Seq(
  (0, "a"),
  (1, "b"),
  (2, "c"),
  (3, "a"),
  (4, "a"),
  (5, "c")
)).toDF("id", "category")

val indexer = new StringIndexer()
val indexed = indexer.transform(df)

val encoder = new OneHotEncoder()
val encoded = encoder.transform(indexed)
encoded.select("id", "categoryVec").show()



1 Answer 1


For a categorical variable, if the fitted model encounters a previously "unseen" category, i.e. it did not exist in the training set when the model was trained; then, you should skip that record. You could also opt to throw an error so that you're notified of the existence of new categories and can re-train the model based on that trigger.

If you skip the records with new categories, you would still be able to evaluate the pipeline successfully. That may be the preferred option for a fully automated production setup.

The one-hot encoding creates a new "column" of data for a new category and if this hasn't been used for training, the ML algorithm has no way of knowing how to use the new dummy variable.

In your sample code, each of the categories are encoded into new variables e.g. is_category_a = 0/1, is_category_b = 0/1 , is_category_c = 0/1, etc. If such a model is sent data with a new category "d", then it would be encoded in another column called is_category_d = 0/1 , but the model would ignore this column (or throw an error) since it doesn't expect its input matrix to contain is_category_d.

Lets assume you've fit a linear regression model with the coefficients as: $$ y = 1 + 2.is\_category\_a + 3.is\_category\_b + 4.is\_category\_c $$ Now, when you try to evaluate a record with new category = "d", then, the model is not able to use the new dummy variable since it doesn't have a coefficient for is_category_d .

Hence, so you should not "tolerate" new categories and process such records.

  • $\begingroup$ Does this hold true even for decision trees? Assuming there are three factor columns, 1 contains a new label wouldn't it be possible to use the 2 known columns to classify the sample? $\endgroup$ Dec 9, 2016 at 9:51
  • $\begingroup$ Yes, the same concept is applicable to decision trees. Lets assume a tree had been constructed using only categories a/b/c, then its decision rules for splits would only contain checks on whether the category is 'a' or 'b' or 'c'. If we use it to determine the leaf node for a record with category 'd', then it wouldn't have any split rule for category 'd' as it wasn't trained on such data. In such cases the tree would give the result (end up in a leaf node) where none of the conditions matched (assuming if the tree model did not give an error for this record). $\endgroup$ Dec 10, 2016 at 3:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.