Failure tolerant factor coding

Question

There are a lot of ml-algorithms which cannot directly deal with categorical variables. A very common solution is to apply binary (dummy-) coding to still properly handle the categorical nature of the data.

Very often e.g. in sk-learn or apache-spark the actual dummy-coder can only handle numeric values. So label-encoding needs to be performed beforehand.

In a real live ml-scenario, the fitted model will encounter new and formerly not known data. Usually, such a label-encoder (string-indexer) for spark has the option to either skip (ignore) a row of data which contains any unknown value or to throw an error. If multiple values require coding this can lead to a big loss of "new" data.

Are there any approaches which "tolerate" up to x new values per row and still properly evaluate the fitted pipeline?

An example for spark string-indexing + dummy-coding is shown below.

val df = spark.createDataFrame(Seq(
  (0, "a"),
  (1, "b"),
  (2, "c"),
  (3, "a"),
  (4, "a"),
  (5, "c")
)).toDF("id", "category")

val indexer = new StringIndexer()
  .setInputCol("category")
  .setOutputCol("categoryIndex")
  .fit(df)
val indexed = indexer.transform(df)

val encoder = new OneHotEncoder()
  .setInputCol("categoryIndex")
  .setOutputCol("categoryVec")
val encoded = encoder.transform(indexed)
encoded.select("id", "categoryVec").show()

http://spark.apache.org/docs/latest/ml-features.html#onehotencoder

Answer 1

For a categorical variable, if the fitted model encounters a previously "unseen" category, i.e. it did not exist in the training set when the model was trained; then, you should skip that record. You could also opt to throw an error so that you're notified of the existence of new categories and can re-train the model based on that trigger.

If you skip the records with new categories, you would still be able to evaluate the pipeline successfully. That may be the preferred option for a fully automated production setup.

The one-hot encoding creates a new "column" of data for a new category and if this hasn't been used for training, the ML algorithm has no way of knowing how to use the new dummy variable.

In your sample code, each of the categories are encoded into new variables e.g. is_category_a = 0/1, is_category_b = 0/1 , is_category_c = 0/1, etc. If such a model is sent data with a new category "d", then it would be encoded in another column called is_category_d = 0/1 , but the model would ignore this column (or throw an error) since it doesn't expect its input matrix to contain is_category_d.

Lets assume you've fit a linear regression model with the coefficients as: $$ y = 1 + 2.is\_category\_a + 3.is\_category\_b + 4.is\_category\_c $$ Now, when you try to evaluate a record with new category = "d", then, the model is not able to use the new dummy variable since it doesn't have a coefficient for is_category_d .

Hence, so you should not "tolerate" new categories and process such records.