I've been searching around for a while without any luck - hopefully someone more knowledgable can give me some advice about the following ML problem that I've been thinking about:

Say you are trying to predict the Rotten Tomato "Tomatometer" score of a film before it's released. Typically you might approach this by compiling a list of features and labels for some existing films and input this into a supervised ML algorithm.

In this example, the feature list will be standard metrics that describe the film such as, budget, filming duration, number of actors, etc., while the label is the Tomatometer score of the film that is given as a value between 0 to 100. Every film can be expressed using this score, but individually they are spread across many genres, country of production, etc. meaning that there are natural subsets within the training data.

Let's say our training data contains only films belonging to five genres (e.g. Action, Thriller, Horror, Fantasy and Documentary), while we want our algorithm to be applicable to films outside of this genre (e.g. Sci-Fi or Animation), but for the sake of question, we do not have access to these entire categories. In this example, were also assuming that some features will be more important to certain genres than others, for example, a large cast size may correlate more with the score for Action films than for Animations.

What is the general way to transform the data to make it invariant to the subgroup (genre), or what ML algorithm can be used here (if any)? Is there a common name for this situation (some keywords I can search?)


1 Answer 1


I'll try to rephrase your question: How does one use the information contained in some categorical feature x to predict y in the presence of unseen category values in the test set?

Assuming the train set is representative of the test set distribution, you'd expect the large categories to also be present in the test set.

We're thus mostly concerned with the small categories which might be present in the train set but absent from the test set and/or present in the test set but absent from the train set.

One way of dealing with such situation is merging small categories (e.g. under 2% of all observations) into a single category. That way you treat any new category level as part of the merged category.

Below I share how I implement the above in python, in a way that can be combined in scikit learn pipelines:

from collections import Counter
import pandas as pd

class mergeSmallCategoryLevels():
    def __init__(self, min_frac):
        self.min_frac = min_frac
    def fit(self, X, y=None, **fit_params):
        category_counts = pd.DataFrame.from_dict(Counter(X), orient = "index", columns = ["count"])
        min_category_count = len(X)*self.min_frac
        large_categories = category_counts[category_counts["count"] >= min_category_count]
        self.large_categories = list(large_categories.index.values)
        return self

    def transform(self, X, **transform_params):
        ans = [val if val in self.large_categories else ".merged" for val in X]
        ans = pd.DataFrame({"category_feature":ans})
        return ans.to_numpy().reshape(-1, 1)
  • $\begingroup$ Thanks for the implementation example! Can this class be used as-is, or should it inherit some base transformer or something? $\endgroup$ Oct 4, 2020 at 10:51
  • $\begingroup$ One tip I'd add to this (good) answer - make sure that your cross-validation set is also stratified by categories, so that your train set and cross-validation set are both representative of the real distribution (well, as representative as you can). $\endgroup$ Oct 4, 2020 at 10:54
  • 1
    $\begingroup$ @ItamarMushkin No need to inherit anything. Estimators in scikit learn only need to implement fit and transform methods. I've used something very similar in a scikit learn pipeline. $\endgroup$
    – Iyar Lin
    Oct 5, 2020 at 12:56
  • $\begingroup$ Why is category_counts a dataframe and not a series, if it has one column? $\endgroup$ Oct 6, 2020 at 5:51
  • $\begingroup$ You mean a pandas Series? While it has one column, it's index contains the actual category names, extracted in the list(large_categories.index.values) part $\endgroup$
    – Iyar Lin
    Oct 7, 2020 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.