Help with type of ML problem: when training data is spread across different subgroups/categories

Question

I've been searching around for a while without any luck - hopefully someone more knowledgable can give me some advice about the following ML problem that I've been thinking about:

Say you are trying to predict the Rotten Tomato "Tomatometer" score of a film before it's released. Typically you might approach this by compiling a list of features and labels for some existing films and input this into a supervised ML algorithm.

In this example, the feature list will be standard metrics that describe the film such as, budget, filming duration, number of actors, etc., while the label is the Tomatometer score of the film that is given as a value between 0 to 100. Every film can be expressed using this score, but individually they are spread across many genres, country of production, etc. meaning that there are natural subsets within the training data.

Let's say our training data contains only films belonging to five genres (e.g. Action, Thriller, Horror, Fantasy and Documentary), while we want our algorithm to be applicable to films outside of this genre (e.g. Sci-Fi or Animation), but for the sake of question, we do not have access to these entire categories. In this example, were also assuming that some features will be more important to certain genres than others, for example, a large cast size may correlate more with the score for Action films than for Animations.

What is the general way to transform the data to make it invariant to the subgroup (genre), or what ML algorithm can be used here (if any)? Is there a common name for this situation (some keywords I can search?)

Answer 1

I'll try to rephrase your question: How does one use the information contained in some categorical feature x to predict y in the presence of unseen category values in the test set?

Assuming the train set is representative of the test set distribution, you'd expect the large categories to also be present in the test set.

We're thus mostly concerned with the small categories which might be present in the train set but absent from the test set and/or present in the test set but absent from the train set.

One way of dealing with such situation is merging small categories (e.g. under 2% of all observations) into a single category. That way you treat any new category level as part of the merged category.

Below I share how I implement the above in python, in a way that can be combined in scikit learn pipelines:

from collections import Counter
import pandas as pd

class mergeSmallCategoryLevels():
    def __init__(self, min_frac):
        self.min_frac = min_frac
        
    def fit(self, X, y=None, **fit_params):
        category_counts = pd.DataFrame.from_dict(Counter(X), orient = "index", columns = ["count"])
        min_category_count = len(X)*self.min_frac
        large_categories = category_counts[category_counts["count"] >= min_category_count]
        self.large_categories = list(large_categories.index.values)
        return self

    def transform(self, X, **transform_params):
        ans = [val if val in self.large_categories else ".merged" for val in X]
        ans = pd.DataFrame({"category_feature":ans})
        return ans.to_numpy().reshape(-1, 1)