Figure out relative importance of entity attributes

I'm trying to understand how various aspects of a movie contribute to its gross revenue. I want to rank a movie's attributes in that sense - the attributes that most strongly determine the revenue are ranked higher.

Let $$A_1,\ldots,A_n$$ be a list of attributes of a movie and let the possible values of $$A_i$$ be $$a_{i1},a_{i2},\ldots$$. Many of these attributes (like primary genre) are categorical and some of them (like rating) are continuous.

Approach 1: Consider $$A_1$$: I can form groups of movies having the same value of $$A_1$$, e.g. all movies with $$A_1=a_{12}$$ form a group. The other attributes in a group can vary freely. I can then calculate the mean of the revenues of all movies within a group, and then take the variance of means of all groups.

This will give me the "variation in average revenue as we change $$A_1$$ values". If this variation is high, that means changing $$A_1$$ significantly affects the average revenue - so $$A_1$$ should be a highly ranked attribute.

Approach 2: Again consider $$A_1$$: fix the values of all other attributes $$A_2,\ldots,A_n$$ and look at movies with the same values for $$A_2,\ldots,A_n$$ but different values of $$A_1$$. Find the variance in revenues of such movies - call it "$$A_1$$ variance". The attributes $$A_i$$ with highest "$$A_i$$ variance" will be ranked higher.

Approach 3: Train some ML model (not sure which one) with revenue as target variable and attributes as features. Then look at feature importances to get attribute importances.

A few queries:

1. What assumptions do I need to check for approach 1? e.g. minimum size of a group, distribution of other attribute values within a group, etc.
2. Are there any potential flaws or gotchas in approaches 1 and 2?
3. What approach would you prefer out of the three?
4. What ML model should be used for approach 3? I'm confused because there are plenty of regression models: Linear regression, GB regression, random forest regressor, etc.
• If your question is what aspects matters, then start with a simple linear least squared regression. The coefficients are easy to interpret. Then if you think there are interactions between aspects you can create interaction terms or drop aspects that seem to not matter. Does that make sense? If on the other hand you want to make predictions, you can still start with a regression but other machine learning techniques might work better. You don't get the same easy way interpret the importance of the aspects. Feb 11 at 19:05
• @Vincent: That's a good point, but most of the features would be categorical and I'm not sure linear regression would be appropriate there? Even if the features were continuous, it'd be a long shot that they'd satisfy the standard linear regression assumptions. That's my current thought process anyway Feb 11 at 19:11
• user9343456 Are you trying to find the important characteristics or predict the performance of movies? I would still do the standard regress as a first step, else you sample size is small and the number of movie characteristics is large, its easy to do. Then I would look at an option to reduce the dimensions. I would look at clustering the characteristics and giving the clusters names. I suppose SVM would work also. Feb 12 at 15:45

When you say

"...how various aspects of a movie contribute to its gross revenue."

I'm assuming you want

"If I were to change $$A_1$$ of a movie, how would it's gross revenue be affected?"

Which is a causal statement. With this assumption in mind, I will rewrite Approach 1 and 2;
Approach 1 - Don't control for anything, and measure the effect $$A_1$$ has on revenue
Approach 2 - Control for everything, and measure the effect $$A_1$$ has on revenue.
Both have fundamental problems, stemming from how the data was generated, and how attributes affect each other;
Approach 1's problem is confounders, attributes that affect both $$A_1$$ and revenue.
Here's a good example of a confounder:

It has been seen that children with a larger shoe size have a better reading ability. The trick is that age affects both reading ability and shoe size, the older you get, the larger your shoe size, as well as (generally) better reading ability. That make age here a confounder of shoe size and reading ability.

So, for the movie example, genre might affect runtime as well as revenue, making it a confounder of runtime and revenue.
Approach 2's problem is colliders (and potentially mediators). A collider is an attribute that $$A_1$$ and revenue affect.

For example if we say that both talent and beauty contribute to an actor's success, then, if we look at successful actors, we will see a negative correlation between beauty and talent (basically saying being beautiful makes you less talented). The reason for this is because, if we see an actor is unattractive, that increases our belief that the successful actor is very talented instead. Here, success was the collider, and when we controlled for it, it warped our connection between talent and beauty.

An example in the movies might be that the number of reruns of a movie is affected by revenue and runtime, making number of reruns a collider of the two, thus, if we control for it, it will warp the relationship between runtime and revenue in ways we don't want.

Another potential problem with Approach 2 is mediators, attributes that affect revenue and are affected by $$A_1$$

An example of a mediator would be Fire → Smoke → Fire Alarm. Fire creates Smoke which triggers the Fire Alarm. If we were to control for Smoke, we would be removing the mechanism Fire uses to trigger Fire Alarm, which would lead us to believe that Fire can't trigger Fire Alarm (which is technically true, fire itself doesn't trigger the Fire Alarm, fire doesn't have a direct effect on Fire Alarm).

We have to be aware of the mediators and that controlling for them turns the total effect of $$A_1$$ on revenue into the direct effect $$A_1$$ has on revenue.
So, If you're looking for the total effect, then Approach 2 falls victim to mediators as well, if you want the direct effect, that would make Approach 1 falls victim instead.

These are the fundamental flaws with Approaches 1 and 2, but there's also the problem of too small bin sizes, that would just be mostly noise (nicely explained in David Cian's answer). You can fix this by either getting more data or making more assumptions (e.g. assuming 10 minutes in a movie's runtime won't make a difference, and binning runtime into 10 minute wide bins).

So, in conclusion (for Approach 1 and 2), If you want to do something like Approach 1 and 2, you need to do a mix of the two. Find out what attributes are confounding and control for them, then, you can safely measure how revenue changes as $$A_1$$ changes, and rank the attributes based on that. Of course, to find the confounders, you need to know the model from which your attributes were generated from (known as a "Causal Model"), which can be tough to find.
There's a great book on of this, "The Book of Why" by Judea Pearl. I highly recommend it, if you're interested in this sort of thing.

Now, let's talk about Approach 3, the more volatile approach I would say;
The thing is, many algorithms can measure "feature importance", and it's hard to say what exactly the algorithm looks for when it says "important" (one thing is for certain though, it won't measure "If I were to change $$A_1$$ of a movie, how would it's gross revenue be affected?").
But if you want an answer, Random Forests don't make any assumptions about the data, and have a nice way of measuring feature importance. However of course, they are highly parametric, and will just memorize (i.e. overfit) the data if left unconstrained, so you need to do quite a bit of parameter tuning.
Another model I like is the KNN, it doesn't assume much about the data (just that it's all on the same scale, which is easy to do), and doesn't have many parameters you have to fine-tune (just "number of neighbours" really), however, it unfortunately doesn't have a good way to measure feature importance.
I did come up with an algorithm to find feature importances involving the KNN though (how it works is it goes through every feature and temporarily removes it, then, it fine-tunes a KNN to it, and measures the accuracy, and assigns the accuracy to the feature, the lower this accuracy, the more important the feature is).

However, in the end, the algorithm I would recommend if you're going with Approach 3 is the Random Forest.

Conclusion

1. "What assumptions do I need to check for approach 1?" - Make sure the bins are large enough to contain meaningful information, this also applies to Approach 2.
2. "Are there any potential flaws or gotchas in approaches 1 and 2?" - Approach 1 falls prey to confounders, Approach 2 falls prey to colliders and you need to keep in mind mediators.
3. "What approach would you prefer out of the three?" - Approach 3, because the other two approaches have fundamental flaws, but if there was an Approach 4 - "Find all of the confounders, control for them, and measure how revenue changes with respect to $$A_1$$", I would choose that instead.
4. "What ML model should be used for approach 3?" - I recommend the Random Forest.
• Thanks for the super useful answer! Approach 3 is what I've been going with as well - basically train the model on a subset, compare val performance to train performance, and if the prediction metric (MAPE in my case) is good enough, then I can rely on the feature importances output by the RF. Feb 17 at 12:10
• No problem! Happy to help :) Feb 17 at 12:17

Approaches 1 & 2

1. Noise

If we adopt a frequentist view, we can consider the attributes of a movie to have an underlying probability distribution (or, if you'd rather be Bayesian, you can choose to believe they come from a certain distribution). For a single attribute, its values are a sequence of random variables drawn from the same distribution. We can model our data for one attribute by considering a data model of the general form $$Y = X + \epsilon$$, where $$Y$$ is a random variable representing one measurement, $$X$$ is a random variable coming from the "true" underlying distribution and $$\epsilon$$ is a noise term.

For some variables, the noise term is negligible (e.g. the runtime of a movie is actually not noisy at all, it's a real, objective, fixed value). For others, noise may follow a normal distribution, or another distribution with mean 0. In that case, over many measurements, the law of large numbers dictates that the mean of your $$Y$$s will converge to the expectation of $$X$$. Here comes the first warning: if you have noisy data and few of them, your data might not mean much.

Finally, you might be so unlucky that you have noise $$\epsilon$$ coming from a distribution with a mean which is not $$0$$, in which case you are likely to have systematically biased data.

In both approaches, you partition your observations into groups/classes (in 1 based on the values of one attribute, in 2 on the values of all other attributes). This can lead to class imbalance, in which case you might end up with some classes with many observations (and presumably little noise) and some classes with few members (maybe only one!) and a possibly a lot of noise.

Furthermore, if you look at variation only, and not correlation, then you might end up with high variance (even comparatively, if one attribute is very noisy perhaps due to previously mentioned class imbalance) and low correlation, which would erroneously lead you to deduce that an attribute has good explanatory power.

2. Multicollinearity

Approach 1 opens you up to the problem of collinearity: two predictor variables might be strongly linked, in which case it's not easy to say which one is really a valuable predictor and which one can be considered just an epiphenomenon. For instance, assume fantasy as a genre is very popular, but people dislike overly long movies. Fantasy movies would then be very long and have a large revenue, but it would be wrong to conclude that the long runtime is responsible for their large revenue. Even worse: while this example used two attributes which you probably have access to, a hidden, unobserved (i.e. latent) attribute might actually be causally responsible for the phenomenon and for the correlation between the two observed attributes.

In approach 2, you're doing something akin to statistical matching, so you at least stand a better chance of avoiding multicollinearity and confounding variables.

Personal recommendation

I would personally start with some exploratory data analysis (EDA). Plot your data! Use heatmaps, small multiples, violin plots, the works, anything that might give you insight into its structure.

Next, start small, go big if needed. Try measures of "linear" correlation, first Pearson correlation. For categorical variables, you can use Spearman rank correlation or Kendall rank correlation. You can also use measures of rank correlation (e.g. those mentioned previously) to estimate non-linear correlation. You can also use SHAP values, Q-Q plots, partial plots.

You can then move on to maybe a simple decision tree (or random forest), or a linear regression if you encode your categorical variables. If that gives you good accuracy, then you can examine what your model has learned. You can use stepwise linear regression, where you either add or subtract attributes one by one and examine the change in predictive power.

You can also try dimensionality reduction, for instance do PCA or ICA then look at which attributes carry a lot of weight in the first few PCA/ICA components. You can also try other types of matrix factorization.

If accuracy is poor with simple models, you can try more complex models, such as neural networks, then use interpretability models designed for those, such as LIME, which builds a locally accurate linear surrogate of a deep network.