Correlation vs Mutual Information vs let-the-model-decide

Question

I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which features to keep in a ML model compared to correlation coefficient.

However, I've just run a simple experiment, where I've created 10 different f(x) where five of them don't have any random noise (y-values can be 100% extracted from x-values) and the other five have random noise:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.feature_selection import mutual_info_regression

# Create x-values and noise
x = np.random.uniform(-10, 10, size=500)
noise = np.random.uniform(-10, 10, size=500)

# Create several y-values
y1 = x
y2 = x + noise
y3 = 3*x
y4 = 3*x + noise
y5 = x**2
y6 = x**2 + noise
y7 = 2**x
y8 = 2**x + 10 * noise
y9 = -3 * x**3 + 9 * x**2 + 6 * x + 53
y10 = -3 * x**3 + 9 * x**2 + 6 * x + 53 + 100 * noise

# Create functions to iterate and plot
ids = [1, 6, 2, 7, 3, 8, 4, 9, 5, 10]
y_funcs = [y1, y2, y3, y4, y5, y6, y7, y8, y9, y10]
titles = ["Y1 = X", "Y2 = X + noise", "Y3 = 3X", "Y4 = 3X + noise", "Y5 = X^2", "Y6 = X^2 + noise",
          "Y7 = 2^X", "Y8 = 2^X + 10*noise", "Y9 = -3X^3 + ... + 53", "Y10 = -3X^3 + ... + 100*noise"]

# Create a grid plot
plt.figure(figsize=(15,6))
plt.tight_layout()
for i, y_vals, title in zip(ids, y_funcs, titles):
    plt.subplot(2, 5, i)
    plt.scatter(x, y_vals, alpha=0.25)
    plt.title(title)
    plt.xticks([])
    plt.yticks([])
plt.show()

Then, I've calculated the correlation coefficients and MI scores between x-feature and each of these y-features. To facilitate comparison between these metrics, I've divided MI scores by 5, so we understand the relative power of this metric compared to correlation coefficient:

# Generate correlation coefficients
### Generate the dataframe
data = np.transpose(np.array([x, y1, y2, y3, y4, y5, y6, y7, y8, y9, y10]))
df = pd.DataFrame(data, columns = ["x", "y1", "y2", "y3", "y4", "y5", "y6", "y7", "y8", "y9", "y10"])

### Define correlation matrix (method = Pearson: standard correlation coefficient)
corr_matrix = df.corr(method="pearson")["x"]

# Generate MI scores
## Get the necessary lists
y_funcs = [y1, y2, y3, y4, y5, y6, y7, y8, y9, y10]
cols = ["y1", "y2", "y3", "y4", "y5", "y6", "y7", "y8", "y9", "y10"]

### Get Mutual Information score for each variable
mi_scores = []
for y_vals, col in zip(y_funcs, cols):
    x_t = np.reshape(x, (-1,1))
    mi_score = mutual_info_regression(x_t, y_vals)
    mi_scores.append(mi_score[0])
mi_scores = np.array(mi_scores)

# Plot the results
x_vals = np.arange(1, 11, 1)

plt.figure(figsize=(15,4))
plt.tight_layout()
ax = plt.subplot(111)

ax.bar(x=x_vals-.15, height=abs(corr_matrix[1:]), color="firebrick", width=.3, label="Correlations")
ax.bar(x=x_vals+.15, height=mi_scores/5, color="dodgerblue", width=.3, label="MI scores / 5")

plt.xticks(ticks=x_vals, labels=cols)
for location in ["top", "right"]:
    ax.spines[location].set_visible(False)

plt.legend()
plt.show()

As we can see on this plot, correlation coefficient does better on several ocassions, while MI score is better when we have a quadratic function, for example (see Y5 and Y6).

Therefore, my question would be: when doing a ML project (so the relations are not so clear, and features can even interact), how should we decide to use each of these metrics to remove a feature (i.e., one that seems to not be correlated with target feature Y)? Even more, should we do any removal at all, instead of letting the ML model decide better?

Happy coding!

Answer 1

This is a typical Feature Selection problem in the world of machine learning. The following are a few fundamental points that can be used before plugging any data into machine learning models.

1. Perform a feature correlation analysis before diving into ML.
2. Correlated features do not provide any additional information for ML models.
3. You just need to have one feature among the correlated ones basically because the other one is redundant.
4. Once those straight correlations are removed, then you can employ more complex feature selection methods that make use of techniques like mutual information.
5. What I would suggest is to use a combination of feature selection methods with a voting strategy to find the best features.
6. With your simple experiment itself you have proved that mutual information does not always win. There are methods like Correlation-based feature selection, Relief-F etc.

In short first check, simple correlations and then use more sophisticated methods. Additionally, mutual information may not always be useful. Rather there is no single method that will always work for all different datasets.

Answer 2

This is an interesting experiment.

A few thoughts:

• Mutual Information (MI) is based on entropy and measures the strength of the statistical association between two variables. I think that conditional entropy is a more common option to select features: I assume this is because it represents how much information is gained about the target from knowing the feature, as opposed to MI which is undirected (i.e. we don't know whether this features helps knowing the target more or the converse).
• Individual feature selection is always an approximation anyway, because most models are able to use the information carried by multiple features together. So eliminating a feature based on its individual contribution might be a mistake, since the feature alone might be useless but in conjunction with other features it might be useful. I interpret this in two ways:
  • Simple correlation is a decent robust choice: since it's a "rough" process anyway, one doesn't need a lot of precision most of the time.
  • If one needs sophisticated feature selection, it's worth using group/subset feature selection. It's a lot more computations but it gives much better results. Genetic feature selection can be a really good method for this.

(disclaimer: I'm not especially skilled about this topic)