How to make sure that the features learned by a neural network are not correlated?

Question

Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the features from the last hidden layer are combined for classification or regression.

How to make sure that the features learned by a neural network are not correlated?

I think that I can force a neural network to learn correlated features in the following way. Remove the output layer, so that the last hidden layer produces $n$ outputs. Then train this network on $n$ target variables that are not independent but are linear combinations of say two or three independent targets (maybe with some noise added to avoid true linear dependence). As neural networks are universal approximators, it should be possible to train such a network. However, in that case, all neurons in the final layer will be redundant except those two or three neurons that correspond to the linearly independent targets.

This redundancy can be present in each layer. How to avoid it? Of course, I can check the correlation of neuron outputs after the training, remove the correlated neurons, and retrain the network but it looks inefficient. How to make the network learn uncorrelated features from the beginning?

What comes to my mind is using different and independent (e.g. orthogonal) activation functions for each neuron within the layer. Are there such networks? Usually, activation functions are taken to be the same.

EDIT: I tried to test if dropout can make learned features less correlated. I did not find that it does (or I am doing it in a wrong way?).

This is a toy problem that I tried: $y = a^2+b^2$ where $a$ and $b$ are uniformly distributed between 0 and 1.

a_train = np.random.uniform(low=0.0, high=1.0, size=10000)
b_train = np.random.uniform(low=0.0, high=1.0, size=10000)
y_train = a_train**2 + b_train**2
X_train = np.hstack([a_train[:,np.newaxis], b_train[:,np.newaxis]])

The input features are not correlated.

np.corrcoef(X_train[:,0], X_train[:,1])[0,1]

0.002136928470247475

Neural network:

model = keras.Sequential(
    [
        layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
        layers.Dense(16, activation="tanh", name="layer2"),
        layers.Dense(2, activation="tanh", name="feature_layer"),
        layers.Dense(1, name="out_layer"),
    ]
)
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)

model.compile(loss='mean_squared_error',
                optimizer=optimizer,
                metrics=['mean_squared_error'])

model.fit(X_train, y_train, epochs=100, batch_size=128, validation_split = 0.2, verbose=1)

feature_extractor = keras.Model(
    inputs=model.inputs,
    outputs=model.get_layer(name="feature_layer").output,
)

features=feature_extractor(X_train).numpy()

The correlation between the outputs of the next-to-the-last layer:

np.corrcoef(features[:,0], features[:,1])[1,0]

-0.7527464552820806

After adding dropout,

model = keras.Sequential(
    [
        layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
        layers.Dropout(0.2),
        layers.Dense(16, activation="tanh", name="layer2"),
        layers.Dropout(0.2),
        layers.Dense(2, activation="tanh", name="feature_layer"),
        layers.Dense(1, name="out_layer"),
    ]
)
model.summary()

The correlation becomes

-0.8645152290698691

Answer 1

Dropout (2014 paper) is my first thought.

By effectively removing N% of your neurons on each pass through the data, you make it harder for any two neurons to work together. When its buddy disappears it is forced to find another way to learn the patterns in the data. On the next epoch, when its old buddy comes back, it has a new perspective on life, its horizons have been broadened, and it is less likely to re-enter that co-dependent relationship.

The second thought was L2 regularization: it forces the network to not rely on just one neuron in a layer, but to get them all involved.

If you are not convinced, just try it. Try to create a network where, say, only two of the neurons in the final layer carry all the signal. Then re-train it using a dropout of 50%, or a higher L2 value. It may be a weaker model overall, but it is going to be really hard to not have all the neurons share the load. (Also try it with and without the bias, as mentioned below.)

different and independent activation functions for each neuron within the layer. Are there such networks?

I'd be interested to hear if you find anything. I suppose you could argue that the bias is creating a unique activation function on each neuron, though I think you are suggesting something like ReLU in some neurons and tanh in others? (A bit tricky to implement efficiently?)

A little bit removed from your question, but the different heads in multi-head attention (in the Transformer model) are another example of getting multiple different "activation functions" into a layer.