# How to make sure that the features learned by a neural network are not correlated?

Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the features from the last hidden layer are combined for classification or regression.

How to make sure that the features learned by a neural network are not correlated?

I think that I can force a neural network to learn correlated features in the following way. Remove the output layer, so that the last hidden layer produces $$n$$ outputs. Then train this network on $$n$$ target variables that are not independent but are linear combinations of say two or three independent targets (maybe with some noise added to avoid true linear dependence). As neural networks are universal approximators, it should be possible to train such a network. However, in that case, all neurons in the final layer will be redundant except those two or three neurons that correspond to the linearly independent targets.

This redundancy can be present in each layer. How to avoid it? Of course, I can check the correlation of neuron outputs after the training, remove the correlated neurons, and retrain the network but it looks inefficient. How to make the network learn uncorrelated features from the beginning?

What comes to my mind is using different and independent (e.g. orthogonal) activation functions for each neuron within the layer. Are there such networks? Usually, activation functions are taken to be the same.

EDIT: I tried to test if dropout can make learned features less correlated. I did not find that it does (or I am doing it in a wrong way?).

This is a toy problem that I tried: $$y = a^2+b^2$$ where $$a$$ and $$b$$ are uniformly distributed between 0 and 1.

a_train = np.random.uniform(low=0.0, high=1.0, size=10000)
b_train = np.random.uniform(low=0.0, high=1.0, size=10000)
y_train = a_train**2 + b_train**2
X_train = np.hstack([a_train[:,np.newaxis], b_train[:,np.newaxis]])


The input features are not correlated.

np.corrcoef(X_train[:,0], X_train[:,1])[0,1]

0.002136928470247475


Neural network:

model = keras.Sequential(
[
layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
layers.Dense(16, activation="tanh", name="layer2"),
layers.Dense(2, activation="tanh", name="feature_layer"),
layers.Dense(1, name="out_layer"),
]
)
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)

model.compile(loss='mean_squared_error',
optimizer=optimizer,
metrics=['mean_squared_error'])

model.fit(X_train, y_train, epochs=100, batch_size=128, validation_split = 0.2, verbose=1)

feature_extractor = keras.Model(
inputs=model.inputs,
outputs=model.get_layer(name="feature_layer").output,
)

features=feature_extractor(X_train).numpy()


The correlation between the outputs of the next-to-the-last layer:

np.corrcoef(features[:,0], features[:,1])[1,0]

-0.7527464552820806


model = keras.Sequential(
[
layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
layers.Dropout(0.2),
layers.Dense(16, activation="tanh", name="layer2"),
layers.Dropout(0.2),
layers.Dense(2, activation="tanh", name="feature_layer"),
layers.Dense(1, name="out_layer"),
]
)
model.summary()


The correlation becomes

-0.8645152290698691

• What is your goal in doing away with correlated features?
– Dave
Feb 17, 2021 at 15:03
• @Dave Sorry, I did not see your comment for some reason. I am thinking how to make networks smaller. Mar 8, 2021 at 11:02

Dropout (2014 paper) is my first thought.

By effectively removing N% of your neurons on each pass through the data, you make it harder for any two neurons to work together. When its buddy disappears it is forced to find another way to learn the patterns in the data. On the next epoch, when its old buddy comes back, it has a new perspective on life, its horizons have been broadened, and it is less likely to re-enter that co-dependent relationship.

The second thought was L2 regularization: it forces the network to not rely on just one neuron in a layer, but to get them all involved.

If you are not convinced, just try it. Try to create a network where, say, only two of the neurons in the final layer carry all the signal. Then re-train it using a dropout of 50%, or a higher L2 value. It may be a weaker model overall, but it is going to be really hard to not have all the neurons share the load. (Also try it with and without the bias, as mentioned below.)

different and independent activation functions for each neuron within the layer. Are there such networks?

I'd be interested to hear if you find anything. I suppose you could argue that the bias is creating a unique activation function on each neuron, though I think you are suggesting something like ReLU in some neurons and tanh in others? (A bit tricky to implement efficiently?)

A little bit removed from your question, but the different heads in multi-head attention (in the Transformer model) are another example of getting multiple different "activation functions" into a layer.

• 1) How dropout and other regularizations make the learned features uncorrelated? As far as I understand, these techniques force the net to learn simpler, more robust features which reduces over-fitting. OK, add dropout, L2, the features became less wiggly, the net learns less noise and more useful patterns in the data but I cannot see how these regularized features are guaranteed to be uncorrelated. 2) Different activations: Yes tricky to do it in the languages that require vectorization (Python, R, Matlab) but no problem for those that don't (C/C++, Java, Fortran, Julia) as you can use loops. Feb 17, 2021 at 8:44
• @VladislavGladkikh I updated my question with more explanation of what dropout does. Hope that helps? Feb 17, 2021 at 9:13
• @VladislavGladkikh Pedantic, but Python and R allow you write a neural net with loops, just as you can write a vector version in C++ (CUDA is itself a C++ library). Feb 17, 2021 at 9:19
• 1) I'll try to experiment as you suggested in you update. 2) I was just going to say that loops in those languages are fast, so no need to deal with a layer/weight matrix as a whole -- easier to play neuron by neuron, but this may be subjective. Feb 17, 2021 at 9:45
• I tried dropout. I did not find that it helps. I edited my question adding an experiment with dropout. Mar 8, 2021 at 10:59