# neural network to find a very simple linear model (scikit-learn)

I'm trying to test different machine learning algorithm to try to find correlation between various data on MRI scans. Since I'm dealing with medical data, I don't have access to many events, but still I'm trying to see what a simple fully connected NN while provides me. By debugging, it looks like even if I enter the output almost as is in the input, a simple NN using the default scikit-learn NN regressor is unable to find a satisfactory model. I tried to reproduce the effect and I'd like to share with you the following code. I guess I'm just doing something wrong, because, basically, I'd like the NN to produce a simple model where label=X2-X1 and it doesn't work to me which is really surprising me.

here is the code:


import sklearn.neural_network as NN
import matplotlib.pyplot as plt
import numpy as np
N_events=100000 # number of rows of dataset
N_features = 2 # number of features
X = np.random.rand(N_events,N_features) # chose the data randomly
labels = X[:,1]-X[:,0] # label = X2-X1

C = NN.MLPRegressor(hidden_layer_sizes=(2,2),max_iter=500000,random_state=1) # very simple scikit-learn NN regressor
n = N_events // 2 # half data for training, half for test

print(X[0:10,1]-X[0:10,0]) # check that indeed label=X2-X1
print(labels[0:10])
C.fit( X[0:n,:] , labels[0:n]) # train the model

# plot y_predicted vs y_true for the test set...
plt.figure()
y_real = labels[n:] #
y_pred = C.predict(X[n:,:])
plt.plot(y_real,y_pred,'.')

# plot y_predicted vs y_true for the training set... even this doesn't work
plt.figure()
y_real = labels[0:n] #
y_pred = C.predict(X[0:n,:])
plt.plot(y_real,y_pred,'.')



I was expecting the NN with 2x2 = 4 degrees of freedom to easily find such a simple model (note that I even don't add unused features which could disturb the fit, since N_features is 2 here), so I guess I'm just not using the code correctly. Can anyone helps me ?

Many thanks

• Why would you expect a 2x2 to work? – gented Mar 18 '19 at 11:57
• since the exact model is label Y is X2-X1, I mean the model have one degree of freedom, and with 2x2 hidden layers (4 free parameters) and a lot of data points (50000 here) I was expecting the NN to find it. I mean there is more free parameters than actual degree of freedom, and enough data to constrain the model right ? – david guez Mar 18 '19 at 12:17
• sorry I meant 8 free parameters (2 input *2 cell in 1st layer = 4 , and 2 cell in first layer*2 cell in 2nd layer = 4 , so 4+4=8 weights, not counting for the bias term) – david guez Mar 18 '19 at 13:06
• What do you mean "expecting to find it"? What is that "doesn't work"? Are you seeing a coding error (unrelated to NN) or are you saying that the NN don't perform well? – gented Mar 18 '19 at 13:06
• no, actually the code is running, but the produced y_pred are very far from the y_true... so it performs very poorly (actually it the code predict always the same y_pred, no matter what the X is) . Indeed, by adding the number of cell in the model (even 3x3) the code produce a better (still not perfect) model, but I was expecting it to perform better. Maybe I'm just wrong, but I thought that fully connected NN would be general enough for such a simple fit...) – david guez Mar 18 '19 at 13:10

The problem is relu which is the default activation function. It zeros the inputs smaller than 0. Problem will be solved by using other functions. For example:
C = NN.MLPRegressor(activation="identity", hidden_layer_sizes=(1,1),max_iter=25,random_state=1)

Note that even 1 hidden layer with 1 neuron is enough to find the $$X_2 = X_1$$ decision boundary.