1
$\begingroup$

I'm trying to test different machine learning algorithm to try to find correlation between various data on MRI scans. Since I'm dealing with medical data, I don't have access to many events, but still I'm trying to see what a simple fully connected NN while provides me. By debugging, it looks like even if I enter the output almost as is in the input, a simple NN using the default scikit-learn NN regressor is unable to find a satisfactory model. I tried to reproduce the effect and I'd like to share with you the following code. I guess I'm just doing something wrong, because, basically, I'd like the NN to produce a simple model where label=X2-X1 and it doesn't work to me which is really surprising me.

here is the code:


import sklearn.neural_network as NN
import matplotlib.pyplot as plt
import numpy as np
N_events=100000 # number of rows of dataset
N_features = 2 # number of features
X = np.random.rand(N_events,N_features) # chose the data randomly
labels = X[:,1]-X[:,0] # label = X2-X1

C = NN.MLPRegressor(hidden_layer_sizes=(2,2),max_iter=500000,random_state=1) # very simple scikit-learn NN regressor
n = N_events // 2 # half data for training, half for test

print(X[0:10,1]-X[0:10,0]) # check that indeed label=X2-X1
print(labels[0:10])
C.fit( X[0:n,:] , labels[0:n]) # train the model

# plot y_predicted vs y_true for the test set... 
plt.figure()
y_real = labels[n:] # 
y_pred = C.predict(X[n:,:])
plt.plot(y_real,y_pred,'.')

# plot y_predicted vs y_true for the training set... even this doesn't work
plt.figure()
y_real = labels[0:n] # 
y_pred = C.predict(X[0:n,:])
plt.plot(y_real,y_pred,'.')

I was expecting the NN with 2x2 = 4 degrees of freedom to easily find such a simple model (note that I even don't add unused features which could disturb the fit, since N_features is 2 here), so I guess I'm just not using the code correctly. Can anyone helps me ?

Many thanks

Improve this question
$\endgroup$
9
  • $\begingroup$ Why would you expect a 2x2 to work? $\endgroup$ – gented Mar 18 '19 at 11:57
  • $\begingroup$ since the exact model is label Y is X2-X1, I mean the model have one degree of freedom, and with 2x2 hidden layers (4 free parameters) and a lot of data points (50000 here) I was expecting the NN to find it. I mean there is more free parameters than actual degree of freedom, and enough data to constrain the model right ? $\endgroup$ – david guez Mar 18 '19 at 12:17
  • $\begingroup$ sorry I meant 8 free parameters (2 input *2 cell in 1st layer = 4 , and 2 cell in first layer*2 cell in 2nd layer = 4 , so 4+4=8 weights, not counting for the bias term) $\endgroup$ – david guez Mar 18 '19 at 13:06
  • $\begingroup$ What do you mean "expecting to find it"? What is that "doesn't work"? Are you seeing a coding error (unrelated to NN) or are you saying that the NN don't perform well? $\endgroup$ – gented Mar 18 '19 at 13:06
  • $\begingroup$ no, actually the code is running, but the produced y_pred are very far from the y_true... so it performs very poorly (actually it the code predict always the same y_pred, no matter what the X is) . Indeed, by adding the number of cell in the model (even 3x3) the code produce a better (still not perfect) model, but I was expecting it to perform better. Maybe I'm just wrong, but I thought that fully connected NN would be general enough for such a simple fit...) $\endgroup$ – david guez Mar 18 '19 at 13:10
0
$\begingroup$

The problem is relu which is the default activation function. It zeros the inputs smaller than 0. Problem will be solved by using other functions. For example:

C = NN.MLPRegressor(activation="identity", hidden_layer_sizes=(1,1),max_iter=25,random_state=1)

Note that even 1 hidden layer with 1 neuron is enough to find the $X_2 = X_1$ decision boundary.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ indeed it makes sense thx! So, since the activation is mainly here to introduce non-linearities, it seems that it actually makes it harder to catch the linear behavior, am I right ? I know that sklearn doesn't allow to specify different activation function per layer, so I'll also try to check what happen with Keras and 2 layers, with a relu or tanh at first layer and a identity activation after the 2nd layer.... many thx for your answer $\endgroup$ – david guez Mar 18 '19 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.