I have an input array X, which is of the shape (38000,32,1); the output array Y is of (38000,1), the element of Y can be 0 or an numerical value, such as 0.040139 or 1.075341, or some other numerical values, etc. A typical vector in X looks like as follows

enter image description here

I would like to build an model to predict Y based on X. This is the model, two convolutional layer, followed by max-pooling, then one convolutional layer, one pooling, then a fully connected layer. The detailed architecture is as follows.

outX = conv_filter(20, 50, strides=1, activation='relu')(inputs)
outX = conv_filter(5, 60, activation='relu')(outX)
 outX = MaxPooling1D(2)(outX)
outX = conv_filter(2, 70, activation='relu')(outX)
outX = MaxPooling1D(2)(outX)

outX = Dense(300, activation='relu')(outX)
outX = Flatten()(outX)
predictions = Dense(1,activation='linear')(outX)
model = Model(inputs=[inputs],outputs=predictions)

The loss function is 'mean_squared_error'.The result does not look very good. One of the reason I can think is about the numerical values for the input vector, which tends to be very small. Can it cause any numerical issue? Are there any suggestions to pre-process them? Besides, are there any improvement can be made on the network architecture? Any suggestions would be highly appreciated.

  • $\begingroup$ Your jump from 300 to 1 might be a small issue as 300 nos will produce a single no $\endgroup$ – Aditya Mar 17 '18 at 23:54
  • $\begingroup$ Also if you are using Telus in the dense layers, then the -ve weights will be killed $\endgroup$ – Aditya Mar 18 '18 at 2:57
  • $\begingroup$ Hi Aditya, could you elaborate why the 300 number of neurons on the first fully connected layer will cause the problem? Besides, what do you mean the "Telus" and "-ve" in the last comment? $\endgroup$ – user297850 Mar 18 '18 at 15:04
  • $\begingroup$ Sorry that was a typo, it's ReLU and the jump from 300 neurons in the dense layer directly to 1 single neuron that's what I meant $\endgroup$ – Aditya Mar 18 '18 at 16:32
  • $\begingroup$ Yes, I am using ReLU, what are the alternatives of activation function based on this scenario? By the way, I am not very clear why you think jumping from 300 neurons to 1 will cause the problem. Any recommendations? $\endgroup$ – user297850 Mar 19 '18 at 16:26

If you haven't figured out what's going on already then you could try a couple of things.

1) Since you mentioned that your input values are very small, you could try normalizing them. I agree that very small (in your case, really small) values might mess up the weight updation. When you do, make sure you normalize first and then do training-testing split

2) Can you tell me how your loss looks during training? Do you see overfitting? If you do, did you use dropout? Did you try L2 regularization on the weights? (only use them on weights, not on biases)

There could be a million things going on and a lot of room for improvement may be there in the network, but I need to have a look at how your loss decreases as the training proceeds. Also, you need to elaborate on how bad your results look? (I assume that your training is quite successful but your testing error is high?). Also , if you're not bound to use CNN, did you try out with an ANN with a few hidden layers and see how that model does?

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.