I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in the training set. The dataset is structured, ie, not images. My original features were highly right skewed; I do my best to transform these into nicer distributions by either taking logs or categorising some of them. I use an architecture of 512->128->128->1, with relu activations in every layer except the last. My loss function is sigmoid cross entropy.

The validation set contains 10 million rows. Initially the validation error goes down, but then starts to go up again after a couple of epochs. On analysing the gradients and weights of each layer, I see that the overfitting coincides with the weights on the final layer only getting larger and larger. The final layer seems to go into overdrive while the rest of the network seems to do very little learning.

I can solve the overfitting problem by using l2 regularisation, but this hurts the validation error. I have yet to find a beta regularisation parameter which doesn't hurt the best validation error I've seen. Dropout makes things even worse.

Granted, the classification problem is very difficult, with probably a very weak signal, but I find that gradient boosted trees are able to generalize much better than a simple, say, 64x64 multi-layer perceptron (the log loss on the training set is the same for both network and gradient boosted tree).

Are there any words of wisdom on how to make this network generalize better given that I've already tried:

  • dropout of varying degrees
  • l1/l2/group lasso regularization
  • adding noise to inputs
  • adding noise to gradients and weights
  • feature-engineering so as to remove/re-represent highly skewed features
  • batch normalization
  • using a lower learning rate on the final layer
  • simply using a smaller network (this is the best solution I've found)

to some or all layers. All methods hurt the validation error so much that the performance is nowhere near how well the tree model does. I would have given up by now were it not for the fact that the tree model is able to do so much better out of sample, but the training log loss for both is the same.

  • $\begingroup$ Why do you use a 2 layer network? seems a bit comlicated for binary classification. Try using a 1 hidden layer and see the results. Do you have low amount of training data compared to validation? $\endgroup$
    – AR_
    Oct 23, 2017 at 9:06
  • $\begingroup$ Using 2 layers gives me slightly better training loss and validation error than using 1 layer. I have 30 million rows in the training set, 500 features. Validation set has 9 million rows. $\endgroup$ Oct 23, 2017 at 9:12
  • 1
    $\begingroup$ It's impossible to answer without knowing (i) your loss function and (ii) your evaluation metric. Often in these imbalance problems, people use loss functions that are unsuited for the metric they want to maximize. $\endgroup$ Oct 27, 2017 at 6:22
  • $\begingroup$ I have updated the question with more details. Both tree and neural network use sigmoid cross entropy. $\endgroup$ Oct 27, 2017 at 7:09
  • 1
    $\begingroup$ are you using sigmoid as the last layer activation with one output node along with Cross entropy OR softmax with 2 output nodes and cross entropy. Just making sure. $\endgroup$ Nov 4, 2017 at 5:23

1 Answer 1


Are you balancing your training dataset 50/50? It seems to me that you are making a lot of changes to account for 80/20 balance when it sounds like you have a deep enough dataset to balance your training to be 50/50 and still have a reasonable size for your training set. How about keeping it to a smaller network - as you and some notes above suggest - and with a 50/50 set on a smaller training sample?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.