I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:

  • Network Architecture: [67 40 5]
  • Optimizer: Adam
  • Learning Rate: exponential decay of base learning rate
  • Validation Accuracy: ~45%
  • Observation: Both training accuracy and validation accuracy stops improving.

Exp 2: Repeated Exp 1 with batchnorm layer

  • Validation Accuracy: ~50%
  • Observation: Got 5% increase in accuracy.

Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network

  • Network Architecture: [67 40 40 40 40 40 40 5]
  • Optimizer: Adam
  • Learning Rate: exponential decay of base learning rate
  • Validation Accuracy: ~55%

Thoughts on what might be happening?


Some things that could be happening:

  1. You do not have enough parameters in your hidden Layer to learn. Try something like [67, 512, 5] or expand deeper with something like [67, 1024, 256, 5]. The idea is that your hidden layer may be too small to capture interactions between the attributes given the amount of training data you have.

A rule of thumb that I have seen is:


Nh = number of hidden neurons.
Ni = number of input neurons.
No = number of output neurons.
Ns = number of samples in training data set.
α = an arbitrary scaling factor usually 2-10.

Which in your case gets you to like 5000 neurons, but since you are oversampling your classes, I would also suggest under-sampling instead, and that gets you maybe to around 1024, which is fairly standard. The [67, 512, 5] configuration is pure gut reaction.

  1. Learning rate is too high. Try starting with lower LR (like 10e-5). Sometimes this works with Adam.

  2. Weight Initialization : double check you are using Xavier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.