# Unable to overfit using MLP

I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:

• Network Architecture: [67 40 5]
• Learning Rate: exponential decay of base learning rate
• Validation Accuracy: ~45%
• Observation: Both training accuracy and validation accuracy stops improving.

Exp 2: Repeated Exp 1 with batchnorm layer

• Validation Accuracy: ~50%
• Observation: Got 5% increase in accuracy.

Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network

• Network Architecture: [67 40 40 40 40 40 40 5]
• Learning Rate: exponential decay of base learning rate
• Validation Accuracy: ~55%

Thoughts on what might be happening?

Some things that could be happening:

1. You do not have enough parameters in your hidden Layer to learn. Try something like [67, 512, 5] or expand deeper with something like [67, 1024, 256, 5]. The idea is that your hidden layer may be too small to capture interactions between the attributes given the amount of training data you have.

A rule of thumb that I have seen is:

Nh=Ns/(α∗(Ni+No))

Nh = number of hidden neurons.
Ni = number of input neurons.
No = number of output neurons.
Ns = number of samples in training data set.
α = an arbitrary scaling factor usually 2-10.


Which in your case gets you to like 5000 neurons, but since you are oversampling your classes, I would also suggest under-sampling instead, and that gets you maybe to around 1024, which is fairly standard. The [67, 512, 5] configuration is pure gut reaction.

1. Learning rate is too high. Try starting with lower LR (like 10e-5). Sometimes this works with Adam.

2. Weight Initialization : double check you are using Xavier.