Is this clear overfitting?

Question

The orange curve is train accuracy and blue is validation accuracy. Is this clear overfitting or should I let it run for more epochs?

With custom dataset (1D data with 70 features) I trained a 2 layer MLP. Network Architecture: [70-200-200-4]. I'm only able to reach ~50+% accuracy. Any suggestions on what steps I can take to improve accuracy? (Obtaining more data isn't an option)

Thanks in advance!

Answer 1

When training accuracy increases while validation accuracy remains constant or decreases, then the model is most likely overfitting, or it may be saturated.

You can try the following methods to increase accuracy:

• Momentum
• Regularization techniques: Early Stopping, Dropout, Weight Decay, etc
• Synthetic Data and related techniques such as SMOTE
• Ensembles

Answer 2

There are a few approaches you might take to establish how and whether accuracy can be improved. Some options sketched out here in brief are to compare network results to a baseline estimator, diagnose misclassifications, apply dimensionality reduction, and network architecture troubleshooting.

Without knowing much about the data, I think you could try to establish whether it is even possible to increase the accuracy of your network by comparing network performance with a baseline estimator like a decision tree or support vector machine. The bonus of using a decision tree is that an algorithm such as ID3 uses information gain to make the splits and this might give you some intuition about your data and whether it's possible to improve accuracy before sinking time into it. If you haven't already, you could do some exploratory analysis to understand how noisy the data/classes are.

It may also be useful to take a closer look at the errors your network is making. A confusion matrix or classification report can help you to diagnose whether your network is struggling with (for example) one particular class or a specific area of the decision boundary. If your classes are unbalanced, you could look at re-weighting your training examples and approaches such as SMOTE etc that Benji mentions.

With 70 features, the network may benefit from some dimensionality reduction. If you apply PCA to the data you can establish how many components you need to explain a reasonable amount of variance in the data, or if it is more complicated. It is possible that your network may respond better if taking the transformed data as input rather than the raw data (on that topic, have you pre-processed your data?).

Finally, a broader architecture search might yield improved results - I'm assuming you may have done this already, but have you tried increasing the units in each layer or cutting out that second hidden layer? You might also try using different activation functions or a different learning rate schedule. Bengio's 'Practical recommendations for gradient-based training of deep architectures' is worth a skim (the article is aimed mainly at deeper networks, but elements of sections 3.1, 3.2 and 4 are relevant).

This answer has only touched briefly on each areas but I'm hoping there are some solid leads you can chase up.

Answer 3

It is hard to determine if this is in fact overfitting because your validation accuracy has yet to start decreasing. Overfitting is characterized by a set of model parameters which perform very well on the training set but do not generalize to different subsets of the data.

I assume you are using Keras based on these plots. To avoid overfitting when training you can save the weights only when the model gets better on your validation set and ignore those which do no better or worse. This will ensure that at the end of the training session you have the weights which generalized the best to the validation set saved to file which can then be loaded into the model.

Saving only the best set of weights in Keras

Use a callback which will run between epochs

name = 'my_model'

if not os.path.exists('weights//'+name):
    os.makedirs('weights//'+name)

# Save the weights using a checkpoint.
filepath='weights//' + name + '//weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5'
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 100
# Fit the model weights.
history = model.fit_generator(generator=training_generator,
                              steps_per_epoch=training_generator.num_batches,
                              epochs=epochs,
                              verbose=1,
                              callbacks=callbacks_list,
                              validation_data=validation_generator,
                              validation_steps=validation_generator.num_batches)

Loading your best set of weights

Then when the training is done you can retrieve the best set of weights using

weight_filename = '...'
model.load_weights('weights//' + name + weight_filename)