4
$\begingroup$

Accuracy vs Epochs

The orange curve is train accuracy and blue is validation accuracy. Is this clear overfitting or should I let it run for more epochs?

With custom dataset (1D data with 70 features) I trained a 2 layer MLP. Network Architecture: [70-200-200-4]. I'm only able to reach ~50+% accuracy. Any suggestions on what steps I can take to improve accuracy? (Obtaining more data isn't an option)

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ Why do you think its overfiting? $\endgroup$ – Aditya May 12 '18 at 5:29
  • $\begingroup$ Because the train accuracy continues to increase whereas the validation accuracy looks to have saturated $\endgroup$ – Ashwin Kannan May 12 '18 at 11:59
1
$\begingroup$

When training accuracy increases while validation accuracy remains constant or decreases, then the model is most likely overfitting, or it may be saturated.

You can try the following methods to increase accuracy:

$\endgroup$
2
  • $\begingroup$ Thanks for your suggestions. I'm trying out SMOTE now. What would you suggest for improving train accuracy? It's not improving beyond 55-56%. Thanks! $\endgroup$ – Ashwin Kannan May 14 '18 at 14:48
  • $\begingroup$ The best thing you could do is to get more data. However, if this is not an option, I would start with the easiest improvements first. In order of difficulty to implement: early stopping, momentum, weight decay, dropout, ensembles, synthetic data. It might also depend on your dataset. Can you provide more details about the specific task and requirements? $\endgroup$ – Benji Albert May 14 '18 at 14:58
0
$\begingroup$

There are a few approaches you might take to establish how and whether accuracy can be improved. Some options sketched out here in brief are to compare network results to a baseline estimator, diagnose misclassifications, apply dimensionality reduction, and network architecture troubleshooting.

Without knowing much about the data, I think you could try to establish whether it is even possible to increase the accuracy of your network by comparing network performance with a baseline estimator like a decision tree or support vector machine. The bonus of using a decision tree is that an algorithm such as ID3 uses information gain to make the splits and this might give you some intuition about your data and whether it's possible to improve accuracy before sinking time into it. If you haven't already, you could do some exploratory analysis to understand how noisy the data/classes are.

It may also be useful to take a closer look at the errors your network is making. A confusion matrix or classification report can help you to diagnose whether your network is struggling with (for example) one particular class or a specific area of the decision boundary. If your classes are unbalanced, you could look at re-weighting your training examples and approaches such as SMOTE etc that Benji mentions.

With 70 features, the network may benefit from some dimensionality reduction. If you apply PCA to the data you can establish how many components you need to explain a reasonable amount of variance in the data, or if it is more complicated. It is possible that your network may respond better if taking the transformed data as input rather than the raw data (on that topic, have you pre-processed your data?).

Finally, a broader architecture search might yield improved results - I'm assuming you may have done this already, but have you tried increasing the units in each layer or cutting out that second hidden layer? You might also try using different activation functions or a different learning rate schedule. Bengio's 'Practical recommendations for gradient-based training of deep architectures' is worth a skim (the article is aimed mainly at deeper networks, but elements of sections 3.1, 3.2 and 4 are relevant).

This answer has only touched briefly on each areas but I'm hoping there are some solid leads you can chase up.

$\endgroup$
0
$\begingroup$

It is hard to determine if this is in fact overfitting because your validation accuracy has yet to start decreasing. Overfitting is characterized by a set of model parameters which perform very well on the training set but do not generalize to different subsets of the data.

I assume you are using Keras based on these plots. To avoid overfitting when training you can save the weights only when the model gets better on your validation set and ignore those which do no better or worse. This will ensure that at the end of the training session you have the weights which generalized the best to the validation set saved to file which can then be loaded into the model.

Saving only the best set of weights in Keras

Use a callback which will run between epochs

name = 'my_model'

if not os.path.exists('weights//'+name):
    os.makedirs('weights//'+name)

# Save the weights using a checkpoint.
filepath='weights//' + name + '//weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5'
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 100
# Fit the model weights.
history = model.fit_generator(generator=training_generator,
                              steps_per_epoch=training_generator.num_batches,
                              epochs=epochs,
                              verbose=1,
                              callbacks=callbacks_list,
                              validation_data=validation_generator,
                              validation_steps=validation_generator.num_batches)

Loading your best set of weights

Then when the training is done you can retrieve the best set of weights using

weight_filename = '...'
model.load_weights('weights//' + name + weight_filename)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.