2
$\begingroup$

I noticed something when looking at the output of the verbose. When I train my model, in the early part of the epoch (first 20 %), the loss is decreasing a lot. And then in the rest of the epoch (last 80%), the loss is very stable and doesn't change that much until the next epoch. It does the same thing.

I build a model that is training a kind of large dataset (60000 entries). I am using Keras and Tensorflow and my model is just a simple regression model with conv2d and dense layers. I am trying to get the best loss function possible. The loss function that I am using is just a simple Mean Squared Error (MSE). I am also using Adam optimizer.

I don't understand how the loss is decreasing a lot at the beginning of the epoch and less for the rest of the training. Should I reduced the size of my database ? Is it considered overtraining then ?

Is there a way to speed up this process (like early stopping but just for the epoch) ? As I see it most of the training is done for the first part of the data. Maybe I am wrong.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Have you already plotted the training and validation loss over multiple epochs? What you would expect is something that looks like an exponential decay.

Chances are your models learns the mean of the target variable in the first few batches, which reduces the MSE-loss already considerably and only later learns the subtle differences in your data. You can try to use different batch sizes as well.

Are you generally happy with the performance on your model? Does it perform well on the test/validation set? If this is the case there might be no reason to be worried about.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes I plotted both of them and I have a nice decreasing exponential plot. I will test different batch_size and play with the steps_per_epoch parameter. But thanks for your input, I was maybe looking to increase the training speed. $\endgroup$
    – Mathieu Châteauvert
    May 19, 2020 at 15:38
1
$\begingroup$

From your description, it seems that you are not shuffling your training data.

You should shuffle your data, and do it differently at every epoch. Once the data is shuffled, you should not see the behavior you describe.

Improve this answer
$\endgroup$
1
  • $\begingroup$ By looking at the documentation of the fit method link, it looks like Shuffle is default to True. My dataset is a numpy arrray and I am not using steps_per_epoch is None. Thanks $\endgroup$
    – Mathieu Châteauvert
    May 19, 2020 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.