1
$\begingroup$

I'm training a logistic regression model on a small dataset. I have about 1300 samples that I split into a training and a testing set (70% and 30% respectively).

The training seems ok, however when I plot the accuracy of my model w.r.t. the epoch, some repeating noisy patterns appear at the end well after the accuracy is stabilised (after 800 epochs, see images below).

The training is done with an Adam optimizer. I'm using a learning rate of 0.04 and a weight decay of 0.07 that I found after doing a random search.

Is it something that may happen when training and is without consequence or does it reflect an issue with my data / training / software implementation ?

enter image description here

enter image description here

$\endgroup$
2
$\begingroup$

It is likely that in your training phase you are reaching local minima, then since you 'persist' at each epoch you get off of your point and then after the 800 epochs you reach it again.

Look at the image below, imagine you reach the blue point, but you keep persisting, looking for another point, then you will be off of your track but it will eventually come back. The pattern will be more clear in logistic regression since theoretically you are applying gradient descent to a simpler function than a complex neural network.

enter image description here

$\endgroup$
  • $\begingroup$ Thank you for your answer Victor, this makes sense. Would you recommend stopping the training at around 600-700 epochs, when the accuracy is stabilized ? $\endgroup$ – Elie Génard Sep 12 at 13:00
  • $\begingroup$ Yes. You can also adjust early_stopping criteria, tol, depending on the library you are using, basically it will stop after no improvement in loss criteria, or a really small one. (Indication of minima) $\endgroup$ – Victor Oliveira Sep 12 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.