0
$\begingroup$

I'm fairly new to neural networks, and I'm seeing some behavior during training I don't understand. I'm working with a pretty straightforward feedforward neural net classifier with 5 hidden layers, 400 neurons per layer, swish activation, cross-entropy loss. When I don't add any regularization, I get a strange semi-regular spiking in my graphs:

loss-accuracy

Its not always as regular as it appears in the pic above. Sometimes its a little more off-kilter:

enter image description here

When I add some regularization, either dropout or l2, this effect seems to abate, and go away entirely if I add enough.

The x-axis in these graphs is epochs, so it doesn't seem like it could be one bad/small batch doing something strange (I actually know the final batch in each epoch is very near full size).

I have about 350,000 training examples, and I'm setting aside about 3000 for my validation set, so it seems like I have plenty of data.

What could be going on here?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.