I'm fairly new to neural networks, and I'm seeing some behavior during training I don't understand. I'm working with a pretty straightforward feedforward neural net classifier with 5 hidden layers, 400 neurons per layer, swish activation, cross-entropy loss. When I don't add any regularization, I get a strange semi-regular spiking in my graphs:


Its not always as regular as it appears in the pic above. Sometimes its a little more off-kilter:

enter image description here

When I add some regularization, either dropout or l2, this effect seems to abate, and go away entirely if I add enough.

The x-axis in these graphs is epochs, so it doesn't seem like it could be one bad/small batch doing something strange (I actually know the final batch in each epoch is very near full size).

I have about 350,000 training examples, and I'm setting aside about 3000 for my validation set, so it seems like I have plenty of data.

What could be going on here?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.