I'm fairly new to neural networks, and I'm seeing some behavior during training I don't understand. I'm working with a pretty straightforward feedforward neural net classifier with 5 hidden layers, 400 neurons per layer, swish activation, cross-entropy loss. When I don't add any regularization, I get a strange semi-regular spiking in my graphs:
Its not always as regular as it appears in the pic above. Sometimes its a little more off-kilter:
When I add some regularization, either dropout or l2, this effect seems to abate, and go away entirely if I add enough.
The x-axis in these graphs is epochs, so it doesn't seem like it could be one bad/small batch doing something strange (I actually know the final batch in each epoch is very near full size).
I have about 350,000 training examples, and I'm setting aside about 3000 for my validation set, so it seems like I have plenty of data.
What could be going on here?