Will my validation loss eventually go down?

Question

I'm currently training a binary classifier that takes in 2 inputs, and outputs which object it thinks is "better."

I have an absolutely massive dataset, about 2 trillion records, and I'm feeding these records into my network about 300k records at a time. Overfitting isn't really a concern as I'm only running one epoch, so the network is only really seeing new data every cycle.

So far, the training loss is decreasing steadily, and my training accuracy is rising.

My validation loss is decreasing slightly, but fluctuating a lot. My validation accuracy is rising at about the same rate as my training accuracy is.

Will I eventually see a large drop in validation loss as the network gets more "confident?" I know that's a very hard question to answer without knowing more details, but have you seen this type of behavior before in your models? I'm just a little spooked since these experiments take up so much time to run.

Answer 1

When training neural network, you need to train many epochs in order for the model to learn enough to generalize. What is happening now the model is changing weights to fit the data that it observes (lower training loss) but does not generalize (as evidenced by high variance on validation data).

There are a couple of options:

• Downsampling your data. Take only the most meaningful data to train.
• Scale up training. Either use graphics processing units (GPUs) or distributed training.
• Run training for longer.

Answer 2

If you have a lot of data and maybe records are very similar i suggest you to sample, probably keeping the same ration between 0 and 1 in order to train for more that 1 epoch your model.
In my experience if you don't reach overfitting you are underfitting which is worse in my opinion and you need to apply techniques to reach that overfitting like train for more epochs, add parameters find new features or try different models (gradient boosting tree are pretty good at overfit)