I'm currently training a binary classifier that takes in 2 inputs, and outputs which object it thinks is "better."

I have an absolutely massive dataset, about 2 trillion records, and I'm feeding these records into my network about 300k records at a time. Overfitting isn't really a concern as I'm only running one epoch, so the network is only really seeing new data every cycle.

So far, the training loss is decreasing steadily, and my training accuracy is rising.

My validation loss is decreasing slightly, but fluctuating a lot. My validation accuracy is rising at about the same rate as my training accuracy is.

Will I eventually see a large drop in validation loss as the network gets more "confident?" I know that's a very hard question to answer without knowing more details, but have you seen this type of behavior before in your models? I'm just a little spooked since these experiments take up so much time to run.

  • $\begingroup$ I'm assuming that you have a large number of objects too. You will need a large network in order to retain the information from so many data points, and indeed loss will eventually flatten out. It sounds like you have data from one of those "Would You Rather?" websites, in which case I would suggest that neural networks are not the way to go here. May I ask how the two objects are being input? Is it simply a vector of zeros with two ones in positions to indicate the two objects? How are you getting output? With a vector equal in length to the input size? $\endgroup$ Oct 18, 2020 at 6:01
  • $\begingroup$ This can only be answered by actually letting it train. Until then, the best answer is "maybe, or maybe not". $\endgroup$
    – noe
    Oct 18, 2020 at 16:27

2 Answers 2


When training neural network, you need to train many epochs in order for the model to learn enough to generalize. What is happening now the model is changing weights to fit the data that it observes (lower training loss) but does not generalize (as evidenced by high variance on validation data).

There are a couple of options:

  • Downsampling your data. Take only the most meaningful data to train.
  • Scale up training. Either use graphics processing units (GPUs) or distributed training.
  • Run training for longer.

If you have a lot of data and maybe records are very similar i suggest you to sample, probably keeping the same ration between 0 and 1 in order to train for more that 1 epoch your model.
In my experience if you don't reach overfitting you are underfitting which is worse in my opinion and you need to apply techniques to reach that overfitting like train for more epochs, add parameters find new features or try different models (gradient boosting tree are pretty good at overfit)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.