2
$\begingroup$

Let's say I am training a neural net (e.g. convolutional network or LSTM).

Generally, the longer the training (more epochs) leads to better accuracy, albeit at times at the expense of overfitting.

Another approach is to duplicate the data. Again, if I have 1000 examples, I could duplicate them 10 times and randomise them, and the neural net will learn better (but again, it may not be able to generalise well).

Is there any difference between duplicating data and adding epochs when training a neural net in terms of its learning?

Improve this question
$\endgroup$
1
$\begingroup$

The most obvious difference is that by adding epochs, you (still) never get the same observation twice in a same batch, which isn't the case when you duplicate data.

I'm no expert in NNs, but it seems that if you do not introduce any noise in the duplicated data, then you might as well just add epochs.

Of course, if you fit two models over a single epoch,

model 1: using your n=1000 observations

model 2: using your n*10 = 10,000 observations (with duplications)

your second model will most probably return a lower loss value. However, a fairer comparison would be to train model 1 with 10 epochs so that in both cases you expose your net to your observations the same number of times in total.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Regarding your first observation, as long as the batch size is small enough, I will not see the same observation in the same batch anyway, $\endgroup$ – user Apr 3 '18 at 2:02
  • $\begingroup$ Its more of sheer luck in this case which will play the role..whether a given set of tuoule will be available twice or not.... But what you can defiantly do is to add random noise to all of them in a well engineered way so as they are not in contradiction with the true ones.. $\endgroup$ – Aditya Apr 3 '18 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.