I'm trying to replicate result of a paper. The paper is a U-net for De-noising of some images. So basically I have a simple U-net that I give noisy data as input and have denoised data as the wanted output (use l2/MSE loss) . So, in the paper and generally in most papers like this (deep learning applied to medical imaging), they say they run the model for something like 300 epochs or they say they run it like ~50 hours. My question is that aren't they supposed to put call backs for the validation loss, so once the validation loss stops improving they stop the training, otherwise the model will overfit? Also does the number of batches in an epoch matter. If because of memory constraints I use 2 batch sizes per epoch, would it be a bad practice?
-
$\begingroup$ Welcome to the site! Can you clarify what you mean by "I use 2 batch sizes per epoch" -- do you mean a batch size of two (i.e., only two images per batch) or two batches (of varying sizes) per epoch? $\endgroup$– timleathartJul 4, 2019 at 21:15
-
$\begingroup$ two batches per epoch. The whole data is something like 1000 samples of (64,64,32) 3D images. So, the input of the data would be (1000,64,64,32,1) for (batch numbers, width, height, depth, number of channels = 1 (grayscale)). $\endgroup$– JackJul 4, 2019 at 21:20
1 Answer
Using validation loss to determine when to stop iterating is a good strategy when you have a lot of labeled training data. However, in medical contexts, it's often the case that labeled data is expensive and/or difficult to come by. You mention that you only have 1000 samples -- this is a pretty small amount of data to be training a deep net with! Setting aside 100 or 200 images to form a validation set will probably hurt the model. You can use a validation set to get an estimate of how many epochs you should train for, and then train on the whole dataset for that many epochs.
In regards to your batch size query: I would recommend working with smaller batch sizes of around 32-128 images. In my experience, the networks converge more quickly when using smaller batch sizes.
-
$\begingroup$ Ok, thanks. But here's what I'm struggling with. If I run 300 epochs as most papers in medical imaging do (100s), aren't they overfitting? I read in a book (I think DL with python) that you put callbacks for early stopping in order to stop overfitting, $\endgroup$– JackJul 4, 2019 at 21:42
-
$\begingroup$ Also, let's set aside the medical imaging. In some other application where you're training on medium sized data. Is it acceptable to train some DL model using 100 epochs or will it be considered overfit? $\endgroup$– JackJul 4, 2019 at 21:44
-
$\begingroup$ It all depends on other factors such as learning rate, regularisation, number of parameters, etc. It's possible to run as many epochs as you like and not overfit if your other hyperparameters allow. As you say, the callbacks are a good idea for combatting overfitting, but only when you have enough data to begin with. $\endgroup$ Jul 4, 2019 at 22:02
-