I have 986 voice signals which have been collected by our team. The data set includes 745 healthy and 150 unhealthy voice signals. I split the data into 70% training and 20% validation and 10% test (unseen) data.

Then, I oversampled the train and validation set. So, we have 1042 training samples (521 healthy and 521 unhealthy) and 298 validation samples (149 healthy and 149 unhealthy). I am applying augmentation on the fly on each batch with 64 batch size and on both training and validation sets. To work with these signals and CNN, I converted the signals into Melspectrograms.

Now, when I apply a five-layer CNN, there is an overfitting and the model can not generalize well on validation data.

What I did so far:

  1. I just applied augmentation on training set and I did not applied augmentation on validation set, the problem still exists.
  2. I used shallow CNN, but the problem still exists.
  3. I used Regularizer (l2), Dropout, Batch normalization, different optimizer (Adam, SGD,…), different learning rate, label smoothing, early stopping, scheduler, …but the problem still exists.
  4. I used different augmentation techniques/ libraries, but the problem exists.
  5. I extracted the best informative chunk (1 second) from the 5 seconds voice signals. I mean I removed background noise and silent, but the problem still exists.
  6. I applied PCA to reduce the number of mels in mel-spectrograms, but the problem still exists.
  7. I applied transfer learning like (Xception, Resnet50, Inception), but the problem still exists.
  8. I tried to get optimized hyperparameters in Melspectrograms like (n-mels, n-ffts, hop-length) by defining cost function using MSE or Euclidean distance, but it was not helpful!!
  9. I added librosa.feature.delta as dynamic feature , but it was not helpful!! 10.Even the cross validation was not helpful!!

But, based on my first experiments, when I prepared the data before fitting any model, I mean when I oversample and augment the data offline(which means to add more data to train or validation set) instead of applying augmentation on the fly (which doesn’t add any data, just make more diverse data in each batch), the results are acceptable.

Based on this explanation, I want to have a good fit and good performance using augmentation on the fly. I would appreciate it if anyone could help me.


1 Answer 1


In some scenarios overfitting is unavoidable and the only solution is to gather more data. It is impossible to know whether that might apply in your situation.

If you are finding that offline augmentation does not cause overfitting but on-the-fly augmentation does cause overfitting, something seems wrong. Check your implementation, and check that you are performing exactly the same augmentation in both cases and that all other hyperparameters are identical. On-the-fly augmentation should be at least as good as offline augmentation, and sometimes better. You mention oversampling; check that you are doing the same oversampling in both situations.

You should not be doing augmentation to data in the validation set, only to data in the training set.

  • $\begingroup$ thanks for the reply. I used offline augmentation on Mel-spectrograms, but I used on-the-fly augmentation on voice signals. so, the augmentation techniques are different. Can it have such effect on the overall result? on the other hand, my aim is to apply augmentation on voice signals instead of Mel-spectrograms. it seems the code and implementation is correct, because each part of the code shows the expected output. My concern is exactly what you said "On-the-fly augmentation should be at least as good as offline augmentation, and sometimes better." $\endgroup$
    – Zara Nz
    Commented Apr 4, 2023 at 15:13
  • $\begingroup$ Can we say when dataset is relatively small the offline augmentation is useful and on the other hand, on-the-fly augmentation is useful when dealing with large datasets ? $\endgroup$
    – Zara Nz
    Commented Apr 4, 2023 at 15:35
  • $\begingroup$ @ZaraNz, yes, of course it can lead to different results, if you use different augmentation strategies you might get different results. I think you need to spend more time debugging more systematically. No, I don't think you can say that. Please don't use this comment thread to ask new questions - I suggest posting any new question using the 'Ask Question' button. $\endgroup$
    – D.W.
    Commented Apr 4, 2023 at 16:21
  • $\begingroup$ Thanks again. I wanted to post a new question for my last question, but I thought it would be confusing without any background about my issue. I ran the code step by step and I got the expected outputs, may there any other bugs in the code in such situation?!! $\endgroup$
    – Zara Nz
    Commented Apr 4, 2023 at 18:32
  • $\begingroup$ @ZaraNz, this isn't sounding like a good fit for the Stack Exchange format. Our goal is to build a high-quality archive of knowledge that will be useful to others. Questions that need interactive support, back-and-forth discussion, or that are relevant only to you are not likely to be a good fit here. I think you know what to do now: you know that the different types of augmentation might make an issue, and you can try experimenting with changing one thing at a time, rather than changing both type of augmentation and offline and on-the-fly at the same time. The ball is in your court... $\endgroup$
    – D.W.
    Commented Apr 5, 2023 at 4:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.