enter image description here

I have an encoder-decoder architecture where I have used top 3 layers of Swin Transformer and few convolutional layer. I tried different approach:

i. Training the Transformer layers as well, on doing so model contains approximately 304,086*2(encoder + decoder) trainable parameters.

ii. Freezing the transformer layers approx. 105 * 2 = 210 (encoder + decoder) total trainable parameters. This also shows I have very few layers of CNN.

On both the approach the validation loss is higher than the training loss. The above depicted curve is for approach (i).

I have 7K trainable data and have used 700 for validation. Also, used L2-Regularization but the results doesn't change.


1 Answer 1


Your model is not small at all, and I would argue that it's actually very large. You can consider how much data you have compared with the number of parameters. Thus you have 700k parameters to fit from less than 7k occurences. This is not small and I have strong doubts this can be solved by any amount of regularization.

  • $\begingroup$ It is the same problem with a model having 86K parameters. I have even used heavy regularization of 1e-2 and dropout after each convolutional layers. $\endgroup$ Commented Nov 7, 2022 at 15:09
  • $\begingroup$ It does not change the situation much. Imagine you have a linear model with 10 parameters and one observation, it is a similar situation. $\endgroup$
    – rapaio
    Commented Nov 7, 2022 at 16:48
  • $\begingroup$ What can be the best situation in this case? I can not increase the data. Decreasing the model from 86K parameters to something below starts to underfit. $\endgroup$ Commented Nov 7, 2022 at 17:08
  • $\begingroup$ I don't know enough details to give you any practical advice. However if you really cannot simplify your model (which I think you should explore more) you can try to augment your data, simplify the input, droput, max layers for convolutions, other configs for convolutions. $\endgroup$
    – rapaio
    Commented Nov 7, 2022 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.