Is using embeddings for regularization a valid practice?

My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and consequently less prone to overfitting.

This sounds as a reasonable approach to me, specially when dealing with data where the entries are vectors of large dimension that exibith high similarity.

However, I could not find a reference of embeddings being used for regularization. Hard to believe that if it was indeed a valid approach, it wouldn't have appeared sooner.

  • $\begingroup$ Hi @AdenilsonArcanjo, welcome to the site. I understand that you mean that using embeddings reduces the number of trainable parameters with respect to one-hot encoding. Is that correct? If not, can you elaborate? $\endgroup$
    – noe
    Oct 11, 2023 at 7:54
  • $\begingroup$ Yes, you are right. To provide more context, I am currently dealing with a dataset comprising protein sequences, each having an approximate length of 400 amino acids, or, equivalently, 400 letters. When these protein sequences are one-hot encoded, they yield vectors with a dimensionality of 400 multiplied by 21. These protein sequences exhibit a high degree of similarity among themselves. Importantly, there is a high level of similarity between entries, when comparing two different entries most of their characters will be equal. $\endgroup$ Oct 11, 2023 at 18:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.