I'm currently using pre-trained transformers to extract embeddings for sequence analysis, which are then used in downstream tasks. My process involves using the extracted embeddings as features for training models tailored to specific applications. Recently, I came across a study 1 that not only utilizes embeddings from an MSA transformer but also trains models directly on the extracted row attention weights independently.

This approach intrigues me since it's not commonly seen in the literature or practices I've encountered. Is it a practical approach to train downstream models directly on attention weights derived from transformers?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.