how can I translate Whisper encodings to SBERT embeddings?

Question

I'm using the Whisper model to recognize speech, and then matching the output text against a list of known questions by generating SBERT embeddings from the text and ranking the known questions by cosine similarity with the SBERT embeddings of the text output from Whisper. It works pretty well, and I'm happy with the level of accuracy.

I'd like to streamline the process a bit, and I understand I can get the encoder embeddings from the whisper model output rather than just the transcribed text.

My question is: What's the best way to fuse these steps together? More generally, is there a good way to translate embeddings from one model vector space to another? What would that task even be called in terms of linear algebra?

Answer 1

The problem here is that the SBERT embedding of a piece of text is a single vector, while the embeddings you get from Whisper are a sequence of vectors. Therefore, it's not a matter of just mapping two embedded spaces, but mapping a sequence of vectors in an embedded space to a single vector in a different space.

Of course, you could train a small multi-head attention to mimic the equivalent SBERT but nothing guarantees that such an approach would give comparable results to computing the SBERT from Whisper's output text, or that it would be computationally worth it.

Answer 2

More generally, is there a good way to translate embeddings from one model vector space to another?

I am not sure if my answer is correct, but, in my opinion, linear transformation of embedding can transfrom from one model vector space to another.

Also, you can argument vector by using PCA (principal component analysis). Principal component analysis is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and enabling the visualization of multidimensional data. wikipedia's explanation is here