I'm using the Whisper model to recognize speech, and then matching the output text against a list of known questions by generating SBERT embeddings from the text and ranking the known questions by cosine similarity with the SBERT embeddings of the text output from Whisper. It works pretty well, and I'm happy with the level of accuracy.
I'd like to streamline the process a bit, and I understand I can get the encoder embeddings from the whisper model output rather than just the transcribed text.
My question is: What's the best way to fuse these steps together? More generally, is there a good way to translate embeddings from one model vector space to another? What would that task even be called in terms of linear algebra?