0
$\begingroup$

I have a dataset consisting of text-based conversations between two humans. One conversation has on average 20 turns and can look as follows:

Person 1: Do you like cooking?
Person 2: Yes. I like cooking very much. I got this hobby when I was 12 years sold.
Person 1: Why do you like it?
Person 2: I have no idea. I like cooking by myself. I like to taste delicious food.
...

With SBERT I can get the embeddings of one turn (e.g., "Hello there, how are you doing?"). Is it also possible to get one embedding with SBERT for several turns or a whole conversation (20 turns)? Are there other models which are capable to do this or are more recent? Afterward, I would like to project the embedding to 2D or 3D space and apply clustering.

$\endgroup$

1 Answer 1

1
$\begingroup$

SBERT supports batch inference, which means that it can process multiple sentences at the same time. For instance (source):

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')

#Our sentences we like to encode
sentences = ['This framework generates embeddings for each input sentence',
    'Sentences are passed as a list of string.', 
    'The quick brown fox jumps over the lazy dog.']

#Sentences are encoded by calling model.encode()
embeddings = model.encode(sentences)

#Print the embeddings
for sentence, embedding in zip(sentences, embeddings):
    print("Sentence:", sentence)
    print("Embedding:", embedding)
    print("")

The limit for how many sentences you can fit will depend on their length and the amount of memory in your GPU/CPU. Note that the needed amount of memory is quadratic on the sequence length. You should find the appropriate limit for the number of sentences empirically.

About projecting the embeddings and clustering, I suggest you use UMAP for the dimensionality reduction and then k-means for clustering.

$\endgroup$
5
  • $\begingroup$ Thank you for your answer. With your code, an embedding is generated for each sentence (e.g. each turn of the conversation). Is it possible to generate one embedding for multiple turns? In my example, I would like to generate one single embedding for the example conversation. $\endgroup$ Mar 3, 2023 at 15:34
  • $\begingroup$ Well, you can either concatenate the sentences together or average the individual sentence embeddings. $\endgroup$
    – noe
    Mar 3, 2023 at 16:59
  • $\begingroup$ I see. Do you think that fine-tuning SBERT on conversations would be an option? If yes, how would you format the conversations for fine-tuning? Should one input sentence/string be Person 1: Do you like cooking?\nPerson 2: Yes. I like cooking very much. I got this hobby when I was 12 years sold.\nPerson 1: Why do you like it? etc? Or should I omit Person 1 and Person 2? Or just concatenate all turns without \n? $\endgroup$ Mar 3, 2023 at 19:59
  • $\begingroup$ Do you know another model than SBERT which can directly handle conversations (i.e. was trained on conversations)? $\endgroup$ Mar 3, 2023 at 20:01
  • $\begingroup$ No, sorry, I am not aware of vectors specifically meant to get embedded vectors out of conversations. $\endgroup$
    – noe
    Mar 3, 2023 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.