Skip to main content
Data Science

Return to Question

Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
added 12 characters in body
Source Link
Ethan
Ethan
  • 1.7k
  • 9
  • 24
  • 39

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:

S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.

By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)

As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the "sentence-Transformer"- https://github.com/UKPLab/sentence-transformersTransformer. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:

S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.

By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)

As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the "sentence-Transformer"- https://github.com/UKPLab/sentence-transformers. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:

S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.

By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)

As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the sentence-Transformer. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?

Source Link
Bloodstone Programmer
Bloodstone Programmer
  • 300
  • 2
  • 3
  • 10

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:

S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.

By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)

As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the "sentence-Transformer"- https://github.com/UKPLab/sentence-transformers. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?