I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:

S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.

By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)

As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the "sentence-Transformer"- https://github.com/UKPLab/sentence-transformers. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?