I'm using BERT to encode sentences. The sentences I'm encoding are quite similar, meaning they all belong to the same overall topic. Therefor, I am using another parameter for measuring similarity. So, when I want to add even more similarity to BERT's encodings, I simply pad a vector of size 100 filled with value 1 to the beginning of BERT's vectors. This makes them more similar when cluster / doing similarity measurement.
Now this i somewhat of a hack, and I probably should except it to perform on all sentence input types. Do any of you have an idea of how to achieve the same, in a more correctly / scalable way.
I've tried finetuning BERT, though this simply takes to long and uses to many resources for my task.