Do we need to apply text cleaning practices for the task of sentence similarity?

Most models are being used with whole sentences that even have punctuation. Here are two example sentences that we wish to compare using SentenceTransformer (all-MiniLM-L6-v2):

sentences = [
    "Oncogenic KRAS mutations are common in cancer.",
    "Notably, c-Raf has recently been found essential for development of K-Ras-driven NSCLCs."] 

# yields that sentence 2 has a score of 0.191 when compared with sentence 1

Will cleaning those sentences change its semantic meaning?

cleaned = ['oncogenic bras mutations common cancer', 
           'notably c-raf recently found essential development bras driven nsclcs.']

# yields that sentence 2 now has a score of 0.327 when compared to sentence 1

It seems the model works better when the text is cleaned. However, nowhere does it say that the input sentences are being / should be cleaned? Would love to know your takes on this.


1 Answer 1


Answer: Transformer based models used for sentence similarity have been trained on huge amounts of data where the text preprocessing part has been handled either at the tokenization step or by the attention mechanism of the transformer.

Applying cleaning methods and then using the cleaned text as input will worsen the quality of the embeddings. The inputs now differ from the ones the model has been trained with.

The attention mechanism will be the one who will neglect tokens that are meaningless and include ones that are meaningful. In that case, a comma or a number can be meaningful in some context and meaningless in another, hence why we should not clean the text on our own but use it as it is.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.