I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with the sum of embeddings of the constituent sentences:
import re
def split_into_sentences(text):
# Split using regular expression to preserve punctuation at the end of sentences
sentences = re.split(r'(?<=[.!?])\s+', text)
return sentences
text = "Tigers, the largest of all big cats, possess an undeniable allure that captivates the human imagination. Their iconic coat of burnt orange, embellished with bold ebony stripes, paints a portrait of both grace and potency, symbolizing the untamed beauty of the natural world. As stealthy hunters, they navigate their diverse habitats with an air of both confidence and mystery, often lurking within the dense undergrowth of jungles or prowling the open grasslands with unparalleled stealth. A tiger's sinuous movements reflect a balance between athleticism and elegance, a testament to their adaptability in various terrains. These felines are not merely ground-bound; their prowess extends to swimming, displaying an unexpected dexterity in the water, and climbing, as they ascend trees with remarkable agility. Intricately patterned, a tiger's stripes are akin to a fingerprint, unique to each individual, and play a vital role in their camouflage while stalking prey. It is the eyes of these creatures, however, that truly leave an indelible mark—an intense amber gaze that radiates an aura of fierce determination. Throughout history and across cultures, tigers have held a mythical status, embodying strength, wisdom, and courage. Yet, the same forces that once revered them now threaten their existence. The encroachment of human activity upon their habitats and the insidious specter of poaching pose grave challenges to their survival. In response, conservation efforts have emerged as a beacon of hope for these magnificent creatures. Collaborative initiatives, alongside advancements in technology and awareness campaigns, strive to protect and preserve their habitats, ensuring a future where tigers continue to roam the landscapes they have graced for millennia. The allure of the tiger, both as a symbol of nature's magnificence and as a reminder of our responsibility as stewards of the planet, remains as potent as ever—a reminder that the fate of these enigmatic creatures is intertwined with the destiny of our world."
sentences = split_into_sentences(text)
text_embedding = get_embeddings([text])[0]
sentence_embeddings = get_embeddings(sentences)
print([np.round(cosine_similarity(text_embedding, e), 3) for e in sentence_embeddings])
print(cosine_similarity(text_embedding, np.sum(sentence_embeddings, axis=0)))
I get the output:
[0.677, 0.462, 0.526, 0.701, 0.586, 0.797, 0.544, 0.697, 0.163, 0.336, 0.32, 0.575, 0.697]
0.8497203877599846
The first line compares each of the sentences with the whole paragraph. The second line compares the sum of sentence embeddings with the embedding for the whole paragraph.
It strikes me as unexpected that the sum of all sentence embeddings (~0.85) does not do much better than the sixth sentence alone (~0.80). Why is that the case?
