I have made a related document finder natively in SQL Server. Currently I use the word as dimensions. Based on that i calculate the cosinus simularty to figure out related article. At the moment the dimension around 100k words . I tried to reduce it by

  • stop word
  • rare or frequent uses words(based on the corpus)
  • only lower case
  • take care of POS
  • fix spelling errors
  • use lemma
  • I have exhausted all linguistic filtering possibilities known to me

I am looking for further options. Mainly for options they can directly done in sql server (CLR are ok, phyton not) Pca or svd i think are good options but there is no implementation yet.

  • $\begingroup$ A further option to reduce the memory and computation cost, if you haven't already tried it, could be to switch from the dense to sparse format. Another option is switching to embeddings as suggested in the answer, however this may change your results. $\endgroup$
    – Valentas
    Apr 13, 2023 at 19:30

1 Answer 1


How are you using the word as a dimension? For many NLP tasks, you typically want to generate an embedding to represent the word, where the embedding dimension is going to be much smaller than the total vocabulary size. This would look something like the following for OpenAI embeddings or HuggingFace GTR-T5 embeddings:

encoder = "text-embedding-ada-002"
def get_openai_embedding(text: str, model: str=encoder) -> list[float]:
    result = openai.Embedding.create(
    return result["data"][0]["embedding"]

def get_t5_embedding(text: str, model: str=encoder) -> list[float]:
    model = SentenceTransformer('sentence-transformers/gtr-t5-xxl')
    embeddings = model.encode(text)
    return embeddings

def get_embedding(text: str, model: str=encoder) -> list[float]:
    if encode_with_openai:
        return get_openai_embedding(text, model)
        return get_t5_embedding(text, model)

Also, if it's feasible, you might want to investigate vector databases instead of SQL - you generate an embedding for a query, and pass the embedding to the vector dB which will return the most similar documents. I've used PineCone and Weaviate but there are many others. OpenAI has some example notebooks here: https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases

  • $\begingroup$ i didn't want to embed additional tools. is there more than a schematic explanation for the embedded words somewhere.... best with comprehensible example data. so far i had no luck. $\endgroup$
    – ozz
    Apr 12, 2023 at 14:27
  • $\begingroup$ How are you converting your words to vectors? If you are just using a list of all words in your vocabulary and each word is represented by its index, that's one hot encoding. Something like word2vec or glove would likely be much better in terms of both size and utility for downstream tasks like determining similarity. There's a good article on embeddings here: towardsdatascience.com/…, but many other sources as well. hth. $\endgroup$ Apr 12, 2023 at 15:03
  • $\begingroup$ i use tf-idf to create a vector for the document. pretty simple but efficient $\endgroup$
    – ozz
    Apr 12, 2023 at 16:21
  • $\begingroup$ so your vector size for tf-idf is going to be pretty close to the number of tokens in your vocabulary. This is why I'm recommending looking at word2vec or glove. All of the preprocessing that you do will help a little bit, but depending on the corpus it's still easy to end up with thousands of words. With an embedding you can not only capture more information about relationships between words, but do it in a much smaller vector, e.g. 300-long vector. $\endgroup$ Apr 12, 2023 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.