there are 2 basic queries that i haven't been able to find answers to on reddit / DSE.. hopefully someone can enlighten / educate me To populate/ ingest the data, typically its broken into chunks and during retrieval you retrieve the top-k results and use a bunch of techniques by combining all of the retrieved chunks a) first of all, the sheer difference between the context available in the query and the vectors stored is quite large. The query is typically 40-50 tokens at max as opposed to 1024 or whatever the chunk size is. Now whatever embeddings you were to use, the distance between these 2 will typically be large right ?
b) even if i were to dismiss the above problem, i m still stuck with judging the quality of the retrieval. All i would have is a score ( whether its generated using cosine / dot / euc is immaterial ) .. wont i have to use some sort of a thresholding mechanism OR another set of algorithms to ensure that a score of 0.4 is ok to proceed to dump into LLM context ?
i experimented with (a) using bert sentence embeddings and a qdrant database and found the results to be sub par in terms of the results returned. In some cases the top-k did not contain any relevant answers ..any idea how practitioners solve this ?