The nature of the data I have is not arranged; however, I'm trying to extract the appropriate sentences for each query as a sample for ground truth. Also, the most critical problem is that I use the BERT model, which searches for the top 500 sentences in which the similarity value is greater than the 0.5 threshold. So the result is disorganized. For example, I got for a query

5 for the relevant, 495 for the irrelevant, and 6 for the Total relevant

How can I avoid the irrelevant number of sentences in the result?

  • $\begingroup$ It seems you have labeled data about the relevant/irrelevant sentences, so you may train a classifier to discard the irrelevant ones. $\endgroup$
    – noe
    Oct 24, 2023 at 9:15
  • $\begingroup$ Thanks for replying. The sentences themselves are not labeled, but it will take time for me to label them manually. Can I share the code here to ensure that I'm on the right way? $\endgroup$
    – Begnnier
    Oct 24, 2023 at 9:39
  • $\begingroup$ How did you know that 495 sentences were irrelevant? $\endgroup$
    – noe
    Oct 24, 2023 at 9:42
  • $\begingroup$ I wrote in the code top_k = min(500, len(content_list)) so the results will be the first top 500 sentences, then I revised them to capture the right sentences for the query I asked $\endgroup$
    – Begnnier
    Oct 24, 2023 at 9:46
  • $\begingroup$ I see. I am afraid that if you can't produce relevant/irrelevant labels easily, I don't see how you can remove the irrelevant ones automatically. Maybe using ChatGPT to create a dataset based on some hand-made examples. $\endgroup$
    – noe
    Oct 24, 2023 at 10:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.