0
$\begingroup$

I have a set of .pdf/.docx documents with content. I need to search for the most suitable document according to a particular sentence. For instance:

 Sentence: "Security in the work environment"

The system should return the most appropriate document which contains at least the content expressed in the sentence. It should be a sort of search bar with advanced capabilities; I have a constraint: I can not have an apriori classification since the number of documents and the related category could vary on time.

How should I address this kind of task?

$\endgroup$

1 Answer 1

1
$\begingroup$

If you are asking how to integrate this, I would leverage existing search technologies such as storing documents in mongo database or using solr indices just to name a few..

If you are asking on the implementation details, take a look on topic modeling, tf-idf, cosine similarity, synonym replacements, k-nearest neighbors to get you started. A lot of these techniques could either be used at index time on incoming documents or at query time to perform minimum work to reduce search space scope even if your documents & queries are dynamically changing. You'll probably want to allocate a test set of ranked documents expected to be returned by sample queries so you can benchmark your improvements.

$\endgroup$
3
  • $\begingroup$ Thanks for your reply. I was asking about the implementation details, thank you again for that. I see your point. I think that anyway such document should be converted into txt (for instance using python OCR facilities) to use topic modeling, tf-idf, etc... isn't it? $\endgroup$ Dec 28, 2021 at 15:45
  • $\begingroup$ Thats right: you'll need to convert docx/pdf to text. I would recommend transforming into some sort semi-structured text format such as json/html so you can capture titles, sections, paragraphs, etc & keep some semantics of the docs. It might get messy with tables & figures.. $\endgroup$
    – eliangius
    Dec 28, 2021 at 16:00
  • $\begingroup$ Thank you @eliangius. is there a python library to do somehow what you mentioned? i.e., transforming the documents into json/html. Thank you again $\endgroup$ Dec 28, 2021 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.