2
$\begingroup$

I have a list of items (size ~50K) and several documents( average page per document ~10). I am trying to find what all items are listed in each document as follows :

for document in documents_folder:
    document_text = extract_content(document)
    for item in item_list:
        if item is found in document_text:
             save(doument,item)

This takes ~5 sec per document to execute.

How can this be optimized ? Is this the best approach ? Is there a better way to do this ?

Any suggestion is appreciated

$\endgroup$
1
$\begingroup$

I have not tried this before but I will suggest:

  1. Get all unique items from all documents:

  2. Create a dictionary where key is that unique item and values is list of document ids (document names) where this item is present.

Once you are done. Just search your item in the dictionary key, it will return all the documents ids where this item is present.

This algorithm is called Inverted Indexing in Information retrieval.

| improve this answer | |
$\endgroup$
  • $\begingroup$ if I use ElasticSearch (inverted indexing) l need to index all documents first and then query for document with items right ? But what I need is opposite - a documet as input, items as output $\endgroup$ – Sreejithc321 Jan 29 '19 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.