5
$\begingroup$

I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, xlsx, docx, and other Microsoft Office file types.

Some documents contain digital text (90%), other documents are scanned copies (10%). Some documents are in english (5%), some are in russian (5%) and rest of them is mix of english and russian (containing translations with similar meaning). Also some documents are packed into zip or rar archives.

What is the best approach to parse all this documents for text data?

What is the best approach to store extracted data?

How to make this data searchable? E.g. if I want to list top 30 documents that contain text similar to my search paragraph.

PS: I am interested to do this task in the shortest time possible, so I guess possible solution would involve big data techniques and distributed computing.

|improve this question
$\endgroup$
  • $\begingroup$ It seems you have a "data lake". There are techniques specific to your storage setup if you look into data lakes. $\endgroup$ – Edmund May 1 at 12:06
1
$\begingroup$

One option is to use Apache SOLR + Apache TIKA (https://lucene.apache.org/solr/ , https://tika.apache.org/).

Apache TIKA has support for most common file formats, it extracts test content from files. Extracted text can be stored in SOLR. SOLR supports various kinds of text + aggregation queries.

Tutorials :

https://blog.webnersolutions.com/apache-solr-indexing-all-files-in-a-folder-recursively

https://lucene.apache.org/solr/guide/6_6/introduction-to-solr-indexing.html

https://lucene.apache.org/solr/guide/6_6/uploading-data-with-solr-cell-using-apache-tika.html

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.