I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, xlsx, docx, and other Microsoft Office file types.
Some documents contain digital text (90%), other documents are scanned copies (10%). Some documents are in english (5%), some are in russian (5%) and rest of them is mix of english and russian (containing translations with similar meaning). Also some documents are packed into zip or rar archives.
What is the best approach to parse all this documents for text data?
What is the best approach to store extracted data?
How to make this data searchable? E.g. if I want to list top 30 documents that contain text similar to my search paragraph.
PS: I am interested to do this task in the shortest time possible, so I guess possible solution would involve big data techniques and distributed computing.