Im trying to build a search engine for a private dataset using vector space model and have encountered following problem.


Dataset is private. It is a collection of unstructed pdf . I have built a parser to parse the pdf and extract relevent information and stored them in MongoDB in the following JSON format.

PDFNAME -> Heading -> SubHEAD -> Key -> Value

But the value could be paragraph, neumeric or could be a table.

Search Query

Syntax of search query the user will be typing will be as following

PDFNAME Heading SubHead Key Value

But here, the user will not be giving exact terms that I have stored in MongoDb. Espicially the Value . User might give one or two word from the paragrapgh/table kind of value stored.

I have clustered (for faster result)the documents based on similar PDFNAME and followed a leader follower approach. I have used cosine similarity to find the match between query and terms from MongoDb building a vector space model of the terms that comes under the leader cluster.

Problems Im facing

Say I have documents containing chemical information Carbon dioxide and its composition in neumeric.

User will be searching for " XYZ test carbon diozide 54% and carbon monoxide 64% "

Q] If using cosine similarity of word vector space then how to differentiate the nuemericals ? Like the % composition of carbondixide and carbon monoxide that might exists in the same document ? Also how to handle the unit that might associated with the value. Sometimes it will be missing in Db and user will be specifying it in the query.

For example, A row in the dataset might look as follows https://jsonblob.com/4ba76936-429e-11ea-bdd3-316e38de44c5

And the user will be querying something like as follows

design pressure 6 barG and molecular weight used is 16.68 when Mothiram Pandidhurai is used

This should result ABC Project with some query matchiing percentage . ie Rank some near documents with some order


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.