Im trying to build a search engine for a private dataset using vector space model and have encountered following problem.
Dataset is private. It is a collection of unstructed pdf . I have built a parser to parse the pdf and extract relevent information and stored them in MongoDB in the following JSON format.
PDFNAME -> Heading -> SubHEAD -> Key -> Value
But the value could be paragraph, neumeric or could be a table.
Syntax of search query the user will be typing will be as following
PDFNAME Heading SubHead Key Value
But here, the user will not be giving exact terms that I have stored in MongoDb. Espicially the Value . User might give one or two word from the paragrapgh/table kind of value stored.
I have clustered (for faster result)the documents based on similar PDFNAME and followed a leader follower approach. I have used cosine similarity to find the match between query and terms from MongoDb building a vector space model of the terms that comes under the leader cluster.
Problems Im facing
Say I have documents containing chemical information Carbon dioxide and its composition in neumeric.
User will be searching for " XYZ test carbon diozide 54% and carbon monoxide 64% "
Q] If using cosine similarity of word vector space then how to differentiate the nuemericals ? Like the % composition of carbondixide and carbon monoxide that might exists in the same document ? Also how to handle the unit that might associated with the value. Sometimes it will be missing in Db and user will be specifying it in the query.
For example, A row in the dataset might look as follows https://jsonblob.com/4ba76936-429e-11ea-bdd3-316e38de44c5
And the user will be querying something like as follows
design pressure 6 barG and molecular weight used is 16.68 when Mothiram Pandidhurai is used
This should result ABC Project with some query matchiing percentage . ie Rank some near documents with some order