Im a newbie in information Retrieval. Currently Im reading a book entitled "An Introduction to Information Retreival" by Christopher D. Manning and Prabhakar Raghavan. Im trying to implement an ai based search tool to search for some relevant information from a private dataset. ( say a chemical or mathematical dataset that invloves more number of numericals and unstructed messy representation of unit of measure)

Following is a row in my dataset in JSON format where keys are the coloumn name and value are the values corresponding to it.

  "ABC Project": {
    "In/Out diameter": " Both in and out are 1”  ",

    "Design Pressure (barG)": {
      "Max": "116 psiG (7.99 barG)",
      "Minimum": "79.7 psiG (5.49 barG)",
      "Design": "174 psiG (11.99 barG)"

    "C02 %": "0.671",
    "MW": "16.68 kg/kmol",
    "TITLE": null,
    "Mothiram": "There is  very dense forest and their lived a king. The name of the king was Pandidhurai. He was very brave ",
    "Thooval delivery material": "- thooval delivery material is panam patta 316/316L  "

The sample search query that will fetch the above mentioned row from my dataset would be

" The project in which molecular wight is nearly 16 kg/kmol and Thooval delivery material panam patta 316l is used in which Mothiram is King Pandidhurai "

What I have done.


Read each row from table(say $T$) as $R_{i}$ and search query as $Q$ do the following preprocessing

  1. convert_lower_case(data)
  2. remove_punctuation(data)
  3. remove_apostrophe(data)
  4. remove_single_characters(data)
  5. convert_numbers(data)
  6. remove_stop_words(data)
  7. stemming(data)
  8. remove_punctuation(data)
  9. Convert Word to Vector for coloumn $C_i$ and do it for each corresponding row in table $R_i$
  10. Build Vector Space Model for coloumn $C_i$ and do it for each corresponding row in table $R_i$
  11. Build a vector space model for query $Q$.
  12. Compute cosine score to select Row having high score

Challenges I'm facing

  1. When I do 'remove_punctuation(data)' Im loosing vital informations (say $"$ denotes inches in the diameter coloumn)

  2. A way to interpret unit of measure in the data

  3. Unable to keep the relation between value ( say 16.78 kg/k mol is stored in multi diamentions and unable to find it is related to MW). I think it could be solved by building a ML classifier and train it to identify the entity MW. Say < 16.78 kg/k mol , MW > and using the unit of measure as a feature. But there are values like percentage composition of different chemical components, IN and OUT diameter , Inside and Outside Temperature. etc

  4. Difficulty in interpreting scientific terms and mapping to one root form ( say MW, mol wt, molwt etc are all different ways of representing molecular weight)

  5. Difficulty in finding the close proximity of neumerical values


  • Please suggest me a step by step approach to build a search tool using this dataset (Please also suggest me an apt algorithm that would be useful in each step)
  • I have read from here that some documents can be tagged as < document, class > and train a Navie Base Classifier Model for better search performance. Is it possible in this case , if so what could be class label you would be suggesting ? UPDATE : I think Navie Base cannot be used as the coloumn or key would be greater than 1000. Im looking for a scalable approach to the problem.
  • Is there any better approach to solve this problem than using a vector space model and computing cosine similarity ?
  • 1
    $\begingroup$ It's not clear to me if you are doing the whole process for each query? Normally in an IR setting the dataset is preprocessed and the vectors are stored, so that when an input query is given one only has to represent the query and find the closest vector. $\endgroup$
    – Erwan
    Feb 3, 2020 at 14:11
  • 1
    $\begingroup$ @Erwan I have updated my question. Waiting for your valuable suggestions $\endgroup$ Feb 4, 2020 at 6:18
  • 1
    $\begingroup$ To me it seems you have multiple different problems. May I suggest you cut down your question in multiples sub questions ? $\endgroup$ Feb 4, 2020 at 11:04
  • 1
    $\begingroup$ @lcrmorin Please do suggest your views and approach to the problem. I'm kind of stuck. Sometime I might new idea from your valuable input $\endgroup$ Feb 4, 2020 at 12:03

1 Answer 1


Here are a few general thoughts about your project:

As far as I understand, you're trying to extract very specific information from a semi-structured database using free-form natural language queries, correct? If yes it's important that you realize that this is a quite ambitious project, reaching a decent stage of quality is probably going to take a lot of work, and the performance is unlikely to be perfect.

Apparently numerical values and units are important pieces of information for matching the query. In this case you should probably implement a special process for those, because standard text processing is not going to work very well.

  • Detection: if there is only a small number of possible ways in which these values are written, it's probably more efficient to use ad-hoc regular expressions. If not, you could try to train a custom NER model.
  • Representation: that's the tricky part imho. For general text vectors are fine, but with vectors it's difficult/impossible to represent special values like those. Given that you have a semi-structured database, you might want to try a more semantic representation adapted to your data: that could involve techniques from semantic role labeling, relationship extraction, etc.

  • Matching: the advantage of a semantic representation is that you could convert the query to a semantic representation and then apply a detailed matching procedure suited to your data (in particular you can compare corresponding numerical values and use a threshold, or return the difference to represent how far they are).

  • 1
    $\begingroup$ Thanks for understanding my usecase. It is exactly im looking for. I understand it is a project that has very less percentae of expected result. I am only expecting atleast 75-85% of accuracy in my search tool and I do see a research scope in it. Thanks for your valid input. Im looking for a systematic approach to this problem. It would be great if you could also suggest me the most apt algorithm that would be useful in each step and explain a bit more about the modus operandi $\endgroup$ Feb 5, 2020 at 5:46
  • 1
    $\begingroup$ @AkhilNadhPC for NER the standard method is to use conditional random fields (sequence labeling), but for the other parts related to semantic representation I'm not sure whether there is any standard method, as it tends to be very specific to the task. In particular in your case you have a specific structured database, so you probably could use expert knowledge suited to your particular data. $\endgroup$
    – Erwan
    Feb 5, 2020 at 20:21
  • 1
    $\begingroup$ Thanks for your input @Erwan . Lastly, Do you know word vectors that are trained on scientific corpora ? $\endgroup$ Feb 6, 2020 at 6:05
  • 1
    $\begingroup$ @AkhilNadhPC apparently from a quick google search there are some at least for the biomedical domain (see for instance ncbi.nlm.nih.gov/pmc/articles/PMC6585427), possibly others but I don't know. $\endgroup$
    – Erwan
    Feb 6, 2020 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.