0
$\begingroup$

I have a corpus of documents whose some lines have information like this:

wt 210 1b 14.4 oz (98 kg)

or

weight: 219 lb (99 kg), height: 5' 1.9" (157 cm)

The format of occurrence of such information varies from document to document. I need the value or the substring corresponding to weight and weight only. Here are my questions regarding the problem:

  1. I have certain regexes that can get the weight value for labeling the lines. However, I do not know how to provide a string as the y-axis, should I convert it to TFIDF vector? Won't that make y-axis hyper-dimensional?

  2. My first intuition is to use Extractive summarizer trained on many other such lines. Is there a better way to handle that?

Thank you.

$\endgroup$
3
  • 1
    $\begingroup$ Welcome to DataScience. Sorry your question is not clear to me: what is the issue with your current solution using regular expressions? This looks like a good solution. And what is the y-axis? I doubt using document summarization makes sense for a case like this. You might want to train a NER model but even this looks like overkill to me. $\endgroup$ – Erwan Jan 19 at 22:26
  • $\begingroup$ Regexes are quite rigid, and the incoming documents will be not exactly in the a couple dozen format of regexes we have on our system. Y-axis is the substring of weight to train the model on. Its generated using the regexes we have. We are hoping to build a model that will be fuzzy enough to extract the value of varying templates/formats without the need to hardcode every variant using regex. $\endgroup$ – abhishah901 Jan 20 at 22:13
  • $\begingroup$ Ok then you should look into Named Entity Recognition, that would be the standard task for extracting specific parts of a text based not only on the target but also the context. You'll need labelled data in order to train a model. $\endgroup$ – Erwan Jan 20 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.