I'm trying to parse some text, and extract data from it. Typical NLP problem.

However the text contains different sections, and I know that the keywords of interest are in specific sections, but all the other sections add a lot of noise to the problem, so I am looking for a way to first isolate the sections of interest.

Let's say we're talking about a cooking recipe (it's not what I'd doing but close enough): there is the description section, the ingredient section, the step-by-step section, and the summary section, and let's say I only care about identifying the ingredients.

One piece of insight is that the sections tend to have different wording, or structure. Like ingredients may be a list, not verbose at all, no verbs, or could include some verb forms to indicate requirement (should, must, require...) while the other sections won't.

I'm leaning towards using Doc2Vec on sentences, and group nearby sentences with high similarity in text structure, but I'm not sure what features to use.

I tried using lemma, as well as dep generated from spaCy, but when I compare a sentence to the next in order, there is 97-99% match most of the time, and no clear separation of sections (which I'd expect from a drop in similarity between the last sentence of one section, and the first of the next one.

I also have a list of 'ingredients' generated from another source. I have tried labelling some data by checking if the 'ingredient' is in the sentence, and then train a classifier with the words, and also with the lemma and dep. Not much luck there.

I tried training a RNN on those. One issue may be the list of 'ingredients' is not great to begin with because some words are ambiguous, and should not be used as an 'ingredient' in some contexts, generating false positives in the labeled training set.

Maybe I need to consider the different cases separately (bullet points type vs. more verbose) but I don't have clean labeled data.

One might say, just spend 2 weeks labelling data... i might get to that.

So the question is, if not too broad, what are good methods to identify paragraphs of similar text within a document, with unsupervised learning?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.