Checking if english sentences have impact catpured in them using NLP

Question

I am looking for high level approach to identify if a given sentence has impact captured in it. For example in below two examples, sentence 2 has impact captured.

Example 1

Sentence 1: I learnt speaking spanish this year.

Sentence 2: I learnt speaking spanish this year which helped expand my business to Mexico.

Example 2

Sentence 1 : I taught machine learning to my students this year.

Sentence 2 : I taught machine learning to my students earlier this year which resulted in increasing base salary of students by 20,000 Dollars.

My initial thought is below approach.

1. think of all the english words like helped, resulted etc.,
2. Apply these words as filters on a public domain data set to get sentences
3. Curate the sentences to see if they fall in the "Sentence 2" category.
4. Apply classification/modelling to categorize into sentences with impact and without impact

Question:

1. Is the above approach good or there are better approaches?
2. What is the best publicly available dateset to tackle this problem.

Any inputs are greatly appreciated. Thanks alot in advance!

Answer 1

The approach you propose is ok but it can be improved in my opinion:

1. Find a corpus (or several corpora), chosen to be representative of the kind data you expect to process eventually. Ideally it would be directly a sample of the target data.
2. Label all the sentences in the corpus or a random subset. This is because you need to preserve the distribution as much as possible, especially the proportion of positive/negative instances. If you start from the set of filtered sentences you'll have two problems:
   • a proportion of positive cases higher than in regular text, and this is likely to make your model over-predict positive cases;
   • sentences containing only the selected trigger words, and this will cause your model to predict as negative any sentence which doesn't have a trigger word (that's a problem since your list of trigger words can't be exhaustive).

Ideally you would manually label all your corpus, but that's probably not realistic. That's why you could try to use your idea of trigger words in a slightly different way in order to label all the instances efficiently:

1. Filter the sentences which contain any trigger word and label these; this is your initial training set
2. Use a semi-supervised approach to label the rest of the sentences (maybe you could consider active learning).