2
$\begingroup$

I have a list of sentences. Examples:

  1. ${INS1}, Watch our latest webinar about flu vaccine
  2. Do you think patients would like to go up to 250 days without an attack?
  3. Watch our latest webinar about flu vaccine
  4. ??? See if more of your patients are ready for vaccine
  5. Important news for your invaccinated patients
  6. Important news for your inv?ccinated patients
  7. ...

I have around 30k of sentences, around 85% of these are sentences that considered as 'good'. By good I mean sentences with no strange characters and sequences of characters such as '${INS1}', '???', or '?' inside the word etc. Otherwise sentence is considered as 'bad'. I need to find 'good' patterns to be able to identify 'bad' sentences in the future and exclude them, as the list of sentences will become larger in the future and new 'bad' sentences might appear.

Is there any way to identify 'good' sentences using Regex, libraries in Python/R, or any other tool?

Thank you

$\endgroup$

2 Answers 2

5
$\begingroup$

There are many methods . But for starters you can try the below method. ref SO post here

import re
if re.search('[^a-zA-Z\s:]', "Watch our latest webinar about flu vaccine"):
     print("bad") 
else :
    print("good")

Later you can try sentiment analysis for positive and negative words.

from textblob import TextBlob
TextBlob("Watch our latest webinar about flu vaccine").sentiment

From here you can look at removing stop words and other techniques to clean up your word corpus.

$\endgroup$
0
$\begingroup$

This solution based on character examples specifically given in the question. If there are more characters that should be used to identify good and bad sentences then they should also be added in the RegEx mentioned below in code.

import re 

sents = [
    "${INS1}, Watch our latest webinar about flu vaccine",
    "Do you think patients would like to go up to 250 days without an attack?",
    "Watch our latest webinar about flu vaccine",
    "??? See if more of your patients are ready for vaccine",
    "Important news for your invaccinated patients",
    "Important news for your inv?ccinated patients"
]

good_sents = []
bad_sents = []

for i in sents:
    x = re.findall("[?}{$]", i)
    if(len(x) == 0):
        good_sents.append(i)

    else:
        bad_sents.append(i)

print(good_sents)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.