Given a sentence I am trying to classify if the sentence contain information about education. For example:

sentence1 = "Require minimum four years of professional degree." 
sentence2 = "no degree required for this job."

For identifying as a first step I have built a vocab which have the set of keywords for identifying education based sentences. I was partly successful until I have problems with sentences like this.

sentence3 = "BE or BTech or any degree equivalent to it"

In my vocab 'BE' is also a keyword as 'BE' represents bachelors of engineering degree (in case of country India). As the parsing of algorithm is done in lower case, the issue is coming with 'BE' will be 'be' in sentence. Because of that every sentence that contains 'be' is recognized as educational sentence.

To built a strong ML model I dont have much data. If I want to use vocab based recognition, for this I have to understand the words beside them in the sentence.

Are there any built models so as to import and identify such scenarios or labelled dataset available for it ? Are there any methods for accomplishing such task ?

  • $\begingroup$ This is naive but have you tried restricting 'BE' to be capital letters? $\endgroup$ – Danny Sep 9 '19 at 10:07
  • $\begingroup$ I cannot restrict capital letters, because there is some other process before my algorithm which gives text in small letters. so as my vocab. $\endgroup$ – Raady Sep 9 '19 at 10:14

I think what you are looking is to differentiate between 'be' and 'BE' based on context. Word2Vec is a good place to start but to determine the difference between words based on contexts is 'Sense2Vec.' Word Sense Disambiguation paper is worth to look at. I am not sure if this is what you're looking for because if you have 'be' in every sentence which is most likely, then this won't be an ideal solution. You might have to do some extra data preprocessing like changing 'be' to 'Bachelor of Engineering' based on your sentence.

|improve this answer|||||
  • $\begingroup$ In my case the text that I receive is already in lower case. I will try with sense2vec $\endgroup$ – Raady Sep 9 '19 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.