Best approach for text classification of phrases with little syntactic difference

Question

So I have the task of classifying sentences based on their level of 'change talk' shown. Change talk is a psychology term used in counseling sessions to express how much the client wants to change their behavior.

So let's say there are two classes: change talk; and non-change talk.

An example of change talk is: "I have to do this." or "I can achieve this."

An example of non-change talk is "I can't do this." or "I have no motivation."

My issue is, if I want to take a machine learning approach in classifying these sentences, which is the best approach? SVM's? I do not have a lot of training data. Also - all the tutorials I look at use sentences with obvious words that can easily be classified (e.g. "The baseball game is on tomorrow." -> SPORT, or "Donald Trump will make a TV announcement tomorrow." -> POLITICS).

I feel my data is harder to classify as it typically does not have keywords relating to each class.

Some guidance on how people would approach this task would be great.

Answer 1

Your problem as you said is a high level of syntax overlapping between your sentences. take a look at these two sentences: Work to live versus live to work. The earlier that you can allow yourself to enjoy other things in life, aside from your job while the latter means obtaining resources so that you can be a functional member of society, and to permit yourself a good lifestyle. They are very different semantically. So when you vectorizing those sentences with techniques like Bag of words or Cosin similarity will be useless as both sentences contain the same corpus. The other problem you are dealing with (based on examples you provided) is dealing with the short text which makes it difficult to be vectorized by other simple but efficient techniques like TF-IDF. so regardless of what classification you are going to use, the performance of the classification model won't be high and that's because the input to the model is not correct.

On the other hand, deep learning methods like RNN or Transformers which solve sequence-to-sequence tasks like yours with ease can be very helpful. Named Entity Recognition models are what you need and given that your data is very domain-specific, you need to train your own model using your data. I recommend the Spacy Python package. So once you have your model, you will have two entities, CHANGE TALK, and NON-CHANGE TALK. Then you can simply count how many of them you have in your paragraph. Of course, that's the simplest way of dealing with your problem. You can add more entities and then they will act as features by which you can train any classification models. Hope this helps.

Answer 2

Often it is not clear at beginning of a project how difficult a task is and which elements will have biggest impact. One approach is to setup a machine learning system to systematically evaluate options and empirically explore the problem.

First setup the simplest possible text classification pipeline where the raw text enters the pipeline and "change" / "not change" prediction comes out of the pipeline. Pick the most appropriate evaluation metrics for that binary classification task.

Then take a model comparison approach where each element in the pipeline is experimentally tested to see if it improves the evaluation metrics. It is most common to experiment with different text encoding methods (e.g., one-hot encoding, count-based, or embeddings) and different algorithms (e.g., naive Bayes, support vector machine (SVM), tree-based models, or neural networks).