Extract Pattern using Short Text Processing

Question

We are facing issue in our project. We have a data set of around 25000 rows, we have a column name title, it contains text data and we have a score column in the data set.We want to use Machine Learning techniques to know what are the factors that make the label High, by factor i mean what are the key words etc.., as use can see we have a short text so what is the most proper technique to extract the maximum knowledge related to the label from this short text,I did some pre processing on the text by cleaning it then extracting ngram features using tf-idf weighting function, then i tried using decision trees algorithm for classification of text For example:

   Score Label                       Tactic Title
1.  High                        Opportunity Movement.   
2.  Low                         Partner Launch.  
3.  High                        Implement Mix panel stories.
4.  Low                         Improve app performance and reduce multiple API Calls.
5.  High                        Review Comments.
6.  Low                         Support SimpleStarta Team Goal. 

Answer 1

Problem 1: "Tactic Titles" Are Really TEXT

i.e. they are not standard "tags". I say this because they seem so structured and tag-like. For instance in text analysis using NLP techniques you mostly deal with texts in which "Review Comments" and "Reviewing Comments" and "Comment Review" are supposed to be considered the same.

Actually the way you already tried is a valid way to do it.

Another idea could be using information theoretic approaches. It means using something like Tf-IDF approach but this time taking classes into account. Here you are going to look for words/n-grams which appear a lot in desired class (High) and not a lot in the other class. A simple formulation could be:

$$Score_{w_i} = \frac{N_{high}}{N_{low}+1}$$ where $N_{high}$ is the number of times the word $w_i$ appeared in class High and $N_{low}$ is the number of times the word $w_i$ appeared in class Low. The constant $1$ is just a smoother to avoid a zero denominator.

If you use it be careful about overfitting as you are learning features in a very implicit and hand-crafted way. To avoid overfitting you may prune your found words based on a min_count or max_count limitation.

Of course this is a simple score and you can improve it by modification (e.g. i did not consider the normalization which is better to be considered, etc.)

For prediction you can also simply use naive Bayes. Count words/phrases based on Bayesian approach (just count the frequency of words happening in each class) and normalize them to probabilities. Now you have most likely data given classes and the probability of classes. A new data comes and $P(C|D)=P(D|C)\times P(C)$ is calculated easily.

Problem 2: The "Tactic Titles" Are Just Fixed Tags (Maybe irrelevant to score)

Means that semantics are not correlated with classes i.e. two titles with similar terms or meanings might be in two different classes.

Then it's not a learning problem. Either you have the label of the new data in the dictionary so you just extract the class, or you don't have it so you don't know about it!

---

How would you want to know that "Partner Launch" is low priority but "Review Comments" is high if you have no meta-information form the organization behind it? How can you predict that based on this most probably "Write Comments" is High or Low? .If you argue that Machine Learning is supposed to learn that latent variable behind, then I would say yes! but please note that maybe that latent phenomenon is not correlated with Text. For instance if you get meta-data about which department is doing any of this tasks, you may end up with a more correlated/causal set of features. The long story short: Maybe the information is not in TEXT but some other aspects.

Answer 2

Have you tried the boring, straightforward approach? Get a list of all words and count how often they occur with a high or a low label. (Exclude words that occur only once or twice, and also the words that occur very often).

For example:

      Score Label     Tactic Title
 1.   High            Build Batmobile
 2.   Low             Repair Batcave
 3.   High            Paint Batmobile
 4.   Low             Paint Batcave

The words that occur more than once are Paint, Batmobile, Batcave. Batmobile has 100% High labels, Paint has 50% and Batcave has 0%. You get the idea.