Multi-label classification for text messages (convert text to numeric vector)

Question

Given a dataset of messages which are labeled with 20 features, I want to predict the value of each feature for a new message.

Dataset example:

message      feature1 feature2 feature3 feature3 feature4 ...
'hi'         1        0        1        1        0        ...
'i am bussy' 0        0        0        0        1        ...
...          ...      ...      ...      ...      ...      ...

Split data into train & test to train the model:

from sklearn.model_selection import train_test_split

x= df.iloc[:,0:1].values
y = df.iloc[:,1:-1].values
train_x, test_x, train_y, test_y = train_test_split(x, y, random_state=42)

Now, my train_x is an array of text values (impossible to fit into a train model), how could I convert them to numeric vectors?

Answer 1

What you want to do is find a vector representation of those strings which are in your $X$ vector. Two such techniques are Bag-of-Words and $n$-grams.

Bag-of-Words (BoW)

This technique will build a dictionary with all the words that exist in your training set. Then we will build a vector with the count of each word in each instance. For example let's consider these three separate instances:

• 'hi'
• 'i am bussy'
• 'how are you doing?'

Then we can see that the following "words" in this training set are: hi, i, am, bussy, how, are, you, doing. So the vector representation of the above strings would be:

• [1, 0, 0, 0, 0, 0, 0, 0]
• [0, 1, 1, 1, 0, 0, 0, 0]
• [0, 0, 0, 0, 1, 1, 1, 1]

There are ways to make this technique more effective by removing tenses from verbs or plurality of words. This is called stemming and should be used with BoW.

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld" 3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld" 4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features. Then you can vectorize the results of the $n$-gram in the same way as BoW.

Term Frequency-Inverse Document Frequency (TF-IDF)

Both of the above techniques can be enhanced with TF-IDF. This removes words that appear too often and do not have much information about the string, for example: the, like, a, ab, is...

For a term $t$ in a string $s$, the weight for that term is given by

$W_{t,s} = TF_{t,s} log(\frac{N}{DF_t})$

where N is the number of strings in your corpus, $TF$ is the number of times the term appeared in the given instance string and $DF$ is the number of documents in which the term appears.