1
$\begingroup$

Given a dataset of messages which are labeled with 20 features, I want to predict the value of each feature for a new message.

Dataset example:

message      feature1 feature2 feature3 feature3 feature4 ...
'hi'         1        0        1        1        0        ...
'i am bussy' 0        0        0        0        1        ...
...          ...      ...      ...      ...      ...      ...

Split data into train & test to train the model:

from sklearn.model_selection import train_test_split

x= df.iloc[:,0:1].values
y = df.iloc[:,1:-1].values
train_x, test_x, train_y, test_y = train_test_split(x, y, random_state=42)

Now, my train_x is an array of text values (impossible to fit into a train model), how could I convert them to numeric vectors?

$\endgroup$
0
$\begingroup$

What you want to do is find a vector representation of those strings which are in your $X$ vector. Two such techniques are Bag-of-Words and $n$-grams.

Bag-of-Words (BoW)

This technique will build a dictionary with all the words that exist in your training set. Then we will build a vector with the count of each word in each instance. For example let's consider these three separate instances:

  • 'hi'
  • 'i am bussy'
  • 'how are you doing?'

Then we can see that the following "words" in this training set are: hi, i, am, bussy, how, are, you, doing. So the vector representation of the above strings would be:

  • [1, 0, 0, 0, 0, 0, 0, 0]
  • [0, 1, 1, 1, 0, 0, 0, 0]
  • [0, 0, 0, 0, 1, 1, 1, 1]

There are ways to make this technique more effective by removing tenses from verbs or plurality of words. This is called stemming and should be used with BoW.

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld" 3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld" 4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features. Then you can vectorize the results of the $n$-gram in the same way as BoW.

Term Frequency-Inverse Document Frequency (TF-IDF)

Both of the above techniques can be enhanced with TF-IDF. This removes words that appear too often and do not have much information about the string, for example: the, like, a, ab, is...

For a term $t$ in a string $s$, the weight for that term is given by

$W_{t,s} = TF_{t,s} log(\frac{N}{DF_t})$

where N is the number of strings in your corpus, $TF$ is the number of times the term appeared in the given instance string and $DF$ is the number of documents in which the term appears.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.