1
$\begingroup$

I am currently working on an use case where feature set contains numeric values such as amount, as well as a review feature which contains long winded english text. the english text will very well differ between train and test data. eg 'i have seen and its good' , 'nto ok','timepass',etc

how do i combine the text feature set with numerical data and feed it to a machine learning model?

i will nt be able to use encoding , these text variables are not categorical values . they are varying .

import pandas as panda
from sklearn.feature_extraction.text import TfidfVectorizer

words = ['i hv paid','i dont like','its good','yum yum']

a = panda.DataFrame({'amount':[10,20,30,40],'word':words})

tf = TfidfVectorizer()

csr = tf.fit_transform(words)

#how do i now use my csr to feed both amount and word to my machine learning model
| improve this question | |
$\endgroup$
2
$\begingroup$

One of the ways to address your use case could be to create 2 separate models, one model using your text data features and another one using your numerical features and combine their results using ensembling.

The other way could be to create numeric features out of your text features (e.g. tf-idf, word2vec) and combine them with your numeric features and feed them to your model.

Sample code-

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
import pandas as pd

words = ['i hv paid','i dont like','its good','yum yum']
tf = TfidfVectorizer()
vector = tf.fit_transform(words)
df = pd.DataFrame([10,20,30,40])
df2 = pd.DataFrame(vector.toarray())
pd.concat([df,df2], axis=1)

Sample output- enter image description here

| improve this answer | |
$\endgroup$
  • $\begingroup$ so just feed the tf ids matrix to classifier c1. and feed remaining n-1 numeric features to classifier c2. and then average out probablities to get the final clssification? did i understand this correct? $\endgroup$ – user1906450 Jan 22 '19 at 18:19
  • $\begingroup$ are they no other ways to combine text+numeric features in a machine learning model? $\endgroup$ – user1906450 Jan 22 '19 at 18:19
  • $\begingroup$ Yes that's correct. The other way could be to create numeric features out of your text features (e.g. tf-idf, word2vec) and combine them with your numeric features and feed them to your model. $\endgroup$ – Amit Rastogi Jan 22 '19 at 18:31
  • $\begingroup$ hi @amit ..thats actually my question..how do i combine the two..would you mind showing an example.. $\endgroup$ – user1906450 Jan 22 '19 at 19:00
  • $\begingroup$ hi @user1906450 I have edited my answer to include the sample code. $\endgroup$ – Amit Rastogi Jan 23 '19 at 17:54
0
$\begingroup$

You start with Vector Space Models such as Bag of Words. Different variations of bag of words can be used here such as Count Vectorizer or TF-IDF. The other approach which has gained a great popularity during recent years are Word Embeddings such as Word2vec.

As your question is pretty basic, I stop here. In case you need more help please provide more details and I can provide you a simple classification model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ hi thank you for your response. yes i have used tf idf across each row..and i am getting a sparse matrix for each. now this is just one field. i have another field called amount which is numeric data. now how do i combine the tfidf sparse matrix with the numeric field .how do i now feed this to my machine learning. let me edit my answer to reflect a quick example $\endgroup$ – user1906450 Jan 22 '19 at 16:41
  • $\begingroup$ TF-IDF is done on whole corpus and gives you a matrix whose columns are words of the vocabulary and rows are document. Each element is the TFIDF value for a specific pair of (doc,word). Then you come up with that sparse numpy matrix and you can add the other feature to it (stackoverflow.com/questions/41937786/…) But there are some things: $\endgroup$ – Kasra Manshaei Jan 22 '19 at 16:49
  • $\begingroup$ 1) TFIDF is a very high-dimensional model. Do you think adding a single column helps a lot? I am not sure. 2) I am not sure if the word count is necessarily a good feature. You may try its classifying ability by using a simple classifier only on this feature. 3) At least try to set the "max_features" parameter of TFIDF (and also min_df, etc) to prune dimensionality of TFIDF. $\endgroup$ – Kasra Manshaei Jan 22 '19 at 16:51
  • $\begingroup$ hi, i just gave a very simple example. my dataset contains close to 9 numeric features and 3 such text fields. from domain knowledge i can vouch that i only need one of those text fields but yes it is absolutely essential to take the text field as a feature. how do i now run my machine learning models? $\endgroup$ – user1906450 Jan 22 '19 at 16:54
0
$\begingroup$

It's important to don't change the main type of csr

from scipy.sparse import csr_matrix, hstack
csr = hstack([csr, csr_matrix(a.amount).T], 'csr')
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.