1
$\begingroup$

I've tried to reduce the problem to it's absolute basics. Assume I have data (csv) as such:

label,text-column,gender-column,day-column
1,"Sample positive text", female, 1
0,"Sample negative text", female, 3
1,"Another positive comment", male, 2
0,"Angry text sample", male, 7

And I have this code that trains on label by using BoW (in this case tf-idf) on the text-column. I do a 70/30 train_test_split and all works well.

vec = TfidfVectorizer()
clf = MultinomialNB()
training_data = pd.read_csv('trainset.csv', delimiter=',')
text_tfidf = vec.fit_transform(training_data['text-column'])
# gen_tfidf = vec.fit_transform(training_data['gender-column'])
X_train, X_test, y_train, y_test = train_test_split(text_tfidf, training_data['label'], test_size = 0.3)
clf.fit(X_train, y_train)

However, for the life of me I simply cannot figure out how to use more than one feature. E.g. I want to use, say, both text-column and gender-column to train the model and see how that impacts accuracy, but I don't understand how to do that!

Am I missing something conceptually important here? Thank you.

Improve this question
$\endgroup$
1
  • $\begingroup$ What is the point of using the vectorizer on the gender column? Usually we do that to tranform the tokenized text into vectors. Why don't you just create a big matrix X where the first columns are the vectorized text, then comes the gender column (using sk-learn categorical encoder), and then the day column. $\endgroup$
    – An old man in the sea.
    Nov 16, 2021 at 15:29

2 Answers 2

Reset to default
1
$\begingroup$

You can one-hot encode the gender column and append it to your tfidf table.

gender = pd.get_dummies(training_data['gender-column'])
X = text_tfidf.join(gender)
Improve this answer
$\endgroup$
1
$\begingroup$

Scikit-learn has ColumnTransformer for heterogeneous data.

Here is a rough code snippet to get you started:

from sklearn.compose                 import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes             import MultinomialNB
from sklearn.pipeline                import Pipeline
from sklearn.preprocessing           import OneHotEncoder

pipeline = Pipeline([('union', ColumnTransformer([('tfidf',  TfidfVectorizer(), 'text-column'),
                                                  ('onehot', OneHotEncoder(),   ['gender-column'])])),   
                     ('clf', MultinomialNB())])
pipeline.fit(X_train, y_train)
Improve this answer
$\endgroup$
2
  • $\begingroup$ OneHotEncoder requires 2d data, and so the ColumnTransformer needs to specify the column in a list: ['gender-column']. $\endgroup$
    – Ben Reiniger ♦
    Nov 15, 2021 at 23:27
  • $\begingroup$ Thanks for pointing that out. I have updated the code snippet. $\endgroup$
    – Brian Spiering
    Nov 16, 2021 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.