Using TF-IDF with other features in scikit-learn

Question

What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. scikit-learn's TF-IDF vectorizer transforms text data into sparse matrices. I can use these sparse matrices directly with a Naive Bayes classifier for example. But what's the way to also take into account the other features? Should I de-sparsify the tf-idf representation of the text and combine the features and the text into one DataFrame? Or can I keep the sparse matrix as a separate column for example? What's the correct way to do this?

Answer 1

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.

Answer 2

Usually, if possible, you'd want to keep your matrice sparse as long as possible as it saves a lot of memory. That's why there are sparse matrices after all, otherwise, why bother? So, even if your classifier requires you to use dense input, you might want to keep the TFIDF features as sparse, and add the other features to them in a sparse format. And then only, make the matrix dense.

To do that, you could use scipy.sparse.hstack. It combines two sparse matrices together by column. scipy.sparse.vstack also exists. And of course, scipy also has the non-sparse version scipy.hstack and scipy.vstack

Answer 3

Imagine you have a dataframe of four feature columns and a target. Two of the features are text columns that you want to perform tfidf on and the other two are standard columns you want to use as features in a RandomForest classifier.

I would use the following code:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction.text import TfidfVectorizer

# Set X and y
X = df[['text1_column_name', 
        'text2_column_name', 
        'standard_feature1', 
        'standard_feature2']]
y = df['target']

# initialise model and vectorizers
model = RandomForestClassifier()
vectorizer1 = TfidfVectorizer()
vectorizer2 = TfidfVectorizer()

# construct the column transfomer
column_transformer = ColumnTransformer(
    [('tfidf1', vectorizer1, 'text1_column_name'), 
    ('tfidf2', vectorizer2, 'text2_column_name')],
    remainder='passthrough')

# fit the model
pipe = Pipeline([
                  ('tfidf', column_transformer),
                  ('classify', model)
                ])
pipe.fit(X,y)