Columntransformer multiple columns with vector inputs

Question

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this.

My question is related to the sklearn's ColumnTransformer class. Considering 'description' corresponds to a text/string data column with 9508 rows. The following works as expected:

In [61]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), 'description')])

In [62]: X=transformer.fit_transform(df)

In [63]: X.shape
Out[63]: (9508, 5913)

However, the following (note the [] around 'description'):

In [64]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), ['description'])])

In [65]: X=transformer.fit_transform(df)

does not work as expected:

In [66]: X.shape
Out[66]: (1, 3)

Note that there is no issue parsing the list of column names for other transformers such as OneHotEncoder(). This is making it difficult to programmatically configure transformers for dataframes containing heterogenous data type columns.

One possibility is to configure transformer for each column individually even if multiple columns require the same transformer but I was wondering if there is better way to deal with this?

Answer 1

See the section 6.1.4 in the documentation.

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("title"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["title"]).

Answer 2

Use make_column_transformer and set remainder to 'passthrough' so all remaining columns that were not specified in transformers will be automatically passed through.

from sklearn.compose import make_column_transformer
...
df = ...

transformer = make_column_transformer((TfidfVectorizer(), ['text_column']),
                                      (OneHotEncoder(), ['categorical_column']), 
                                       remainder='passthrough')
X = transformer.fit_transform(df)