5
$\begingroup$

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this.

My question is related to the sklearn's ColumnTransformer class. Considering 'description' corresponds to a text/string data column with 9508 rows. The following works as expected:

In [61]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), 'description')])

In [62]: X=transformer.fit_transform(df)

In [63]: X.shape
Out[63]: (9508, 5913)

However, the following (note the [] around 'description'):

In [64]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), ['description'])])

In [65]: X=transformer.fit_transform(df)

does not work as expected:

In [66]: X.shape
Out[66]: (1, 3)

Note that there is no issue parsing the list of column names for other transformers such as OneHotEncoder(). This is making it difficult to programmatically configure transformers for dataframes containing heterogenous data type columns.

One possibility is to configure transformer for each column individually even if multiple columns require the same transformer but I was wondering if there is better way to deal with this?

$\endgroup$
  • 1
    $\begingroup$ From the documentation CountVectorizer accepts "An iterable which yields either str, unicode or file objects", but when you pass a list of columns to ColumnTransformer, an actual pandas.DataFrame is fed to CountVectorizer.fit. So I guess you will have to copy the transformation for each column. $\endgroup$ – Ian Liu Rodrigues Jan 17 '19 at 14:36
1
$\begingroup$

See the section 6.1.4 in the documentation.

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("title"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["title"]).

| improve this answer | |
$\endgroup$
0
$\begingroup$

Use make_column_transformer and set remainder to 'passthrough' so all remaining columns that were not specified in transformers will be automatically passed through.

from sklearn.compose import make_column_transformer
...
df = ...

transformer = make_column_transformer((TfidfVectorizer(), ['text_column']),
                                      (OneHotEncoder(), ['categorical_column']), 
                                       remainder='passthrough')
X = transformer.fit_transform(df)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.