My dataframe has some free text fields named: {'title', 'description', 'location'}

I prepared this text column by: concatenating all into a new column, dropping numbers, dropping words less than 3 char, etc...

As final step in preparing, I removed Stop Words. So, probably at this moment the column only contains relevant keywords.

How can I convert these keywords sentences (several words per row, as space separated words, in just one string column) into dummy columns and get the 0-1 values when, per row, this column contains the keyword?

I was checking the CountVectorizer object, but I couldn't find any use case like this one. So, it is probably not the proper tool to use...


Let's say that your dataframe is called df and the column with your preprocessed text as text.

What you want to do, as you already thought, is to apply a CountVectorizer.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

count = CountVectorizer(binary = True)
bow = count.fit_transform(df['text'].values)

pd.DataFrame(bow.toarray(), columns = count.get_feature_names())

If you remove the binary parameter from the CountVectorizer you will get the actual frequency of the keyword and not just the appearance with 0-1.

| improve this answer | |
  • $\begingroup$ thanks Tasos, the binary parameter was the point i've missed. $\endgroup$ – Cristian C. Bittel Oct 4 '19 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.