convert keywords in one column into several dummy columns

Question

My dataframe has some free text fields named: {'title', 'description', 'location'}

I prepared this text column by: concatenating all into a new column, dropping numbers, dropping words less than 3 char, etc...

As final step in preparing, I removed Stop Words. So, probably at this moment the column only contains relevant keywords.

How can I convert these keywords sentences (several words per row, as space separated words, in just one string column) into dummy columns and get the 0-1 values when, per row, this column contains the keyword?

I was checking the CountVectorizer object, but I couldn't find any use case like this one. So, it is probably not the proper tool to use...

Answer 1

Let's say that your dataframe is called df and the column with your preprocessed text as text.

What you want to do, as you already thought, is to apply a CountVectorizer.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

count = CountVectorizer(binary = True)
bow = count.fit_transform(df['text'].values)

pd.DataFrame(bow.toarray(), columns = count.get_feature_names())

If you remove the binary parameter from the CountVectorizer you will get the actual frequency of the keyword and not just the appearance with 0-1.