# Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words. A dedicated function, returning a tuple, was expected to memorise the value of the innermost objects in the two additional dataframe, but so far I've been failing in my attempts.

Below what I've been doing. So long I do something like the below, everything is good data["unigrams"] = data["Keywords"].apply(nltk.word_tokenize)

However, the resulting dataframe still contains the stopwords.

I therefore written a small function that manipulate the value and return a tuple, but I then fail to assign to the dataframe with the following error: TypeError: expected string or bytes-like object

The assignment I do, is something like this. I couldn't get any other syntax working, but this is not what I'm willing to figure out now.

data = data.assign(Tokenized = lambda x: doIt(x['Keywords']),
Filtered = lambda y: doIt(x['Keywords']))


The doIt function code is:

def doIt(keyword):
filtered = []
tokenized = nltk.word_tokenize(keyword)

for w in tokenized:
if w not in stop_words:
filtered.append(w)



When recalled individually the doIt("this is my string") code works fine, though, returning the following tuple: (['this', 'is', 'my', 'string'], ['string'])

In an attempt to debug the function, I then realised that something but be connected with the length of the inner object. In fact, a call like this

vals = tokenAndRemoveStopWords("this is my string")
dfObj = pd.DataFrame(vals, columns=['A', 'B'])


returns the following error: ValueError: 2 columns passed, passed data had 4 columns

As far as I understand then, the problem seems to be with the inner object that is later unpacked to be fitted in the dataframe that is evidently smaller. However, my attempt was to get the first object in the tuple and memorised as a list; the same for the second.

Not sure whether I was able to explain the concept properly though, but I'm in my infancy with this Data Science bits.

Thanks Andrea