So I have a column called "plot" in a dataframe and i want to create a new one called "keywords" which only has the important words of plot. here is the code:

 import pandas as pd
 import numpy as np
 from sklearn.metrics.pairwise import cosine_similarity 
 from sklearn.feature_extraction.text import CountVectorizer
 import re  
 import nltk
 from nltk.corpus import stopwords 
 df = pd.read_csv('IMDB_Top250Engmovies2_OMDB_Detailed.csv')
 df = df[['Title','Genre','Director','Actors','Plot']]
 df['Keywords'] = ''

 for index,row in df.iterrows():
     plot = row['Plot']
     plot = re.sub('[^a-zA-Z]'," ", plot)
     plot = plot.lower()
     plot = plot.split()
     plot = [i for i in plot if not i in set(stopwords.words('english'))]
     plot = ' '.join(plot)                                          
     row['Key_words'] = str(plot)

And here is the output :(

enter image description here

Link to the csv : https://query.data.world/s/uikepcpffyo2nhig52xxeevdialfl7

Thank you !

  • 1
    $\begingroup$ Please, please. Avoid to include code in images. It wouldn't be possible for anyone to help you if he/she cannot copy paste your code to run it locally. You can edit your question and format the post with the original code. $\endgroup$
    – Tasos
    Apr 2, 2019 at 8:33
  • $\begingroup$ Also: It's recommended that you clearly state what doesn't work. I kinda pieced together that you'd like your column to contain things. Welcome btw. $\endgroup$ Apr 2, 2019 at 10:05

2 Answers 2


It could be something like this

create function here:

def important_words(plot):
    # your code here
    return plot

make use of apply function:

df["Keywords"] = df.Plot.apply(lambda x: important_words(x))

Iterrow passes a copy of the row, not the reference. This should fix your problem:

df.loc[index,'Keywords'] = str(plot)

However, I would recommend using apply, imho it is more elegant. And it is alot faster.

That would looks something like this

def string_to_keywords(string):
    plot = re.sub('[^a-zA-Z]'," ", string)
    plot = plot.lower()
    plot = plot.split()
    return " ".join([i for i in plot if not i in set(stopwords.words('english'))])

df["Keywords"] = df["Plot"].apply(string_to_keywords)
  • $\begingroup$ BTW: Extracting everything but the stopwords is a nice starting point. Once you get this to work, you might want to look into tfidf or attention, if you want to get sophisticated about it. $\endgroup$ Apr 2, 2019 at 10:08
  • $\begingroup$ Ummm how to use appy here ? $\endgroup$ Apr 2, 2019 at 10:59
  • $\begingroup$ I added that to the answer Abhinav. $\endgroup$ Apr 2, 2019 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.