I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description.
# each item is a list of tokenized job_description tok = [nltk.word_tokenize(job.lower()) for job in job_desc] # ignore stop words, bullets, etc. And put it into one list from nltk.corpus import stopwords stop = stopwords.words('english') def clean_token(what_to_clean): cleaned_tok =  for lists in what_to_clean: for item in lists: if len(item)>2 and (item not in stop): cleaned_tok.append(item) return cleaned_tok
After cleaning job description I've found most frequent words using:
freq = nltk.FreqDist(clean_token(tok)) most_freq_words = freq.most_common(100)
[('data', 211), ('experience', 78), ('learning', 70), ('business', 65), ('team', 53), ('science', 51), ('machine', 48),.....
From here I only want to extract words like machine, python, C+, technical skills. How can I go about it?
Also you can see there is word "machine" showing up 48 times and I am not sure whether it is talking about machine learning how can I go about this, I know if I want to make predictions I could've used CountVectorizer and n-grams.