I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description.

# each item is a list of tokenized job_description
tok = [nltk.word_tokenize(job.lower()) for job in job_desc]

# ignore stop words, bullets, etc. And put it into one list
from nltk.corpus import stopwords
stop = stopwords.words('english')

def clean_token(what_to_clean):
    cleaned_tok = []
    for lists in what_to_clean:
        for item in lists:
            if len(item)>2 and (item not in stop):
    return cleaned_tok

After cleaning job description I've found most frequent words using:

freq = nltk.FreqDist(clean_token(tok))
most_freq_words = freq.most_common(100)

Which outputs:

 [('data', 211),
 ('experience', 78),
 ('learning', 70),
 ('business', 65),
 ('team', 53),
 ('science', 51),
 ('machine', 48),.....

From here I only want to extract words like machine, python, C+, technical skills. How can I go about it?

Also you can see there is word "machine" showing up 48 times and I am not sure whether it is talking about machine learning how can I go about this, I know if I want to make predictions I could've used CountVectorizer and n-grams.


1 Answer 1


From what I understand, you need more than 1-word terms. Thus, it's better to go for n-grams.

Start with creating n-grams

import re
from nltk.util import ngrams

s = s.lower()
s = re.sub(r'[^a-zA-Z0-9\s]', ' ', s)
tokens = [token for token in s.split(" ") if token != ""]
output = list(ngrams(tokens, 2))

You should get 1-gram, 2-grams and 3-grams at least to cover all possible terms.

Then you can use a Computer Science Ontology (CSO) which includes about 14k technical terms.

An alternative would be to use their own Classifier that gets a text as input and annotate terms on it from the same ontology. You can find a demo here https://cso.kmi.open.ac.uk/classify/

When I try to classify the abstract of Machine Learning from the Wikipedia article, I get the following results

Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to effectively perform a specific task without using explicit instructions, relying on patterns and inference instead. It is seen as a subset of artificial intelligence. Machine learning algorithms build a mathematical model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to perform the task.1:2 Machine learning algorithms are used in a wide variety of applications, such as email filtering, and computer vision, where it is infeasible to develop an algorithm of specific instructions for performing the task. Machine learning is closely related to computational statistics, which focuses on making predictions using computers. The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning. Data mining is a field of study within machine learning, and focuses on exploratory data analysis through unsupervised learning.[3][4] In its application across business problems, machine learning is also referred to as predictive analytics.

  • artificial intelligence
  • computer systems
  • computer vision
  • correlation
  • analysis
  • data mining
  • email
  • inference
  • machine learning
  • optimization
  • unsupervised learning
  • $\begingroup$ when using n-grams does it consider both 2 word only, when there is a word "machine learning" does it take only machine learning as two words for "machine" and "learning" separately as well so consider all 3 possibilities? $\endgroup$
    – haneulkim
    Commented Apr 24, 2019 at 22:40
  • $\begingroup$ When you use 2-grams it doesn’t care about one word terms. So there isn’t “machine” on 2-grams. Just “machine learning”. To cover all the cases, calculate all 1-gram, 2-grams and 3-grams. If you believe you need to capture concepts with 4 words, go also for 4-grams. $\endgroup$
    – Tasos
    Commented Apr 25, 2019 at 5:59
  • $\begingroup$ oh I see. Thanks for the help! $\endgroup$
    – haneulkim
    Commented Apr 26, 2019 at 1:27
  • $\begingroup$ Hey, can we use CSO like how we use stopwords in nltk?? $\endgroup$
    – haneulkim
    Commented Apr 27, 2019 at 2:59
  • $\begingroup$ It’s the same idea. But I don’t think there is an implementation out of the box $\endgroup$
    – Tasos
    Commented Apr 27, 2019 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.