1
$\begingroup$

from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found:

freq = nltk.FreqDist(lemmatized_list)
most_freq_words = freq.most_common(100)

which outputs:

[('data', 179),
 ('experience', 86),
 ('work', 78),
 ('business', 71),
 ('team', 59),
 ('learn', 56),
 ('model', 49),
 ('skills', 47),
 ('science', 41),
 ('use', 41),
 ('build', 39),
 ('machine', 37),
 ('ability', 36),.....

and so on. My problem is I do not want to consider words like "experience", "work", and only consider keywords related to data science. I'm guessing there is a corpus for data science terms which I can use like how I use stop word corpus to not select them. Let me know if there is a way, Thanks!

$\endgroup$
  • 1
    $\begingroup$ If you have access to many non-datascience job postings too, you can use some sort of tfidf to down-weight words common in everything. $\endgroup$ – Andy M Apr 27 at 15:26
  • $\begingroup$ But job descriptions on same job will have many occurences of technical skill words in each job description and that would be down-weighted. $\endgroup$ – h_musk Apr 27 at 15:55
0
$\begingroup$

I have a way through which you can solve your problem. For it you will require a,

  • A pretrained embedding generator. It can be Word2Vec or GloVe. Any of them could work.

Next, we have a corpus of words which have higher frequencies. Suppose we have a set of 100 such words where the 1st word has the highest frequency.

Now, we convert every word in this set to a vector using our pretrained word embedding. Hence you will have a set of vectors for the words from the corpus. Let's call it $z_i$

We have the word "data science". Get a vector for this too. Let's call it $x$

  1. Measure the euclidean distance between the vector $x$ and $z_i$.
  2. Or, you can measure cosine similarity betwee $x$ and $z_i$.
  3. Both of above methods will produce a set of values which would show proximity of $x$ with values of $z_i$.
  4. From these 100 values, we get the least 10 values and convert them to words again.

These 10 words would have the highest similarity with the word "data science".

$\endgroup$
0
$\begingroup$

Overall, I agree with Andy M's suggestion.

To address the issue you point out and get rid of words work and experience, you can probably ignore the n-most-frequent words in the general corpus that also appear in the data science corpus, and keep the rest as the data-science-related terms.

So, in a more pythonic way:

general_texts = [
    ['a', 'sentence', 'about', 'experience'],
    ['another', 'sentence', 'typed', 'at', 'work'],
    ['work', 'experience'],
    ...
]


data_science_texts = [
    ['data', 'science', 'experience'],
    ['work', 'on', 'machine', 'learning'],
    ...
]

freqdist_gnrl = Counter()
freqdist_ds = Counter()

for text in general_texts:
    freqdist_gnrl.update(text)

for text in data_science_texts:
    freqdist_ds.update(text)

mostfreq_words_gnrl = freqdist_gnrl.most_common(2)   # 'work', 'experience'

words_ds = [
    w for w, _ in freqdist_ds.most_common()
    if w not in mostfreq_words_gnrl        # every word other than 'work' or 'experience'
]

In this example, I have used 2 as n for the n-most frequent terms to make it work but, over a larger corpus, you can probably take a few hundred words.

After applying this filter, the first k words in the variable words_ds should all be related to data science to a reasonable extent.

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.