2
$\begingroup$

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-related skills despite of missing "machine learning" keyword. How do we search the data through its meaning and related keywords I wonder? I have checked many algorithms also Like LSA, LDA, LSI but cannot find a resource which gives the implementation of the above.

$\endgroup$

2 Answers 2

0
$\begingroup$

There are many possible options.

One option is to create a dictionary of related terms. Then look for documents that contain those related terms.

This can be done with built-in data structures like Python's dict and pattern matching tools like regular expression (regex).

$\endgroup$
0
$\begingroup$

I can see two ways to do this:

  1. Rule based approach:

You can create a list of keywords such as following:

keywords = ['machine-learning', 'machine learning', 'AI', ...]

And then you can search through the documents that you have.

  1. BERT based approach:

Here is some base code that you might use:

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')
def encode(text):
    inputs = tokenizer.encode_plus(text, return_tensors='pt', max_length=512, truncation=True)
    outputs = model(**inputs)
    return outputs.last_hidden_state[:, 0, :].detach().numpy()

resumes = [...]  # list of resumes
resume_vectors = [encode(resume) for resume in resumes]
query_vector = encode("machine learning")
from sklearn.metrics.pairwise import cosine_similarity

similarities = [cosine_similarity(query_vector, resume_vector) for resume_vector in resume_vectors]
top_resumes = sorted(zip(resumes, similarities), key=lambda x: x[1], reverse=True)

You can use this approach to find similar words from the resumes using cosine similarity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.