I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-related skills despite of missing "machine learning" keyword. How do we search the data through its meaning and related keywords I wonder? I have checked many algorithms also Like LSA, LDA, LSI but cannot find a resource which gives the implementation of the above.
2 Answers
There are many possible options.
One option is to create a dictionary of related terms. Then look for documents that contain those related terms.
This can be done with built-in data structures like Python's dict
and pattern matching tools like regular expression (regex).
I can see two ways to do this:
- Rule based approach:
You can create a list of keywords such as following:
keywords = ['machine-learning', 'machine learning', 'AI', ...]
And then you can search through the documents that you have.
- BERT based approach:
Here is some base code that you might use:
from transformers import AutoTokenizer, AutoModel
import torch
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')
def encode(text):
inputs = tokenizer.encode_plus(text, return_tensors='pt', max_length=512, truncation=True)
outputs = model(**inputs)
return outputs.last_hidden_state[:, 0, :].detach().numpy()
resumes = [...] # list of resumes
resume_vectors = [encode(resume) for resume in resumes]
query_vector = encode("machine learning")
from sklearn.metrics.pairwise import cosine_similarity
similarities = [cosine_similarity(query_vector, resume_vector) for resume_vector in resume_vectors]
top_resumes = sorted(zip(resumes, similarities), key=lambda x: x[1], reverse=True)
You can use this approach to find similar words from the resumes using cosine similarity
.