2
$\begingroup$

I'm trying to build a model to map job descriptions to ESCO occupations which is a taxonomy for job titles. Every ESCO occupations have a title, a description and some essential skills. Ideally I would have built a classification model but since I don't have labelled data that's out of the question.

So my idea was to generate text embeddings from every ESCO occupation and then for an input job description, and using cosine similarity, find the most similar ESCO occupation to that job description. I'm using this model to generate the embeddings, which is an XLM-roBERTa which was pre-trained on job market data. I use the mean of the embeddings for every token as the job description's final embedding. However the results are very bad, it fails to find most relevant ESCO occupations.

Here's how I compute the embeddings:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("jjzha/esco-xlm-roberta-large")
model = AutoModel.from_pretrained("jjzha/esco-xlm-roberta-large")

sample_job_title = "We are looking for a junior software developer with experience in React and Python."
encoded_input = tokenizer(sample_job_title, padding=True, truncation=True, return_tensors="pt")
with torch.inference_mode():
    output = model(**encoded_input)
embedding = output.last_hidden_state.mean(dim=1)

From this output I retrieve output.last_hidden_state, which should correspond to the tokens embeddings, and then compute the mean embedding. This returns a pytorch tensor of shape (1, 1024). I have another tensor of shape (3007, 1024), esco_embeddings which corresponds to the embeddings for every 3007 ESCO occupation. I then compute cosine similarity by doing:

similarities = torch.nn.functional.cosine_similarity(embedding, esco_embeddings, dim=1)

And find the k most similar ESCO occupations by computing

most_similar = torch.topk(similarities, k)

I thought that the problem might be in the embeddings itself, with XLM-roBERTa generating embeddings for every token and not one emebdding for the whole text.

Does anyone have an idea why that isn't working, and how it could be fixed? Maybe there's a better approach?

Thanks for the help.

Improve this question
$\endgroup$
5
  • $\begingroup$ Please provide some info on your computations, like how do the embeddings loom like, how do you compute similarity etc. $\endgroup$
    – picky_porpoise
    Commented Apr 19 at 18:38
  • $\begingroup$ Make sure you're not including padding tokens when you compute the mean embedding. Do you have a paired dataset of (description, title) pairs? Best way to do this would be to use a small MLP to predict title embedding from description embedding. $\endgroup$
    – Karl
    Commented Apr 20 at 3:26
  • $\begingroup$ @picky_porpoise I just updated my post to provide the info $\endgroup$
    – GanaelD
    Commented Apr 22 at 8:13
  • $\begingroup$ @Karl if I'm not mistaken, there are no padding tokens left in the embedding? Aren't padding tokens only there in the tokenizer output? Also, I have a paired dataset of (description, title) for ESCO occupations and also a dataset of (job description title, corresponding ESCO occupation) but it's limited (around 15k entries). Knowing that there are around 3k ESCO occupation titles, would my dataset be enough? $\endgroup$
    – GanaelD
    Commented Apr 22 at 8:20
  • $\begingroup$ @Karl How would such a MLP work? How would that help? $\endgroup$
    – GanaelD
    Commented Apr 22 at 8:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.