I have a dataset of companies and research projects that they were involved in. A subset of the dataset is shown below.

enter image description here

I am trying to find a way to model similarity between the company and the research projects using this description.

For each pair of descriptions, I would like to output a number, similar to cosine similarity, which will indicate how similar the description in second column is, to the project title in third column.

How can I go about this?

Thanks in advance!


So the questions asks for how to compute similarity between the organisation description and project titles.

One initial thought would be to use a Doc2Vec model (concept, implementation), which will take the organisation descriptions and project titles as input and output a n-dimensional vector in semantic space for the given text.

From this, you can at least has a baseline which you can use with cosine similarity to see how similar the organisation description is to the project title.

| improve this answer | |
  • $\begingroup$ Thanks for the link. A few further technical details would be nice. Following radimrehurek.com/gensim/auto_examples/core/… : 1) I'm a bit unclear on what I need to install and where I can find it, I have Python, pandas, numpy etc but nothing specific for NLP as yet. 2) when constructing the corpus, should I include the org descriptions as many times as they appear or just once each? 3) any guidelines to choosing an appropriate value of num_topics (dimensionality of LSI space I guess)? $\endgroup$ – Mobeus Zoom Aug 7 at 16:21
  • $\begingroup$ also is there a way of using a pre-trained model (for, say, tokens) to improve the performance, compared to just reproducing exactly what goes on in radimrehurek.com/gensim/auto_examples/core/… $\endgroup$ – Mobeus Zoom Aug 7 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.