My question might sound a little bit stupid but I am trying to come up with a way to measure the relevance of the title of a text, let's say a piece of news headline, to the content. My idea would be to work with word2vec and have a go with cosine similarity. Are there any more efficient and proper ways to work with this task?

  • 1
    $\begingroup$ No, your question is not stupid :) I think your idea to compute a similarity score with word2vec makes sense. Afaik there's no standard way to measure relevance, there are many methods and also many ways to defined relevance. $\endgroup$ – Erwan May 5 at 16:25

Word2Vec and cosine similarity are certainly a plausible solution. Note that the longer the text is, the more function words are in the text, therefore the average Word2Vec vectors are more similar to each other with the growing text length. This can be partially solved by removing the stop-words.

Another method to try might be cosine similarity with representation from contextual embeddings like ELMo or BERT. Also if you have some training examples (small hundreds might be enough) training a classifier based on BERT should work.

Finally, the fanciest solution is using the summarization model. Generating a title from an article content is one of the tasks that is done in summarization literature. If your data is similar to datasets that are used for training summarization models (or you have enough domain-specific data to train your own summarization model), you can use the model to estimate the probability of the title given the content. Even models that perform poorly on generation, perform well when used discriminatively.

| improve this answer | |
  • $\begingroup$ Thanks a lot for your input. I've been actually considering generating headlines based on a rough summarization of the content. This will come after I do what I've already proposed with the relevance score. I am aware of BERT but I did not experiment with it. Will look into it. Thanks again! $\endgroup$ – Andreea-Codrina Moldovan May 7 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.