Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but very similar in an IT context.
My question is if there is a benchmark dataset for this challenge. I've been looking, but I can't find anything.
Thanks in advance.
I think some datasets used for word sense disambiguation (WSD) would be an option.
WSD is the task of classifying an ambiguous word into its correct meaning. For instance "apple" would have meaning 1 the fruit and meaning 2 the tech company. As a consequence a labelled dataset identifies the correct context for the meaning.
I don't know any specific dataset but I assume that state of the art papers mention and use these datasets.
