2
$\begingroup$

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational database" should produce words like "mysql", "Oracle", "Sybase", "Microsoft SQL Server" etc...

My thinking is to treat the initial text as a set of words (after lemmatization and stop words removal) and predict words that should be in that set. I can then take all of my texts (of which I have a lot), remove a single word and learn to predict it.

My initial thought was to use word2vec and find words similar to my set. But this doesn't work very well, as I don't want similar words but words that go together in many sentences, which is sort of the task that word2vec is training on to create its embedding...

How would you model this problem? I don't think RNNs are relevant here because I want to use a set of words - they do not have any order, I'm not trying to predict the next word in any way. However, the size of the set could vary I think...

What kind of NN architecture would you use for this sort of problem?

UPDATE:

To further explain the purpose of the predictor and why word2vec doesn't work for me:

I did train a word2vec model on my data (which is very domain specific and I have lots of data on it). Now my problem is that it produces words that are similar to the ones I input to it... and not ones that could be in the context.

e.g: I input "computer manufacturer" to get most similar words and i get:

(u'equipment_manufacturer', 0.7463390827178955), (u'manufacture', 0.7410058975219727), (u'manufacturer_distributor', 0.7196526527404785), (u'distributor', 0.6950951814651489), (u'desktop_computer', 0.6632821559906006)

These are terms that have similar meaning...

But I'm not looking for similar meaning. I would be more interested in getting "Dell" for example as the output. Another example is to associate "Apple_inc" with informatics, OS, hardware.. - not similar words, but related ones...

So my thinking now is to build a word2vec CBOW model, but not to use just the embeddings that are created but actually use the network to predict new words from context...

Improve this question
$\endgroup$
4
  • $\begingroup$ The exact task is not clear: in your example the words don't come from the text, are you looking for any word semantically relevant for the words in the text? And if yes is there a limit on the number of words? $\endgroup$
    – Erwan
    Commented Jul 2, 2019 at 12:46
  • $\begingroup$ Yes, words that are related, from the same topic, that could be in that sentence. Hence my logic is to mask one word and try to learn it from the set of the other words... My example is the kind of words I would hope to get (and it would make sense that they would be found in a sentence similar to the example one...) $\endgroup$
    – Oren Matar
    Commented Jul 2, 2019 at 14:34
  • $\begingroup$ Does this article, "NLP: Predict Text Using Word2Vec Model" have what you are looking for? $\endgroup$
    – Greg Thatcher
    Commented Jul 7, 2019 at 4:50
  • $\begingroup$ can't see the article as I don't have an account on that site... $\endgroup$
    – Oren Matar
    Commented Jul 8, 2019 at 7:29

1 Answer 1

Reset to default
0
$\begingroup$

That problem is commonly called multi-label classification, learn to predict zero or more text labels for each instance.

A multilayer perceptron (MLP) architecture could work.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.