I am a sophomore student who's interested in deep learning and its method layering up some linear/non-linear operations and constructs up the complex function through the network.

I'd like to comprise a network that finds the word upon the given explanations :

e.g.1 input: a spherically shaped fruit which tastes sour and sweet and gets red when matured well output: apple

e.g.2, input: to make up the boundary between multiple options for the base of one's behavior or decision output: determine.

So basically it's a classification problem but the class might be more than 20,000.

I'd like to name this task "rephrase neural network".

Any good reference or starting point, or, any network recommendation for comprising this neural network?


1 Answer 1


In a perfect setting, where you have infinite sentences (definitions).
enter image description here

You could use some kind of attention models by chunking sentences into smaller pieces.

[a spherically shaped] [fruit] which [tastes sour] and [sweet] and gets [red] when matured well.

With a distributed representation framework, every little piece of information is simply like a binary indicator (the feature exists or not).

Maybe each piece is vague by itself and it could not tell you anything about the target yet by combining them together they define useful information, see products of experts

To learn p(Y|X) where x consists of the sentence, y is the embedding of some class, so may similar classes share similar sentences.
If you have a model trained on the sentences and learned to project similar sentences near to each other, and assume you learned the word classes by some word2vec like algorithm, you could generalize to new pair even if the sentence and that class target were never paired (never seen in your training data), their respective feature vectors have been related to each other by the mapping function.

At the end of the day if you have a smooth function and your input is continuous then any small change of the input will introduce small change at the output, yet how could we truly extract the knowledge exists in the data. see Generating Sentences from a Continuous Space

  • $\begingroup$ a bit hard to understand upon my level of background knowledge. Would you just recommend me some network structure that worth trying? I can handle python with pytorch or tensorflow. $\endgroup$
    – snapper
    May 6, 2018 at 12:00
  • $\begingroup$ your problem could be formulated as Many to One sequence-based structure, where many are some high-level concepts extracted from the raw input words, and the one is the embedding of the class target, but be careful if you don't have enough definitions (examples), the problem is not solvable, see github.com/yunjey/pytorch-tutorial/tree/master/tutorials/… for more intuition $\endgroup$ May 6, 2018 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.