I have a binary classification task which has the following specification:

Input: Chunk of text (not more than a few sentences, mostly a sentence).
Additional Input: For each input sample there is additional information available (which is also some text of similar length, max 2-3 sentences).
Problem: Classify text content using additional context
Problem Type: Binary classification

Essentially the task boils down to classifying the content conditioned on the context or p(content|context).

I was thinking of effective ways to encode the text for classification using a deep neural network. I searched for recent works but the existing literature mostly use the technique mentioned below.

I was wondering if there is a better way to encode context? Which has shown to be effective in some domain as compared to just concatenation.

Encode then concatenate: This involves encoding the context using ways similar to the content and then concatenating the feature representation before classification. This is the widely used technique. There are a lot of variants of this technique (different ways to encode text e.g. using tfidf rep, word embedding, LSTMs, CNNs ) which are widely explored.

Are there any better ways to bring context information during classification?

P.S: Recursive RNN is something that I have on my TODO.

  • $\begingroup$ Why are you treating context as a different part of the input? Does a classifier using context + content concatenated not work? $\endgroup$
    – kbrose
    Oct 4, 2018 at 17:14
  • $\begingroup$ Concatenating both does not make sense as my goal is to 'predict a class based on content keeping in mind the context'. There must be a better way than to just concatenate. $\endgroup$ Dec 18, 2018 at 7:05
  • 1
    $\begingroup$ Concatenation does “keep in mind the context”. $\endgroup$
    – kbrose
    Dec 18, 2018 at 13:52
  • $\begingroup$ But there is still a possibility that the classifier learns to predict 'label based on both content & context' is we just concatenate. Basically, i wanted the classifier to not take decisions based on full information from context $\endgroup$ Dec 18, 2018 at 13:56
  • $\begingroup$ @kbrose think of a situation when we have to classify some text based on content, while keeping in mind the relationship of content with some reference text, which I am referring to as context $\endgroup$ Dec 18, 2018 at 13:58

1 Answer 1


A Recursive Neural Network (RNN) is best suited to solve your problem. And if you want to take a step further try Recursive Neural Tensor Network. You can generate compositions for entire sentences using RNN both for your content and context data. This will help you map your sentences to a higher dimensional space and as an output you will get numerical values (or embeddings) both for your content and context. Feeding these values into a simple logistic classifier will do your task. For a brief overview on RNNs take a look at this lecture by Richard Socher.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.