I have a binary classification task which has the following specification:

Input: Chunk of text (not more than a few sentences, mostly a sentence).
Additional Input: For each input sample there is additional information available (which is also some text of similar length, max 2-3 sentences).
Problem: Classify text content using additional context
Problem Type: Binary classification

Essentially the task boils down to classifying the content conditioned on the context or p(content|context).

I was thinking of effective ways to encode the text for classification using a deep neural network. I searched for recent works but the existing literature mostly use the technique mentioned below.

I was wondering if there is a better way to encode context? Which has shown to be effective in some domain as compared to just concatenation.

Encode then concatenate: This involves encoding the context using ways similar to the content and then concatenating the feature representation before classification. This is the widely used technique. There are a lot of variants of this technique (different ways to encode text e.g. using tfidf rep, word embedding, LSTMs, CNNs ) which are widely explored.

Are there any better ways to bring context information during classification?

P.S: Recursive RNN is something that I have on my TODO.

  • $\begingroup$ Why are you treating context as a different part of the input? Does a classifier using context + content concatenated not work? $\endgroup$ – kbrose Oct 4 '18 at 17:14
  • $\begingroup$ Concatenating both does not make sense as my goal is to 'predict a class based on content keeping in mind the context'. There must be a better way than to just concatenate. $\endgroup$ – Pinkesh Badjatiya Dec 18 '18 at 7:05
  • 1
    $\begingroup$ Concatenation does “keep in mind the context”. $\endgroup$ – kbrose Dec 18 '18 at 13:52
  • $\begingroup$ But there is still a possibility that the classifier learns to predict 'label based on both content & context' is we just concatenate. Basically, i wanted the classifier to not take decisions based on full information from context $\endgroup$ – Pinkesh Badjatiya Dec 18 '18 at 13:56
  • $\begingroup$ @kbrose think of a situation when we have to classify some text based on content, while keeping in mind the relationship of content with some reference text, which I am referring to as context $\endgroup$ – Pinkesh Badjatiya Dec 18 '18 at 13:58

A Recursive Neural Network (RNN) is best suited to solve your problem. And if you want to take a step further try Recursive Neural Tensor Network. You can generate compositions for entire sentences using RNN both for your content and context data. This will help you map your sentences to a higher dimensional space and as an output you will get numerical values (or embeddings) both for your content and context. Feeding these values into a simple logistic classifier will do your task. For a brief overview on RNNs take a look at this lecture by Richard Socher.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.