2
$\begingroup$

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model.

the DL model is actually quite simple too. the embedding layer is trainable and no weights has been passed to it. enter image description here

And after the training, while I was making predictions with unseen text, I could realise that only one of the key-words had huge importance in predictions. To make sure that that is the case, I have changed input texts to only include those key-words and softmax-probabilites were more or less same.

So, what changes do I make in my model, from embedding-technique to network, or in anyway to not have the model rely on some of the words and force it to consider entire sentence?

$\endgroup$
1
$\begingroup$

Dense layer | Feature in-dependency

There are a couple of things that may be going on in your experiment. First of all, it may as well be that the particular keyword you are seeing to play the most important role is intrinsically correlated with a correct prediction.

That aside, by feeding the network words in the form of a dense layer, these words are introduced with no intrinsic inter-dependency and are treated as orthogonal in the forward pass.

RNN layer | Feature step-wise dependency

force it to consider entire sentence

I think what you may be saying is that, some there is some "sense" in the ordering or temporal structure of your text, and that you would like to impose that as an intrinsic characteristic of your data as it is being passed over to your network. The above is exactly what a recurrent layer is meant to be doing: impose a step-wise dependency between the words/features as they take place in time and only allow your data to be seen in this particular recurrence structure. In this way, a fixed-length representation of your target/input sequence is downprojected into the hidden state vector of the last timestep of your sequence.

Your model will get adjusted based on this particular last-hidden-state distillation of your input texts, and thus enforced to look at the entirety of sentences by proxy.

If you try this, beware of vanishing gradients ;)

$\endgroup$
2
  • $\begingroup$ huge thanks for the detailed answer and insights @hH1sG0n3. I will replace the dense layer after the embedding layer with a GRU or LSTM and train the network again. $\endgroup$ Nov 3 at 17:13
  • 1
    $\begingroup$ No problem, depending on what is the length of your input sentences you may want to check that your backprop adjustments do follow all the way back. You can do that by checking the gradient flow, there are libraries for checking and visualising both in keras and pyrtorch - good luck! If it solves your question feel free to tick my answer as a solution :D $\endgroup$
    – hH1sG0n3
    Nov 3 at 17:15
1
$\begingroup$

Adding Dropout, randomly omitting units during training, encourages a neural network to learn a more robust representation.

$\endgroup$
1
  • $\begingroup$ Thanks for this insight. as shown in the picture attached, I have 2 drop out layers. so, do i just increase the count of dropout layers? or increase the dropout rate? $\endgroup$ Nov 3 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.