0
$\begingroup$

I am performing text classification as Good [1] or Bad [0]. The texts are preprocessed and converted to Vectors using Google Word2Vec. Further CNN architecture is used for training. I have roughly 13000 texts as Bad[0] and 5450 texts as Good[1] for training (making it a roughly 70:30%) The issue starts when I realize I don't have enough compute power (2GB GPU). Hence I compromise and use 100 dimensions of word embeddings from Word2Vec (instead of the 300). After certain hyperparameter tuning in the CNN architecture, I am able to obtain a 30-35% precision, which I am happy with. After months, I have an 8GB GPU in the server and I implemented the 300-dimensional word embeddings in Word2Vec and kept for training. Ideally, I should have obtained better results; instead, the loss and accuracy don't change with time for every epoch. Thus it predicts all texts as Bad[0]. Can you please help me identify the problem and If I am missing out anything here!

EDIT:

I would like to add some clarifications: In the Linux server with GPU 1070Ti 8GB, I tried with three experiments in this order: a) 300-dimensional word embeddings b) 100-dimensional word-embeddings and c) 105-dimensional word embeddings I have obtained no change in accuracy and loss for a) and c). However, for b), it is exactly the same results as I obtained with my local GPU(750Ti Nvidia 2GB). In short, its working fine for 100-dimensional in the server.

Now, since I can't assign 300d word vectors in my local GPU, I make up the same experiment as c) with 105d vectors in the local GPU to just check if there's any fault in code, and surprisingly it's giving around 30% precision much similar to earlier results.

I am having a hard time figuring out the issue in the server GPU as its working fine for 100d word vectors but fail to give proper predictions for other dimensional word-embeddings.

I am attaching some results which might make it a lot clear:

1.) Trained with 100d vectors in local(2GB) GPU precision recall f1-score support

    0.0       0.70      0.83      0.76      1973
    1.0       **0.31**      0.17      0.22       850

avg / total 0.58 0.63 0.60 2823

2.) Trained with 100d vectors in server (8GB) GPU precision recall f1-score support

    0.0       0.70      0.77      0.73      1973
    1.0       **0.30**      0.23      0.26       850

avg / total 0.58 0.61 0.59 2823

3.) This is the same result for both 105d and 300d vectors trained in server GPU precision recall f1-score support

    0.0       0.70      1.00      0.82      1973
    1.0       **0.00**      0.00      0.00       850

avg / total 0.49 0.70 0.58 2823

4.) Trained with 105d vectors in local(2GB) GPU precision recall f1-score support

    0.0       0.70      0.83      0.76      1973
    1.0       **0.30**      0.18      0.23       850

avg / total 0.57 0.64 0.61 2823

$\endgroup$
1
$\begingroup$

You have imbalanced Data, that's why your network always predicts the most frequent answer (he is in a local minima).

There are solutions to avoid this kind of problem:

  • Balance your data, use as much "good" as "bad" in your training set.

  • Change your cost function, e.g. the cost of error on "good" should be higher than "bad".

(don't use both at the same times or your model will be imbalanced in the other class)

If you look for imbalanced data on google you will find lot of articles on this problem, since it is common.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.