How to improve the binary classification model for text (News Articles) of Recurrent Neural Net with word emmbeding?

Question

I am trying to do binary classification of news articles using Recurrent Neural Net with word embedding. Following are the parameters of the model:

Data:
    8000 labelled news articles (Sports:Non-sports::15:85)

Parameters:
    embedding size = 128
    vocabulary size = 100000
    No. of LSTM cell in each layer = 128
    No. of hidden layers = 2
    batch size = 16
    epochs = 10000

Result:
    AUC on training set = 0.60
    AUC on testing set = 0.55

As the both training and testing error is high model is underfitting and require more data. So I have couple of doubts here:

1. What would be the optimum data size required?
2. Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.

Answer 1

I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods. If you have a vocabulary size of 100,000 then you really need to use some extra feature extraction.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 8000 instances!!! Yes. But, you also have 100,000 features for each instance. This is insufficient even for shallow machine learning models. For a neural network, i usually suggest to follow this very general rule of thumb,

$\#examples = 100 * \#features$.

So you will need a MASSIVE amount of data to properly train your neural network model. Moreover, if you have a skewed dataset then you should expect to be using even more training examples, to show sufficient examples such that the model is capable of distinguishing the two classes.

I would suggest a less intensive model. You should try to use: naive bayes, kernel-SVM or knn for text classification. These methods would do MUCH MUCH better than a neural network. Especially considering you have a skewed dataset!!

My Suggestion

I would start with bag of words and then use kernel-SVM. This should be a good starting point.

A recurrent neural network is 0% recommended for the amount of data you have.

Answer 2

There is a major problem of imbalance in your dataset, the classes are in the ratio 15:85. Also the data is lesser, but firstly you will have to address the problem of imbalance. That is probably the root cause of the underfitting of your data. Tweaking some parameters might give you better results, but I dont think you will experience any significant improvement until you address the formerly mentioned isssue.

Edit : I mainly use CNNs for text classification, and in experience have always found it performing at par or better than RNNs.

Answer 3

To mitigate the problem of class imbalance, you can do:

1. Apply weighted cross entropy.
2. When you are feeding each batch to the model, try to make sure there is no class imbalance in the batch data.

Training word embeddings on such small data doesn't make sense. Try to use pre-trained word vectors like the Google news word vectors.

Answer 4

You need to train your classifier on a external larger labelled dataset like this one. (vocab size 1M+)

You may need to slice the data to include only sports articles.

You can use the 8000 articles as the test data for the classifier.