2
$\begingroup$

I am trying to do binary classification of news articles using Recurrent Neural Net with word embedding. Following are the parameters of the model:

Data:
    8000 labelled news articles (Sports:Non-sports::15:85)

Parameters:
    embedding size = 128
    vocabulary size = 100000
    No. of LSTM cell in each layer = 128
    No. of hidden layers = 2
    batch size = 16
    epochs = 10000

Result:
    AUC on training set = 0.60
    AUC on testing set = 0.55

As the both training and testing error is high model is underfitting and require more data. So I have couple of doubts here:

  1. What would be the optimum data size required?
  2. Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.
$\endgroup$
3
  • $\begingroup$ What is the vocabulary size? $\endgroup$ Jan 31, 2017 at 6:50
  • $\begingroup$ @HimanshuRai Vocabullary size is 100000. I should update it in the question $\endgroup$
    – Eudie
    Jan 31, 2017 at 7:20
  • $\begingroup$ How does one determine the parameters and the values of the same? $\endgroup$
    – kRazzy R
    May 29, 2018 at 20:43

4 Answers 4

4
$\begingroup$

I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods. If you have a vocabulary size of 100,000 then you really need to use some extra feature extraction.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 8000 instances!!! Yes. But, you also have 100,000 features for each instance. This is insufficient even for shallow machine learning models. For a neural network, i usually suggest to follow this very general rule of thumb,

$\#examples = 100 * \#features$.

So you will need a MASSIVE amount of data to properly train your neural network model. Moreover, if you have a skewed dataset then you should expect to be using even more training examples, to show sufficient examples such that the model is capable of distinguishing the two classes.

I would suggest a less intensive model. You should try to use: naive bayes, kernel-SVM or knn for text classification. These methods would do MUCH MUCH better than a neural network. Especially considering you have a skewed dataset!!

My Suggestion

I would start with bag of words and then use kernel-SVM. This should be a good starting point.

A recurrent neural network is 0% recommended for the amount of data you have.

$\endgroup$
0
1
$\begingroup$

There is a major problem of imbalance in your dataset, the classes are in the ratio 15:85. Also the data is lesser, but firstly you will have to address the problem of imbalance. That is probably the root cause of the underfitting of your data. Tweaking some parameters might give you better results, but I dont think you will experience any significant improvement until you address the formerly mentioned isssue.

Edit : I mainly use CNNs for text classification, and in experience have always found it performing at par or better than RNNs.

$\endgroup$
1
$\begingroup$

To mitigate the problem of class imbalance, you can do:

  1. Apply weighted cross entropy.
  2. When you are feeding each batch to the model, try to make sure there is no class imbalance in the batch data.

Training word embeddings on such small data doesn't make sense. Try to use pre-trained word vectors like the Google news word vectors.

$\endgroup$
6
  • $\begingroup$ Just one clarification for the point 2. Did you mean each batch size have the equal imbalance(15:85)? $\endgroup$
    – Eudie
    Feb 26, 2017 at 14:03
  • $\begingroup$ I train embedding on a separate big data which also consist of this set. $\endgroup$
    – Eudie
    Feb 26, 2017 at 14:04
  • $\begingroup$ You can ensure that each batch has almost no class imbalance. For eg., if your batch size is 10, try to make it somewhere around (Sports:Non-sports::5:5) $\endgroup$ Feb 27, 2017 at 7:30
  • $\begingroup$ but, what if real data is imbalanced say 85:15? $\endgroup$ Jun 26, 2017 at 13:27
  • $\begingroup$ I assume you mean test data by "real" data. There is no issue with class imbalance for test data. You are just assigning scores (or class) for one single sentence from test data. class imbalance in test data has no effect on learnt model. $\endgroup$ Jun 27, 2017 at 9:18
0
$\begingroup$

You need to train your classifier on a external larger labelled dataset like this one. (vocab size 1M+)

You may need to slice the data to include only sports articles.

You can use the 8000 articles as the test data for the classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.