Convolution Neural Network Loss and performance

Question

I have a set of about ~100,000 training examples. Ratio of positive to negative example is roughly 1:2. The true ratio is more like 1:100 so this represents a major downsampling of the negative class. It is also a very noisy dataset - the examples were automatically generated through distant supervision. Each example represents a set of sentences and has 700 columns. Number of rows may vary from 10 to 100 (maybe even more).

I used a Convolution Neural Network in Tensorflow to train my model (model architecture similar to the one described here) with only 2 epochs and stored the loss, f-score , precision and recall every 10 steps. I evaluated the model on a validation set (which too was generated automatically through distant supervision with negative class downsampling resulting in pos:neg ratio of ~1:2) every 100 steps. Here are the hyperparameters:

batch size: 60 for train, 100 for validation 
epochs: 2
convolution filter sizes: 700x5, 700x6, 700x7, 700x8, 700x9, 700x10 
number of convolution filters per filter size: 125 (so total of 750 filters)
dropout: 0.5
l2reg: 0.001
lr: 0.001

I'm seeing some strange behavior with the model and I don't understand why.

My training precision, recall and f-score go over 0.95 in about a 100 steps (6000 examples) and then plateaus. The loss falls down from 0.8 to 0.2 in about 200 steps and then fluctuates between 0.1 and 0.4.

On the validation set my precision, recall and f-score are over 0.95 starting from the first time I evaluate it on the 100th step. Loss fall slightly from 0.3 to 0.2.

When I evaluated on a real-world test set (without downsampling negative class so it has the true ratio of pos:neg), the actual precision and recall were 0.37 and 0.85.

My results are not making any sense to me. I use tensorflow metrics for calculating training precision, recall and fscore and scikit-learn metrics for calculation validation precision, recall and fscore. I can't find anything wrong in the code but I don't understand why I should have such results unless there is a bug. I would have understood having low precision and recall all through - the class imbalance favors the negative class and my set is noisy. However, I am very confused about why I'm having such misleadingly high scores all through..

Given that my dev dataset is also noisy and generated in the same manner as the train set, the dev results might just be useless and it is possible that the model is overfitting the noisy set. But I still don't understand why the scores are so high so soon. Also, if overfitting is the issue, do you think I should make the dropout even higher?

I've attached a screenshot of the graphs and would really appreciate your thoughts on this. Blue is train and red is dev. Thanks a lot!

Answer 1

Precision

Your step change in precision looks to be almost entirely explained by the change in positive class frequency. It is reasonable to expect the proportion of false positives to increase when increasing the proportion of negative examples. Even if you assume your cv results were perfect, then you would see some increase.

As an example, assume you have cv results representative of test results - which means same distribution before random under-sampling, and no over-fit to the cv set.

Say you measured precision at 0.97 with a t:f ratio of 1:2, and for the sake of simplicity that this was due to the following confusion table:

      Predicted:  T    F
Real T           97    3
Real F            3  197

What precision should you expect when going to the real distribution? That is the same as multiplying the bottom row of the confusion table by 50. Precision is $\frac{TP}{TP+FP}$, so your expected precision would be $\frac{97}{97+150} \approx 0.39$

Recall

The same effect does not impact recall, because it is about the ratio between true positive and false negative. So when you change the ratio of positive to negative classes, in theory recall should be unaffected.

In your case, recall has been affected, but a lot less than precision. that is promising. A drop from 0.95 to 0.85 between cv and test is not great perhaps, but it doesn't point to a really major problem, just room for improvement.

There are a few possible causes. The ones that I can think of are:

• Your test set might be too small, so estimates of precision and recall have large error. So in fact there is no problem . . .

• Test distribution might be different to train and cv set.

• Train/CV set split might allow some data leakage (e.g. they share some common features such as data about the same person, and should be split by that common feature). In which case CV estimates could be too high.

• Your mechanism for under-sampling the negative class may be biased.

What to do?

First of all, these results are unlikely to be anything directly do with faults in the model, and are not informed much by the training curves. They are also not that bad out of context (i.e. they are much better than simply guessing which items are in the positive class) - the question is more whether you could improve on them, and what the costs are to you for the different types of error. It might be worth you actually assigning real-world comparable costs to each type of error, to help decide whether your model is successful/useful and to pick the best model later on.

One thing from the training curves is that your cv and training loss look pretty close. It implies you are not over-fitting to the training data (or you should check to a train/cv data leak). You may have room to add more parameters and improve the model in general.

It is possible you could make the model even better with different hyper-parameter choices, feature engineering etc, and that would improve the scores. There is no general advice for that though, it depends on what you have available.

It might be worth experimenting with training on the unbalanced training set (take the raw data without undersampling) and instead weighting the loss function, so that costs are larger for inaccurate classification of positive class. This is not guaranteed to fix your problem, but will increase the amount of data you use for training.

Otherwise, you should investigate whether any of the possible causes listed above is likely and try to apply fixes.

Finally, in this situation, it is not unheard of to have a four-way data split:

• A ratio-adjusted set split two ways:

  • Training data
  • CV or "Dev" set A

• A same as production set split two ways:

  • CV or "Dev" set B
  • Test set

CV set A is used to perform early stopping and low-level model selection.

CV set B is used to perform high-level model selection against production metric.

Test set is used to assess the chosen "best" model without bias.