4
$\begingroup$

As background, I am using a Deep Neural Network built using Keras to classify inputs into 5 categories.

The current structure of the network is:

  • Input layer (~450 nodes)
  • Dense layer (750 nodes)
  • Dropout layer (750 nodes, dropout rate = 0.5)
  • Dense layer (5 nodes)

The issue I'm having if one of overfitting. My model performs well on the held-out test set (a proportion of my training set), with accuracy sitting right around 99%. However, when I look to apply the model onto unlabelled data, it is only able to classify ~67% of observations into any category, before even considering the correctness of those classifications!

I think the issue may be around my feature and training set generation process. I generated the training set using a rules-based string matching method. This generated a training set of around 3.6 million observations (10% of population).

However, one of the largest features for my input layer is an embedding of the same text used to generate the training set. Therefore, the words matched to generate the training set are also embedded and used as features. Worth noting the text is around 140 characters per observation, and I matched bigrams from that text (so there is other information in that text that would be useful as a feature).

I would remove this feature altogether, however this is the richest information associated with each observation.

Is there a way to solve this without removing that feature altogether?

Hope this makes sense and happy to provide more clarification.

Simplified explanation:

  • My model performs well on my training and test sets.
  • Performs badly on unlabelled data.
  • Each observation is associated with a block of text.
  • To label my training set I used string matching on that text.
  • The text is also a feature (embedding).
  • Is this causing my poor performance on unlabelled data (model learning those string matches?).
  • If so what can I do?

EDIT: Also happy to hear if you think the issue is something else.

Improve this question
$\endgroup$
8
  • $\begingroup$ Not quite sure what is going on, could you maybe post the relevant code? $\endgroup$
    – matthiaw91
    Dec 4, 2019 at 13:13
  • $\begingroup$ Not sure the code would hep because it's not a coding issue, more conceptual (and this is part of quite a large project). But will add some clarification. $\endgroup$
    – Jamie
    Dec 4, 2019 at 13:19
  • $\begingroup$ Probably irrelevant to your overfitting problem, but how does the model fail to classify observations? $\endgroup$
    – Ben Reiniger ♦
    Dec 4, 2019 at 13:33
  • 1
    $\begingroup$ I think the main confusion here is the process of "generating training set" and I assume the problem lies there. What do you mean by "generating data"? what is the original raw data? $\endgroup$
    – Kasra Manshaei
    Dec 4, 2019 at 14:23
  • 1
    $\begingroup$ I mean the process of getting labelled examples to use as training data. I had a set of rules - i.e. any observation whose text contained "mortgage repayment" was a "housing" observation. By using these rules i created a set of labelled examples to train my model on. The trained model is used to identify those observations that aren't captured by my rules. $\endgroup$
    – Jamie
    Dec 4, 2019 at 14:34

1 Answer 1

Reset to default
2
$\begingroup$

I would say that there are no "problems" in the sense what is happening is to be expected.

First of all, here is a key ML reminder which somehow often gets lost:

  • Performing well on the test set is pointless if that test set is not representative (i.e. is not similar to unlabeled instances)
  • Adding more training data does not help if that training data isn't covering new situations

You say you created your training set using string matching rules, I assume that these rules are similar to " "mortgage repayment" was a "housing" observation." as you pointed out in the comments.

Since your model considers bigrams as input, it is not surprising that it found a way to reverse-engineer which string matching rules you used.

To improve your model, I would look at what mistakes your model currently makes on unlabeled data, this should provide you with training instances that do not fit your string matching rules.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks Valentin, this is useful. When looking at misclassified instances, should I look to generate rules based on common mistakes? Hand labelling would take a long time to have an effect with 3.6m current training examples. But then couldn’t my model just learn that new rule! $\endgroup$
    – Jamie
    Dec 5, 2019 at 9:08
  • 1
    $\begingroup$ Indeed, manual labeling is challenging. I would indeed perhaps refine your rules so that your training labels are less noisy $\endgroup$
    – Valentin Calomme
    Dec 5, 2019 at 9:09
  • 1
    $\begingroup$ Thanks again - one last thing. Do you think removing the bigrams used in rules from the text before embedding could also be a good strategy? $\endgroup$
    – Jamie
    Dec 5, 2019 at 9:11
  • 1
    $\begingroup$ Perhaps not, an approach could be to add noise (typos) in your data so that it's harder for your network to learn the rules you used to label it. Or even train with less data to avoid overfitting $\endgroup$
    – Valentin Calomme
    Dec 5, 2019 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.