2
$\begingroup$

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to prevent overfitting. While tuning the dropout rate, I am increasing the value from 0.2 -> 0.3 -> 0.4 -> 0.5.

I want to know if I should increase the training data size to have a more accurate comparison. What I mean is suppose I am having training data of size 1 million for a dropout rate 0.2. Should I increase the training data size to 1.5 million for dropout rate 0.3?

Calculation:

1000000 * 0.8 * 0.8 * 0.8 = 512000 (for dropout ratio 0.2)

1500000 * 0.7 * 0.7 * 0.7 = 514500 (for dropout ratio 0.3)

$\endgroup$
  • $\begingroup$ Several clarifications needed: 1) what are you referring to with "input data size"? (i don't think it's the batch size because it's too large, i guess it's not the amount of training data, as you would be using all you have since the beginning) 2) with what do you want "a better comparison"? $\endgroup$ – ncasas Mar 7 '18 at 7:26
  • $\begingroup$ 1) Input data size is actually the training data size. I am not using all I have since there is some amount of preprocessing required before I can feed it to my model. 2) By better I mean more accurate. I have edited this in the question. $\endgroup$ – anas17in Mar 7 '18 at 9:23
  • $\begingroup$ Just to clarify that how is input data is effecting your dropouts? Dropouts are used to prevent overffiting naturally and depends on your network deepness? Also dropout is not applied near the input layers as such. $\endgroup$ – Aditya Mar 7 '18 at 9:26
  • $\begingroup$ Usually dropout refers to nodes not so much examples. $\endgroup$ – S van Balen Mar 7 '18 at 14:22
1
$\begingroup$

No, you should only tune one hyperparameter at a time. If you change two hyperparameters and the performance increases, how do you know which of the two parameters is responsible?

If you have the time, do a full grid search on the dropout rate and the training data size.

$\endgroup$
  • $\begingroup$ What does increasing datasize implies? $\endgroup$ – Aditya Mar 7 '18 at 10:09
  • $\begingroup$ @Aditya increasing data-set size can help the problem of over-fitting. It leads to have a better estimate of the real distribution. $\endgroup$ – Media Mar 7 '18 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.