2
$\begingroup$

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to prevent overfitting. While tuning the dropout rate, I am increasing the value from 0.2 -> 0.3 -> 0.4 -> 0.5.

I want to know if I should increase the training data size to have a more accurate comparison. What I mean is suppose I am having training data of size 1 million for a dropout rate 0.2. Should I increase the training data size to 1.5 million for dropout rate 0.3?

Calculation:

1000000 * 0.8 * 0.8 * 0.8 = 512000 (for dropout ratio 0.2)

1500000 * 0.7 * 0.7 * 0.7 = 514500 (for dropout ratio 0.3)

Improve this question
$\endgroup$
4
  • $\begingroup$ Several clarifications needed: 1) what are you referring to with "input data size"? (i don't think it's the batch size because it's too large, i guess it's not the amount of training data, as you would be using all you have since the beginning) 2) with what do you want "a better comparison"? $\endgroup$
    – noe
    Mar 7, 2018 at 7:26
  • $\begingroup$ 1) Input data size is actually the training data size. I am not using all I have since there is some amount of preprocessing required before I can feed it to my model. 2) By better I mean more accurate. I have edited this in the question. $\endgroup$
    – anas17in
    Mar 7, 2018 at 9:23
  • $\begingroup$ Just to clarify that how is input data is effecting your dropouts? Dropouts are used to prevent overffiting naturally and depends on your network deepness? Also dropout is not applied near the input layers as such. $\endgroup$
    – Aditya
    Mar 7, 2018 at 9:26
  • $\begingroup$ Usually dropout refers to nodes not so much examples. $\endgroup$
    – S van Balen
    Mar 7, 2018 at 14:22

1 Answer 1

Reset to default
1
$\begingroup$

No, you should only tune one hyperparameter at a time. If you change two hyperparameters and the performance increases, how do you know which of the two parameters is responsible?

If you have the time, do a full grid search on the dropout rate and the training data size.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What does increasing datasize implies? $\endgroup$
    – Aditya
    Mar 7, 2018 at 10:09
  • $\begingroup$ @Aditya increasing data-set size can help the problem of over-fitting. It leads to have a better estimate of the real distribution. $\endgroup$
    – Green Falcon
    Mar 7, 2018 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.