2
$\begingroup$

When I add more hidden layers to my CNN (e.g. Dense Layers) it seems that the model needs more training samples to produce good results for classes with few training samples. In the single layer case the model provided better results, even for classes with few samples. (multi-class text classification with >10 classes)

Is there any evidence that my assumption is correct?

$\endgroup$

1 Answer 1

2
$\begingroup$

Yes, this is common knowledge. Every time you add a parameter to a model, you will need to give it more data in order for it to learn as well as the simpler model. Every individual weight in a neural network is a parameter. The more weights, the more parameters, and the more data is needed.

One of the fundamental tasks in model-building is finding a good tradeoff between having enough parameters to learn fine details and having enough data to train all those parameters. Too many parameters lead to overfitting in part because if you don't have enough data you end up memorizing individual cases, as opposed to learning about and averaging over a spread across several cases.

This is one reason why class imbalance is a problem. If you don't have enough data on one of the classes, then that class will be poorly understood by the model.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer. Could you provide a link to a paper which shows that relation between “amount of available data” & “model depth” & “overfitting”. $\endgroup$ Oct 8, 2018 at 14:00
  • $\begingroup$ @PredictedLife refer to any stats or machine learning textbook, e.g. Elements of Statistical Learning talks about it in an early chaper. $\endgroup$ Oct 8, 2018 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.