1
$\begingroup$

Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is coadaptation a sign of overfitting?

$\endgroup$
1
$\begingroup$

Co-adaptions in simple English term would mean co-operation. If you think nodes of a NN as workers it would mean missing even a few workers would result in failure of the NN to do something substantial. This happens mainly due to few nodes of NN outputting values which get cancelled by other nodes (so we can remove those nodes altogether - essentially redundant units), thus resulting in inefficient use of the NN. Thus instead of say 2 nodes, 8 nodes are performing the same job with 6 nodes cancelling each other out. Here is the best illustration of the same.

In case of XOR approximation you need minimum (I'm not sure) 2 nodes to approximate the function correctly, but you can use 100 nodes and most of them will be redundant.

Also this will clearly cause over-fitting as the nodes instead of generalising on the training data are micro-managing the training data.

Now we do not want our NN's to behave in such a manner, we want the other nodes to take over jobs of missing nodes (if any are missing) just like our brain takes over function when one centre (Say speech, hearing is damaged) and the state of art mechanism currently is dropout in layers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.