3
$\begingroup$

The weights of my first and last convolution layers do change in a noticeable way. However, the rest of my convolution layers, in the middle, do not.

I should add that all convolution layers' biases change noticeably, however the kernel weights of the middle layers (as opposed to the first and last layers) do not change significantly.

Has anyone else encountered this?

Can anyone suggest why this is might be happening?

Can I rule out vanishing gradients...?

Any ideas for resolution (if this is at all a problem)?

Thanks in advance.

I'm happy to add details based on comments.

Improve this question
$\endgroup$
2
$\begingroup$

Its ofcourse dataspecific, but this behavior is mostly likely not harmful. What could indicate dangerous behaviours is wild behaviour of cv-metric, learning rates etc...

You can think of every cnn layer task to learn specific features/pecularities of a photo. In the first layers they will catch the most general patterns, edges etc... Middle layers are more specific and they look at some finer details but not all too specific. And final layers are there for the small details that can really discriminate the classes and make the right prediction.

So in your case weights are not updated significantly because information in middle layers is not that significant.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for taking the time to answer. I do not understand your answer. In particularly: Are you saying that the first and last layers get trained first and that middle layers get trained later (in time)? Also, what do you mean by the word "generic" (in the sentence "than later in the middle its generic"). Thank you. $\endgroup$ – Elegant Code Mar 4 '20 at 23:40
  • $\begingroup$ I reformulated my answer $\endgroup$ – vienna_kaggling Mar 5 '20 at 9:45
  • $\begingroup$ Thanks. It's much clearer now 👍 $\endgroup$ – Elegant Code Mar 5 '20 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.