Skip to main content
3 of 3
softened the stipulation that changing the activation function will not affect the training of the network
zephyr
  • 131
  • 1
  • 9

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler route to an answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights and biases of the network - which we are free to do arbitrarily. This will not affect the performance of the network, but certainly will the initialization. Previously I had stated that it will not affect the training either - but I'm now not sure I can state this with full confidence.

zephyr
  • 131
  • 1
  • 9