Skip to main content
Data Science

Return to Answer

softened the stipulation that changing the activation function will not affect the training of the network
Source Link
zephyr
zephyr
  • 131
  • 1
  • 9

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler route to an answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights and biases of the network - which we are free to do arbitrarily - and should. This will not affect the trainingperformance of the network (except possibly in, but certainly will the initialization stage. Previously I had stated that it will not affect the training either - wherebut I'm now not sure I can state this does need to be taken into account and where the below answers are worth exploring)with full confidence.

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily - and should not affect the training of the network (except possibly in the initialization stage - where this does need to be taken into account and where the below answers are worth exploring).

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler route to an answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights and biases of the network - which we are free to do arbitrarily. This will not affect the performance of the network, but certainly will the initialization. Previously I had stated that it will not affect the training either - but I'm now not sure I can state this with full confidence.

added 19 characters in body
Source Link
zephyr
zephyr
  • 131
  • 1
  • 9

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily - and should not affect the training of the network (except possibly in the initialization stage - where this does need to be taken into account and where the below answers are worth exploring).

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the activation function.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily - and should not affect the training of the network (except possibly in the initialization stage - where this does need to be taken into account and where the below answers are worth exploring).

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily - and should not affect the training of the network (except possibly in the initialization stage - where this does need to be taken into account and where the below answers are worth exploring).

Source Link
zephyr
zephyr
  • 131
  • 1
  • 9

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the activation function.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily - and should not affect the training of the network (except possibly in the initialization stage - where this does need to be taken into account and where the below answers are worth exploring).