New answers tagged

0

Short answer: When updating the weights (or parameters) of your machine learning architecture, you move along the gradient of the loss function applied to the empirical data and the data that your model predicts. This gradient can (and hopefully will, but doesn't have to) decrease as the number of epochs increases, so training will go on just fine. Example. ...


1

Shortly after the success of ReLU was demonstrated, there was a flood of research papers describing the performance of different exotic activations. There are probably hundreds of different activation functions that have been published, many of which just never caught on. You can literally use any univariate function as an activation, so the space of "...


Top 50 recent answers are included