I found the following figure in the original Mish paper (https://arxiv.org/abs/1908.08681).

enter image description here

I understand that this figure describes how the loss is being changed, if the change is smooth or not. But the paper don't describe how it was generated, what does the color on the figure mean (no colorbar), or what is the plotted along x and y axis. Can anyone help in this regard?


From the article you cite, I have found this figure which looks similar to yours.

enter image description here

On the same page of the article, they say:

Having a smooth profile also plays a role in better gradient flow, as shown in Fig. 3, where the output landscapes of a five-layered randomly initialized neural network with ReLUand Mish are visualized. The landscapes were generated by passing in the co-ordinates to a five-layered randomly initialized neural network which outputs the corresponding scalar magnitude. The output landscape of ReLU has a lot of sharp transitions as compared to the smooth profile of the output landscape of Mish.

It seems your figure (and the one above) does not show the loss landscapes (which is the figure 4 in the article) but the landscape of the global neural network using different activation functions. I guess the neural network they used to produce these figures has 2 inputs (the coordinates of the plots), 3 hidden layers and one scalar output. The color in the plots corresponds to the scalar output value.

The authors present these figures because :

Smoother output landscapes suggest smooth loss landscape [...]

This is my understanding of the article, hope it helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.