1
$\begingroup$

I plot the landscape using this code, and I notice the landscape shape has changed a lot. My understanding is that the optimizer does not change the loss landscape. But now I'm confused if its just that I made a mistake in generating the plot or that indeed the optimizer does change the landscape.

$\endgroup$

1 Answer 1

5
$\begingroup$

No, the optimizer does not change the loss landscape. The loss landscape consists of the possible parameters of your model and the incured loss of a model which has these parameters. We could for instance write it as $$ \lbrace (\theta , L(m_\theta (x), y) ) \mid \theta \in \Theta \rbrace, $$ where $\Theta$ is some set of admissible parameters, $m$ is your model, $L$ is the (average) loss, $x$ are features and $y$ are observations. An optimizer lets you find good parameters $\theta$, but does not change the landscape.

However, the package loss-landscapes you are referring to does not plot the loss landscape! It does two things (see the README.md):

  • It plots a landscape based on a subspace of the parameter space $\Theta$, since the latter is too high-dimensional to be properly visualized.
  • It plots this modified landscape 'around' the parameters $\theta^*$ of your fitted model.

Thus, there is no reason to believe your plot will look the same after changing the optimizer. In particular, if you use a different optimizer for fitting, you will most certainly estimate different parameters $\theta^*$ and so your plot will contain a different part of the (subspace of) the landscape.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.