I always thought the SGD was a loss function then I read this on a notebook
model.compile(loss="sparse_categorical_crossentropy", optimizer=keras.optimizers.SGD(lr=1e-3), metrics=["accuracy"])
now I am confused what's the difference between loss and optimizer ? are they both used at the output layer to calculate the loss? or is the optimizer something used in each layer?
Think of loss function what to minimize and optimizer how to minimize the loss.
loss could be mean absolute error and in order to reduce it, weights and biases are updated after each epoch. optimizer is used to calculate and update them.
Your loss is cross entropy, your variant of gradient descent is stochastic gradient descent, and your optimizer for stochastic gradient descent would seem to be the momentum optimizer according to the keras docs. (https://keras.io/api/optimizers/sgd/). Here is the intuition:
The loss is a way of measuring the difference between your target label(s) and your prediction label(s). There are many ways of doing this, for example mean squared error, squares the difference between target and prediction. Cross entropy is a more complex loss formula related to information theory.
Gradient descent algorithms like batch, minibatch, and stochastic gradient descent specify how many samples from your training data you are going to use at each step to compute the "gradient" (derivative) of your loss with respect to each of the parameters in your model, and then it makes changes to your parameters proportional to the gradient. SGD does "optimization" but it is not an "optimizer".
Your optimizer, ie: momentum helps like so, "Momentum was invented for reducing high variance in SGD and softens the convergence". Essentially smoother gradients can help speed up model training.(https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6).
