My understanding of gradient descent as an optimizer for a neural network is as follows:

Let $w$ be a vector of weights encoding a configuration of the network, and $l : w \mapsto \textrm{network loss}$ a function which calculates the loss over some batch of data given this configuration. Then, the weight update is $-\alpha \nabla l(w)$, where $\alpha$ is the learning rate, since the vector $\nabla l(w)$ represents the direction of greatest increase in the neighborhood around $w$.

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

  • $\begingroup$ What is the application of this vector loss? $\endgroup$ – Emre Sep 23 '17 at 17:10
  • $\begingroup$ I don't really have an application in mind, but I noticed that tensorflow optimizer objects take vectors in addition to scalars, and I was curious how they defined the optimization $\endgroup$ – William Merrill Sep 23 '17 at 17:25
  • $\begingroup$ Worry about crossing that bridge when you get to it then. This problem is called en.wikipedia.org/wiki/Multi-objective_optimization $\endgroup$ – Emre Sep 23 '17 at 17:30
  • $\begingroup$ @Emre: Is the TensorFlow optimiser performing multi-objective optimisation when you pass in a multi-value tensor to minimise? It seems unlikely to me. $\endgroup$ – Neil Slater Sep 23 '17 at 18:16
  • $\begingroup$ I don't know but I doubt it too. It is not common. Usually one makes the trade-off between the objectives explicit with a linear combination. $\endgroup$ – Emre Sep 23 '17 at 18:21

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

Generally in neural network optimisers it does not*, because it is not possible to define what optimising a multi-value function means whilst keeping the values separate. If you have a multi-valued loss function, you will need to reduce it to a single value in order to optimise.

When a neural network has multiple outputs, then typically the loss function that is optimised is a (possibly weighted) sum of the individual loss functions calculated from each prediction/ground truth pair in the output vector.

If your loss function is naturally a vector, then you must choose some reduction of it to scalar value e.g. you can minimise the magnitude or maximise some dot-product of a vector, but you cannot "minimise a vector".

* There is a useful definition of multi-objective optimisation, which effectively finds multiple sets of parameters that cannot be improved upon (for a very specific definition of optimality called Pareto optimality). I do not think it is commonly used in neural network frameworks such as TensorFlow. Instead I suspect that passing a vector loss function into TensorFlow optimiser will cause it to optimise a simple sum of vector components.

  • $\begingroup$ Thanks, this makes a lot of sense, since there is no consistent ordering of R^2, for example. The reason why I was asking is that Tensorflow allows you to pass an un-reduced vector or tensor to an optimizer object, and I was wondering how they handled this. $\endgroup$ – William Merrill Sep 23 '17 at 17:24
  • $\begingroup$ @WilliamMerrill: Actually I don't know, but I'd expect something basic like tf.reduce_sum is applied. It may be a good follow-up question. $\endgroup$ – Neil Slater Sep 23 '17 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.