Gradient descent with vector-valued loss

My understanding of gradient descent as an optimizer for a neural network is as follows:

Let $w$ be a vector of weights encoding a configuration of the network, and $l : w \mapsto \textrm{network loss}$ a function which calculates the loss over some batch of data given this configuration. Then, the weight update is $-\alpha \nabla l(w)$, where $\alpha$ is the learning rate, since the vector $\nabla l(w)$ represents the direction of greatest increase in the neighborhood around $w$.

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

• What is the application of this vector loss? – Emre Sep 23 '17 at 17:10
• I don't really have an application in mind, but I noticed that tensorflow optimizer objects take vectors in addition to scalars, and I was curious how they defined the optimization – William Merrill Sep 23 '17 at 17:25
• Worry about crossing that bridge when you get to it then. This problem is called en.wikipedia.org/wiki/Multi-objective_optimization – Emre Sep 23 '17 at 17:30
• @Emre: Is the TensorFlow optimiser performing multi-objective optimisation when you pass in a multi-value tensor to minimise? It seems unlikely to me. – Neil Slater Sep 23 '17 at 18:16
• I don't know but I doubt it too. It is not common. Usually one makes the trade-off between the objectives explicit with a linear combination. – Emre Sep 23 '17 at 18:21

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

Generally in neural network optimisers it does not*, because it is not possible to define what optimising a multi-value function means whilst keeping the values separate. If you have a multi-valued loss function, you will need to reduce it to a single value in order to optimise.

When a neural network has multiple outputs, then typically the loss function that is optimised is a (possibly weighted) sum of the individual loss functions calculated from each prediction/ground truth pair in the output vector.

If your loss function is naturally a vector, then you must choose some reduction of it to scalar value e.g. you can minimise the magnitude or maximise some dot-product of a vector, but you cannot "minimise a vector".

* There is a useful definition of multi-objective optimisation, which effectively finds multiple sets of parameters that cannot be improved upon (for a very specific definition of optimality called Pareto optimality). I do not think it is commonly used in neural network frameworks such as TensorFlow. Instead I suspect that passing a vector loss function into TensorFlow optimiser will cause it to optimise a simple sum of vector components.

• Thanks, this makes a lot of sense, since there is no consistent ordering of R^2, for example. The reason why I was asking is that Tensorflow allows you to pass an un-reduced vector or tensor to an optimizer object, and I was wondering how they handled this. – William Merrill Sep 23 '17 at 17:24
• @WilliamMerrill: Actually I don't know, but I'd expect something basic like tf.reduce_sum is applied. It may be a good follow-up question. – Neil Slater Sep 23 '17 at 17:44