2
$\begingroup$

My understanding of gradient descent as an optimizer for a neural network is as follows:

Let $w$ be a vector of weights encoding a configuration of the network, and $l : w \mapsto \textrm{network loss}$ a function which calculates the loss over some batch of data given this configuration. Then, the weight update is $-\alpha \nabla l(w)$, where $\alpha$ is the learning rate, since the vector $\nabla l(w)$ represents the direction of greatest increase in the neighborhood around $w$.

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

Improve this question
$\endgroup$
5
  • $\begingroup$ What is the application of this vector loss? $\endgroup$
    – Emre
    Sep 23, 2017 at 17:10
  • $\begingroup$ I don't really have an application in mind, but I noticed that tensorflow optimizer objects take vectors in addition to scalars, and I was curious how they defined the optimization $\endgroup$
    – William Merrill
    Sep 23, 2017 at 17:25
  • $\begingroup$ Worry about crossing that bridge when you get to it then. This problem is called en.wikipedia.org/wiki/Multi-objective_optimization $\endgroup$
    – Emre
    Sep 23, 2017 at 17:30
  • $\begingroup$ @Emre: Is the TensorFlow optimiser performing multi-objective optimisation when you pass in a multi-value tensor to minimise? It seems unlikely to me. $\endgroup$
    – Neil Slater
    Sep 23, 2017 at 18:16
  • $\begingroup$ I don't know but I doubt it too. It is not common. Usually one makes the trade-off between the objectives explicit with a linear combination. $\endgroup$
    – Emre
    Sep 23, 2017 at 18:21

1 Answer 1

Reset to default
2
$\begingroup$

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

Generally in neural network optimisers it does not*, because it is not possible to define what optimising a multi-value function means whilst keeping the values separate. If you have a multi-valued loss function, you will need to reduce it to a single value in order to optimise.

When a neural network has multiple outputs, then typically the loss function that is optimised is a (possibly weighted) sum of the individual loss functions calculated from each prediction/ground truth pair in the output vector.

If your loss function is naturally a vector, then you must choose some reduction of it to scalar value e.g. you can minimise the magnitude or maximise some dot-product of a vector, but you cannot "minimise a vector".

* There is a useful definition of multi-objective optimisation, which effectively finds multiple sets of parameters that cannot be improved upon (for a very specific definition of optimality called Pareto optimality). I do not think it is commonly used in neural network frameworks such as TensorFlow. Instead I suspect that passing a vector loss function into TensorFlow optimiser will cause it to optimise a simple sum of vector components.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, this makes a lot of sense, since there is no consistent ordering of R^2, for example. The reason why I was asking is that Tensorflow allows you to pass an un-reduced vector or tensor to an optimizer object, and I was wondering how they handled this. $\endgroup$
    – William Merrill
    Sep 23, 2017 at 17:24
  • $\begingroup$ @WilliamMerrill: Actually I don't know, but I'd expect something basic like tf.reduce_sum is applied. It may be a good follow-up question. $\endgroup$
    – Neil Slater
    Sep 23, 2017 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.