So I trained a 2 layer Neural Network for a regression problem that takes $D$ features $(x_1,...,x_D)$ and outputs a real value $y$. With the model already trained (weights optimised, fixed), can I find the input $(x_1,...,x_D)$ that gives a maximum $y$?

Particularly, I want to know if:

  • This is possible by finding the derivative of the neural network functions composition with respect to $(x_1,...,x_D)$, and equal that to zero. In theory, that should give me either a maximum/minimum, so I could just evaluate those values and pick the ones that gives me highest $y$.
  • This is possible by using some kind of gradient descent, but updating the weights in the opposite direction (this is, with a $+$ sign, gradient ascent). But here I don't see clearly how to implement it. Like, in the normal NN Gradient Descent, we update the weights W, and we use it by evaluating on the training set and computing a loss function. If I want to apply gradient ascent here, which is the loss function? and the training dataset?
  • $\begingroup$ Unless you want to solve it analytically, it should be simple. Libraries such as pytorch and keras have functionality for optimization of a function. Or you can look at your network as a blackbox function and do a simple numerical optimization, without using the gradients. $\endgroup$
    – Valentas
    Commented Nov 17, 2022 at 9:53


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.