In addition to training the weights of a neural network, I also want to optimize other parameters (that are constant but satisfy some conditions over the entire data set). As an example, one can consider a loss function as follows:

$$ L(x)= \sum_{x \in D} B_{nn}(x) - \eta x, $$ where $D$ is the data set. I want to minimize this value of $\eta$ (a generalization parameter) in addition to optimizing the weights of the network. Can I just simply add $\eta$ into the list of trainable parameters and apply SGD on it? Or is there another simpler way to learn this parameter as well?

  • $\begingroup$ Can you add more information about the purpose of the generalization parameter? As stated, you loss function would always be minimized by selecting the largest possible $\eta$ $\endgroup$
    – zachdj
    Commented Feb 8, 2023 at 20:15
  • $\begingroup$ The purpose of the generalization parameter $\eta$ in my case to ensure that the function B works also for the points outside of the training data set. In my case, this happens when eta is negative, so I need to minimize it and bring it to a negative value (in which case the loss function will actually increase instead of getting minimized). So the neural network and eta work against each other, and I want to find a near-optimal value of both. $\endgroup$
    – Acad
    Commented Feb 8, 2023 at 20:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.