I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net :

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(3, 3)
        self.fc2 = nn.Linear(3, 3)
        self.fc3 = nn.Linear(3, 1)

    def forward(self, x):
        x = torch.selu(self.fc1(x))
        x = torch.selu(self.fc2(x))
        x = self.fc3(x)
        return x

And here is the optimization loop:

    params = list(net.parameters())
    optimizer = torch.optim.Adam(params, lr=learning_rate,betas=(moment[0], moment[1]), amsgrad=True)
    for i in range(nb_train):        
        for k_ in range(nb_mean):                          
            value = f(many arguments)
            crit = crit + value     
        crit = crit / nb_mean

The net b is hidden inside crit.

Besides, I know a very bad approximation b_approx of the minimum b_opt. But strangely, this approximation is better than what the net gives me. So in place of b, I have put b+b_approx, where hopefully |b_approx| >> |b|.


  • I have tuned the learning rate (from 1e-1 to 1e-100 which is really weird) and the momentum
  • I have chosen multiple activation function and to my understanding, selu is generally the best choice.
  • The default initialization of the weights seems optimal in Pytorch, but I am not an expert
  • I have normalized the inputs
  • I have tried SGD and Adam.
  • I have tried simpler neural nets, such as a unique layer without activation function.

Could anybody give me an idea as to why the gradient descent converges to suboptimal solutions and how to correct it ? And also, the gradient descent always has spikes, however smal the learning rate : is it a usual behaviour for a loss function ?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.