I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net :

class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.fc1 = nn.Linear(3, 3)
self.fc2 = nn.Linear(3, 3)
self.fc3 = nn.Linear(3, 1)

def forward(self, x):
x = torch.selu(self.fc1(x))
x = torch.selu(self.fc2(x))
x = self.fc3(x)
return x


And here is the optimization loop:

    params = list(net.parameters())
for i in range(nb_train):
for k_ in range(nb_mean):
value = f(many arguments)
crit = crit + value
crit = crit / nb_mean
crit.backward(retain_graph=True)
optimizer.step()


The net b is hidden inside crit.

Besides, I know a very bad approximation b_approx of the minimum b_opt. But strangely, this approximation is better than what the net gives me. So in place of b, I have put b+b_approx, where hopefully |b_approx| >> |b|.

Moreover:

• I have tuned the learning rate (from 1e-1 to 1e-100 which is really weird) and the momentum
• I have chosen multiple activation function and to my understanding, selu is generally the best choice.
• The default initialization of the weights seems optimal in Pytorch, but I am not an expert
• I have normalized the inputs
• I have tried SGD and Adam.
• I have tried simpler neural nets, such as a unique layer without activation function.

Could anybody give me an idea as to why the gradient descent converges to suboptimal solutions and how to correct it ? And also, the gradient descent always has spikes, however smal the learning rate : is it a usual behaviour for a loss function ?