I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net :
class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.fc1 = nn.Linear(3, 3) self.fc2 = nn.Linear(3, 3) self.fc3 = nn.Linear(3, 1) def forward(self, x): x = torch.selu(self.fc1(x)) x = torch.selu(self.fc2(x)) x = self.fc3(x) return x
And here is the optimization loop:
params = list(net.parameters()) optimizer = torch.optim.Adam(params, lr=learning_rate,betas=(moment, moment), amsgrad=True) for i in range(nb_train): for k_ in range(nb_mean): value = f(many arguments) crit = crit + value crit = crit / nb_mean crit.backward(retain_graph=True) optimizer.step() optimizer.zero_grad()
The net b is hidden inside crit.
Besides, I know a very bad approximation b_approx of the minimum b_opt. But strangely, this approximation is better than what the net gives me. So in place of b, I have put b+b_approx, where hopefully |b_approx| >> |b|.
- I have tuned the learning rate (from 1e-1 to 1e-100 which is really weird) and the momentum
- I have chosen multiple activation function and to my understanding, selu is generally the best choice.
- The default initialization of the weights seems optimal in Pytorch, but I am not an expert
- I have normalized the inputs
- I have tried SGD and Adam.
- I have tried simpler neural nets, such as a unique layer without activation function.
Could anybody give me an idea as to why the gradient descent converges to suboptimal solutions and how to correct it ? And also, the gradient descent always has spikes, however smal the learning rate : is it a usual behaviour for a loss function ?