Why use different variations of Softmax in training and validation for neural networks with Pytorch?

Question

Specifically, I'm working on a modeling project, and I see someone else's code that looks like

def forward(self, x):
    x = self.fc1(x)
    x = self.activation1(x)
    x = self.fc2(x)
    x = self.activation2(x)
    x = self.fc3(x)
    x = self.activation3(x)
    # use log softmax + NLLLoss in training; softmax to make predictions
    if self.training:
        x = self.log_softmax(x)
    else:
        x = self.softmax(x)
    return x

For context, this is using PyTorch, and it is on a classification problem. The criterion is NLLLoss. What's the rationale behind using log_softmax for training but using softmax for actual predictions?

Answer 1

It's more of a pytorch implementation thing. log_softmax() outputs the raw logits and they are passed to NLL Loss in training. During inference you just need the probabilities so softmax will suffice.

You don't use different algorithms for training and testing, the question is quite misleading as currently stated. You are using different implementations of the same algorithm.

You can find more on the matter here:

• Pytorch discuss
• Pytorch discuss #2
• Pytorch docs

P.S: I remember encountering the same question on a Udacity scholarship. If you 're reading code from one of their courses, there is probably an explanation on the solution jupyter.