Specifically, I'm working on a modeling project, and I see someone else's code that looks like

def forward(self, x):
    x = self.fc1(x)
    x = self.activation1(x)
    x = self.fc2(x)
    x = self.activation2(x)
    x = self.fc3(x)
    x = self.activation3(x)
    # use log softmax + NLLLoss in training; softmax to make predictions
    if self.training:
        x = self.log_softmax(x)
        x = self.softmax(x)
    return x

For context, this is using PyTorch, and it is on a classification problem. The criterion is NLLLoss. What's the rationale behind using log_softmax for training but using softmax for actual predictions?


1 Answer 1


It's more of a pytorch implementation thing. log_softmax() outputs the raw logits and they are passed to NLL Loss in training. During inference you just need the probabilities so softmax will suffice.

You don't use different algorithms for training and testing, the question is quite misleading as currently stated. You are using different implementations of the same algorithm.

You can find more on the matter here:

P.S: I remember encountering the same question on a Udacity scholarship. If you 're reading code from one of their courses, there is probably an explanation on the solution jupyter.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.