0
$\begingroup$

Lets say i have a tensor of order 256, dimensions indexed from 0 to 255.

Lets say i am writing a function implementing the softmax operation because i am a newbie and i want to understand the fundamental principles of machine learning.

Lets say the softmax is to be applied to dimensions 2 51 and 139.

Will the final result stay the same, regardless of whether i do softmax(2,51,139) or softmax(51,2,139) or softmax(139,51,2)?

Disregard the obvious problem of floating point operations not being really commutative.

$\endgroup$
1
  • $\begingroup$ Hi @Maciej, welcome to the site. If you find the answers to your question useful, please consider upvoting them. Also, please consider accepting one (with the tick mark ✓ next to it) if you consider it correct or, alternatively, please describe in a comment why you consider it incorrect or not clear enough. $\endgroup$
    – noe
    Apr 1 at 11:11

1 Answer 1

1
$\begingroup$

Softmax is applied along a single dimension, not along many dimensions.

For instance, in the Pytorch docs you can see that softmax receives param dim to specify the dimension over which it is:

Parameters

dim (int) – A dimension along which Softmax will be computed (so every slice along dim will sum to 1).

If you are referring to applying softmax sequentially over different dimensions, then the answer is that the order certainly matters. In the final result, only the dimension to which the softmax was applied last would be normalized.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.