0
$\begingroup$

Here's an example, between three classes a, b and c and their softmax probabilities from an imagined seq2seq algorithm.

In this case it's pretty obvious that class c is the most likely single label for the entire sequence (or, imagine, a labelled subsequence of 100 timepoints). But how likely, given softmax outputs? Should I just sum them for each class directly? Use the argmax? Something else, more correct?

In the case below, the probabilities are

softmax probabilities:  [ 7.98  9.99 82.03] 
sum:  100.0

argmax probabilities:  [ 3  6 91] 
sum:  100

While close, they're not quite the same.

enter image description here

import numpy as np
import matplotlib.pyplot as plt

def softmax(x):
    """Compute softmax values for each sets of scores in x."""
    e_x = np.exp(x - np.max(x, ))
    return e_x / e_x.sum(axis=0) # only difference

a = np.random.normal(1, 1, 100)
b = np.random.uniform(2, 1, 100)
c = np.random.exponential(3, 100) + np.linspace(0, 5, 100)

y_pred = np.column_stack((a, b, c))
y_pred = np.apply_along_axis(softmax, axis = 1, arr = y_pred)

argmax_class = y_pred.argmax(axis = 1)
argmax_class_top = np.bincount(argmax_class).argmax()
softmaxprob_sum = y_pred.sum(axis = 0).round(2)

print("softmax probabilities: ", softmaxprob_sum, "\nsum: ", sum(softmaxprob_sum))
print()
print("argmax probabilities: ", np.bincount(argmax_class), "\nsum: ", sum(np.bincount(argmax_class)))


fig, ax = plt.subplots(nrows = 2)
ax[0].plot(y_pred, label = "class softmax")
ax[0].set_ylabel("probability")
ax[0].plot(y_pred.sum(axis = 1), label = "sum")

ax[1].plot(argmax_class, label = "argmax")
ax[1].set_ylabel("class")
ax[1].set_yticks([0,1,2])

[ax.legend(loc = "upper right") for ax in ax]
plt.show()
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.