1
$\begingroup$

I have created confusion matrix with python and I have gotten weird results that made me worried:

enter image description here

as you can see, I have value of two, and if I understand correct, normalized matrix cannot have value higher than 1. and also, some columns' sum is not 1.

this is how I created this matrix:

df_confusion = pd.crosstab(df['labels'], df['prediction'])
df_conf_norm = df_confusion / df_confusion.sum(axis=1)

and then I just plot it.

my question is - is it possible to get value of 2? does the sum of column make sense not to be 1? do I have mistake? and if correct, what can explain that?

EDIT enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ Please print a few rows of - labels and prediction. $\endgroup$
    – 10xAI
    Commented Jul 9, 2020 at 13:45
  • $\begingroup$ @10xAI I have jsut added $\endgroup$
    – Reut
    Commented Jul 9, 2020 at 18:33

1 Answer 1

Reset to default
1
$\begingroup$

Switch the axes:

# Build the table
confusion_table = pd.crosstab(df["prediction"], df["labels"])

# Normalise
confusion_table = (confusion_table / consufion_table.sum(0))

# Verify
print(confusion_table.sum())

Alternatively, you can play with confusion_table.divide and the axis argument: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.divide.html

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thankyou for your answer, I did it and now the sum is ~1, like I get 1.01 or 0.99 , do you know why that can happen? $\endgroup$
    – Reut
    Commented Jul 12, 2020 at 9:30
  • $\begingroup$ Mathematically, it should not happen. What can happen is that you have small differences because of the floating point but it's more of the order of 1e-9 than 1e-2. You can also use the function confusion_matrix provided by sklearn.metrics, that way you are certain that it's correct $\endgroup$
    – qmeeus
    Commented Jul 12, 2020 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.