2
$\begingroup$

I have created confusion matrix with python and I have gotten weird results that made me worried:

enter image description here

as you can see, I have value of two, and if I understand correct, normalized matrix cannot have value higher than 1. and also, some columns' sum is not 1.

this is how I created this matrix:

df_confusion = pd.crosstab(df['labels'], df['prediction'])
df_conf_norm = df_confusion / df_confusion.sum(axis=1)

and then I just plot it.

my question is - is it possible to get value of 2? does the sum of column make sense not to be 1? do I have mistake? and if correct, what can explain that?

EDIT enter image description here

$\endgroup$
  • $\begingroup$ Please print a few rows of - labels and prediction. $\endgroup$ – 10xAI Jul 9 at 13:45
  • $\begingroup$ @10xAI I have jsut added $\endgroup$ – Reut Jul 9 at 18:33
1
$\begingroup$

Switch the axes:

# Build the table
confusion_table = pd.crosstab(df["prediction"], df["labels"])

# Normalise
confusion_table = (confusion_table / consufion_table.sum(0))

# Verify
print(confusion_table.sum())

Alternatively, you can play with confusion_table.divide and the axis argument: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.divide.html

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thankyou for your answer, I did it and now the sum is ~1, like I get 1.01 or 0.99 , do you know why that can happen? $\endgroup$ – Reut Jul 12 at 9:30
  • $\begingroup$ Mathematically, it should not happen. What can happen is that you have small differences because of the floating point but it's more of the order of 1e-9 than 1e-2. You can also use the function confusion_matrix provided by sklearn.metrics, that way you are certain that it's correct $\endgroup$ – qmeeus Jul 12 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.