# Tag Info

17

Definitions Accuracy: The amount of correct classifications / the total amount of classifications. The train accuracy: The accuracy of a model on examples it was constructed on. The test accuracy is the accuracy of a model on examples it hasn't seen. Confusion matrix: A tabulation of the predicted class (usually vertically) against the actual class (thus ...

15

To get a confusion matrix from the test data you should go througt two steps: Make predictions for the test data For example, use model.predict_generator to predict the first 2000 probabilities from the test generator. generator = datagen.flow_from_directory( 'data/test', target_size=(150, 150), batch_size=16, class_mode=...

8

You are just predicting if Play = Yes or Play = No. The confusion matrix would look like this: Predicted +------+------+ | Yes | No | +-------------------+ A | | | | c | Yes | TP | FP | t | | | | u +-------------------+ a | | | | l | No | FN | TN | | ...

7

Considering you have two lists y_actual and y_pred ( I assume you made a typo error on x_test and x_pred as in your code), you can pass the two lists to this function to parse them def perf_measure(y_actual, y_pred): TP = 0 FP = 0 TN = 0 FN = 0 for i in range(len(y_pred)): if y_actual[i]==y_pred[i]==1: TP += 1 ...

6

Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like: from sklearn.metrics import confusion_matrix y_true = ['Cat', 'Dog', 'Rabbit', 'Cat', 'Cat', 'Rabbit'] y_pred = ['Dog', 'Dog', 'Rabbit', 'Dog', 'Dog', 'Rabbit'] classes=['Cat', 'Dog', 'Rabbit'] ...

5

Maybe not the answer you would like to hear, but I have to admit that I find wikipedia page on the topic definitively well done... https://en.wikipedia.org/wiki/Precision_and_recall

5

Yes and No! depending on what do you mean by minimization. When you say minimizing $f$ and $g$ according to something, you are actually looking for a point which minimizes both. It does not mean that this point necessarily finds the minimum of $f$ or $g$. So yes in this sense. But in case you mean a point in which both of them are in their minimum, this ...

5

You can! The trick is that you actually know two other critical variables: the number of positive and negative examples (P and N). You can then use them to algebraically solve for the confusion matrix: $recall=\frac{TP}{TP+FN}=1-\frac{FN}{P}\Rightarrow$ $FN = P(1-recall)$ $recall=\frac{TP}{TP+FN}\Rightarrow (recall)(TP+FN)=TP\Rightarrow TP(1-recall)=... 4 You can apply a technique I described in my masters thesis (page 48ff) and called Confusion Matrix Ordering (CMO): Order the columns/rows in such a way, that most errors are along the diagonal. Split the confusion matrix into multiple blocks such that the single blocks can easily printed / viewed - and such that you can remove some of the blocks because ... 4 I do not know a standard answer to this, but I thought about it some times ago and I have some ideas to share. When you have one confusion matrix, you have more or less a picture of how you classification model confuse (mis-classify) classes. When you repeat classification tests you will end up having multiple confusion matrices. The question is how to get ... 4 There are a few ways to achieve your "master confusion matrix". Sum all the confusion matrices together: Like you suggested, summing this results in a confusion matrix. The problem with this is you can not interpret totals. Average the entries. This method is the same as number one, but you divide each entry by the number of trials (~400 in your case). ... 4 sklearn.metrics.classification_report provides precision and recall for all classes along with F-score and support. It might prove to be helpful in your case of 3 classes. 4 Adding to the answer above, The labeling totally depends on how you define it. You can define 0 as negative or as positive. However, for the sake of understanding and ease of readability, keep it meaningful. The instances that are correctly predicted are given by the diagonal. Here, '1' is True Negative or for the class labelled as 0 and '5' is True ... 4 A confusion matrix is a table that is often used to describe the performance of a classification model. The figure you have provided presents a binary case, but it is also used with more than 2 classes (there are just more rows/columns). The rows refer to the actual Ground-Truth label/class of the input and the columns refer to the prediction provided by ... 4 Seems like you understand the meaning of the confusion matrix, but not the logic used to name its entries! Here are my 5 cents: The names are all of this kind: <True/False> <Positive/Negative> | | Part1 Part2 The first part explains if the prediction was right or not. If you have only True Positive and True ... 4 If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an inverse relationship (@pythinker's answer), but for models worse than random, they have a direct relationship (@kbrose's example). It is worth noting that we can ... 3 Create a method that does the printing for you: def print_confusion_matrix(y_true, y_pred): cm = confusion_matrix(y_true, y_pred) print('True positive = ', cm[0][0]) print('False positive = ', cm[0][1]) print('False negative = ', cm[1][0]) print('True negative = ', cm[1][1]) And use it like this print_confusion_matrix(x_test, x_pred) ... 3 I would strongly recommend against using a confusion wheel to visualize your confusion matrix. As impressive and fancy as they look, confusion wheels are visually complicated and unintuitive to read. Good data visualizations summarize information in a way that is simple, clear, and intuitive. Confusion wheels have none of those properties. Unfortunately, ... 3 Please find the below: False Negative (FN): prediction is NEGATIVE, actual outcome is POSITIVE, result is 'False Negative' - Why is that? Shouldn't it be 'False Positive'? Answer : The predictive model supposed to give the answer as 'Positive', but it predicted as 'Negative', which means Falsely predicted as Negative aka False Negative. False Positive (FP):... 3 Thanks for clear statement of the problem. The point is that if you want to decrease false negatives, you should sufficiently lower the threshold of your decision function. If the false negatives are decreased, as you mentioned, true positives increase but false positives can also increase. As a result, recall will increase and precision will decrease. 3 You are correct @Tolga, both can increase at the same time. Consider the following data: Prediction | True Class 1.0 | 0 0.5 | 1 0.0 | 0 If you set your cut off point as 0.75, then you have $$Precision = \frac{TP}{TP + FP} = \frac{0}{0 + 1} = 0$$ $$Recall = \frac{TP}{TP + FN} = \frac{0}{0 + 1} = 0$$ then if you decrease your cut ... 3 In a multiclass problem there is one score for each class, counting any other class as a negative. For example for class 1: TP instances are gold standard class 1 predicted as class 1 FN instances are gold standard class 1 predicted as class 2,3 or 4 FP instances are gold standard class 2,3 or 4 predicted as class 1 TN instances are gold standard class 2,... 2 If you want to figure it out how this ROC happens , you would better LIST the tuples including your "predicted" values and the "truth" values, and SORT with the "predicted" value , then PLOT the ROC . In your case , the tuples and points should be like this : predicted truth (x,y) 0.53 0 (6/6,14/14) 0.55 0 (5/6,14/14) 0.57 1 (4/6,14/14) ... 2 if yHat are your predictions and yval are your y true then tp = sum((yHaT == 1) & (yval == 1)); fp = sum((yHaT == 1) & (yval == 0)); fn = sum((yHaT == 0) & (yval == 1)); precision = tp / (tp + fp); recall = tp / (tp + fn); F1 = (2 * precision * recall) / (precision + recall); 2 A confusion matrix gives you the following: [TP, FP] [FN, TN] where TP = 'true positives'; FP = 'false positives'; FN = 'false negatives'; TN = 'true negatives'. You can read more here: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/ By taking TP+TN and dividing by TP+FP+FN+TN, you can get the classification accuracy of your model. ... 2 I suggest PyCM lib for confusion matrix analysis. Example : >>> from pycm import * >>> y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] # or y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]) >>> y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] # or y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]) >>> cm = ... 2 Here is some code I tried and worked for me: pred= model.predict_generator(validation_generator, nb_validation_samples // batch_size) predicted_class_indices=np.argmax(pred,axis=1) labels = (validation_generator.class_indices) labels2 = dict((v,k) for k,v in labels.items()) predictions = [labels[k] for k in predicted_class_indices] print(... 2 from sklearn.metrics import recall_score If you then call recall_score.__dir__ (or directly read the docs here) you'll see that recall is The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives If you go down to where they define micro, it says 'micro': Calculate metrics ... 2 you can also use PyCM lib for multi-class confusion matrix analysis. Your Problem : >>> print(cm) Predict 0 1 2 Actual 0 1 0 0 1 0 1 2 2 0 1 0 Overall Statistics : 95% CI ... 2 1) It depends in what you define as positive and negative. Generally, and in particular in medicine, people tend to label$0$as negatives and$1$as positives, thus being$1\$ the abnormal case. But this is completely arbitraty, you can do as you wish. 2) 0 are always displayed in the first row and column. That is, your model has classified one 0 correctly ...

Only top voted, non community-wiki answers of a minimum length are eligible