Questions tagged [metric]
A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.
249
questions
0
votes
0
answers
28
views
What if many different models reach the same maximum metrics
I am talking about trying different algorithms, different parameters, different stacking configurations that all improve upon previous baselines, and yet a lot of them have exactly the same values on ...
- 636
3
votes
2
answers
729
views
Precision, recall and importance of them in the imbalance problem
I have a test dataset. The dataset is an imbalanced dataset. The total training instances for the dataset is 543 among them minority class(yes) is 75 and the majority class(No) is 468. The class of ...
- 359
1
vote
1
answer
416
views
Select threshold (cut-off point )for binary classification by desired fpr persentage value
I want to recreate catboost.utils.select_threshold(desc) method for CalibratedClassifierCV model.
In Catboost I can select ...
- 131
1
vote
1
answer
235
views
Average precision, balanced accuracy, F1-score, Matthews Correlation Coefficient, geometric means
Average precision, balanced accuracy, F1-score, Matthews Correlation Coefficient, geometrics means are the few evaluation metrics for imbalanced data. However, all this metrics can lead to different '...
- 13
1
vote
0
answers
1k
views
Equal error rate for multiclass (non-binary) classifier
In many biometrics identification papers they measure they performance by computing Equal Error Rate (EER). When dealing with verification problem, or any other binary classification problem - the ...
- 123
1
vote
1
answer
91
views
How to test the confidence for a rule based system?
I have a multi-class dataset and I generated based on it rules. That is, if certain features are seen then it must be a certain class. I chose only rules with precision 1 (with respect to the whole ...
- 131
1
vote
1
answer
57
views
MAE divided by median metric
I have a regression task for which my best models has a Mean Absolute Error (MAE) of approximately 15,000. The median value of the target variable is approximately 150,000.
I want to report that the ...
- 67
1
vote
1
answer
1k
views
Calculate RMSE based on R squared and vice versa
If for example I have the value of RMSE can I calculate the $R^2$? And vice versa if I have the value of $R^2$ can I calculate the value of RMSE?
I have all predictions, dataset, training set, and ...
0
votes
1
answer
593
views
precision and recall at k for movielens dataset
I wanted to recreate a very simple collaborative filtering example with the 1M movielens dataset I have from Kaggle (https://www.kaggle.com/datasets/odedgolden/movielens-1m-dataset) and then ...
- 183
0
votes
0
answers
245
views
XGBClassifier's predictions are not probabilities with objective='binary:logistic'
I am using a XGBoost's XGBClassifier, a binary 0-1 target, and I am trying to define a custom metric function. It supposedly receives an array of predictions and a DMatrix with the training set ...
- 127
1
vote
0
answers
11
views
Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?
I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
1
vote
1
answer
31
views
the error value of two model is different, one has smallest MAE but another one have smallest MSE
I have two machine learning model, the model result different error value in MAE and MSE. M1 have smallest MAE, but M2 have smallest MSE.
Can anyone explain to me, why this happen?
I also add my ...
- 21
0
votes
0
answers
326
views
How do I compute the Weighted average ROC Curve?
So i have a multiclass problem and successfully computed the micro and macro average curves, how do I calculate the weighted value for each TPR and FPR?
- 11
0
votes
1
answer
31
views
What metrics work well in unbalanced assemblies?
I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
- 314
1
vote
1
answer
2k
views
specificity for 3 class
I was reading an answer in qoura to calculate the specificity of a 3 class classifier from a confusion matrix. In the below answer
https://www.quora.com/How-do-I-get-specificity-and-sensitivity-from-a-...
- 125
1
vote
0
answers
53
views
Normalization and Denormalization
I have few queries. 1) Is normalization required for ANN / CNN /LSTM ? 2) If we normalize the data with MinMax Scaler, then in that case how to denormalize it and when to denormalize it so that we ...
- 11
1
vote
0
answers
29
views
How can I calculate de AUC PR of my classifiers in a multiclass scenario?
I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced.
Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
- 139
3
votes
0
answers
313
views
Why is NDCG high even for wrongly ranked predictions?
The NDCG (Normalized Discounted Cumulative Gain) metric for ranking is defined as DCG/IDCG, where IDCG is the ideal DCG and is said to take values in [0, 1].
However, since the DCG will always be ...
- 131
0
votes
1
answer
65
views
how to interpret precsion recall value in binary classification of scikit-learn
I am working with binary classification and my classification report generated through scikit-learn looks like the image below. I am confused I have two precision-recall values one for class 0 and the ...
- 171
2
votes
3
answers
107
views
Measuring performance of customer purchase predictions
My goal is to develop a model that predicts next customer purchases in USD (Update: During the time period of the dataset, if no purchase was made by the customer, the next purchase label is set to ...
0
votes
0
answers
61
views
Which error metric is best for financial returns?
I am trying to predict price change i.e most of the time values around zero (+ and -). In my backtest I predict only one period during test. I would like to know in each iteration which model was the ...
- 214
1
vote
1
answer
104
views
Computing precision in case of multi-label classification
When evaluating a multi-label model for precision by averaging the precision of each sample, would it be appropriate to a) ignore those samples where no prediction is being made? Or is it more ...
- 11
1
vote
0
answers
92
views
How can the F_beta score with a search for the best threshold be implemented as a metric in tf.keras?
I'm trying to implement a custom metric in TensorFlow using the Keras API. The metric will be used on a binary classifier with one single output node. On a single evaluation, the metric is supposed to ...
- 11
0
votes
1
answer
127
views
What happens to auc when true positive rate grows
How does change in true positive rate affects AUC? Does increase of TPR lead to increase of AUC as well?
- 183
-1
votes
1
answer
142
views
How to show combined overall accuracy for a multi-ouput model in Keras?
I have a model of the following structure. It has 6 outputs. Given an image, the model predicts classes of 6 different components from the image.
The metrics I used are:
As you can see it outputs an ...
- 1
0
votes
1
answer
40
views
Is it bad to use "coefficient of determination" for recommendation?
This is a general question about recommendation:
Is it a bad idea to use "coefficient of determination"($R^2$) as metrics for recommendation?
I am building a model of recommendation and ...
- 3
0
votes
1
answer
29
views
Specificity over 100
I am construction a deep neural network for a classification task, when I look at the metrics, I have a specificity of 1.04. Is it possible to have this metric over 100 ? How do you interpret it?
...
- 147
1
vote
1
answer
50
views
Average of metrics using 10-fold
I'm working with 10 k-fold cross validation and I'm wanting to average the metrics, but I'm not getting it with sklearn. This is the way I am doing it and the metrics are being printed by fold.
...
4
votes
1
answer
1k
views
Choose ROC/AUC vs. precision/recall curve?
I am trying to get a clear understanding on various classification metrics, including knowing when to choose ROC/AUC as opposed to opting for the Precision/Recall curve.
I am reading Aurélien Géron's ...
- 289
1
vote
1
answer
4k
views
How could we interpret a SI Scatter Index and RMSE?
SI is RMSE divided by the average value of the observed values (or the predicted values? am confused)?
is SI = 25% acceptable? (is the model good enough? )
- 11
1
vote
1
answer
5k
views
Which Keras metric for multiclass classification
I have a multiclass classification data where the target has 11 classes. I am trying to build a Neural Net using Keras. I am using softmax as activation function ...
- 2,065
0
votes
0
answers
28
views
How can i adapt accuracy metric for multiclass classification?
I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
- 526
1
vote
1
answer
510
views
Should I apply Softmax before calculating metrics Precision or similar?
I am using PyTorch Lightning (there is no tag for this and I don't have enough reputation to create one) and am facing a multi classification problem.
My loss function is ...
- 173
0
votes
1
answer
973
views
2
votes
0
answers
409
views
How to identify precision, recall, IoU, and mAP in these results for my trained Tensorflow model?
I have trained a Single Shot Detector model (using Tensorflow), and have run the evaluation metrics. However, I am not entirely sure what to make of them. Doing a computer vision literature search, ...
- 141
0
votes
1
answer
636
views
Is sensitivity the same as recall in multiclass classification?
In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
1
vote
0
answers
14
views
Similarity between binary vector with hierarchal structure
I have dataset of binary vectors, where each vector composed from several small vector coming from a different parent category.
Each of those categories has a different size e.g. ...
- 111
2
votes
3
answers
4k
views
RMSE vs R-squared
Question: Which is a better metric to compare different models RMSE or R-squared ?
I searched a bit usually all the blogs say both metrics explain a different idea, R-squared is a measure of how much ...
- 399
1
vote
1
answer
31
views
'Collision' resolution for precision in object detection
For object detection we often use metrics based on precision/recall. My question is what is generally the process of matching the prediction and ground truth bound boxes, when there are multiple ...
- 353
1
vote
0
answers
12
views
No Recall metric in MLEval library in Python
I am exploring different AutoML libraries in Python. Found MLEval from Alteryx.
When I try to use this tutorial, I have an interesting result. I was trying to add ...
- 111
1
vote
1
answer
47
views
How to interpret the Precision Recall AUC
The ROC AUC has an intuitive interpretation: the probability that the score of a randomly sampled 1-labeled item will be higher than a randomly sampled 0-labeled item.
Is there a similar ...
- 121
1
vote
2
answers
177
views
If we dont specify any distance in KNN model, how is n_neighbors parameter calculated?
If we don’t specify the distance, how is the n_neighbors calculated?
1
vote
0
answers
88
views
Recall, Precision and f2 compute in python tensorflow
How can I compute precision,f2 and recall values in my code?
Please find my code below:
<...
- 11
0
votes
1
answer
73
views
What metric should I use to achieve perfect score when choosing all possible results?
A guy told me that he can predict which player I would choose from Greece's Euro 2004 Champion football team. Assume my choice was random.
He then goes ahead and names all the players of the team.
He ...
- 291
0
votes
1
answer
22
views
How to compare error metrics for model with and without Seasonality?
I am aiming to guage the difference in my model performance from using data with and without Sesonality removal. My approach to Seasonality removal is taking the log of the column data and then ...
- 63
1
vote
1
answer
76
views
Using Z-test score to evaluate model performance
I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model?
I have a binary model that ...
- 2,089
1
vote
2
answers
154
views
If ROC is used to find a threshold, but AUC is threshold invariant, why use AUC?
Say I have a binary classifier. I calculate ROC to select an ideal threshold of say, 0.6.
Then, I look at the AUC.
But wait! If AUC doesn't change by selecting an 0.6 threshold, then what makes AUC ...
- 477
3
votes
0
answers
93
views
How do I make inference about test metrics for entire population from sample metrics?
Generally we calculate specific metrics for ML models on a test set (and we try to make that test set representative). I'm not clear on how to make inference about the same metrics for the population ...
- 299
2
votes
1
answer
28
views
Create new performance indicators (error metrics)
I am wondering if any of you happen to know of a procedure/approach/rationale to develop new performance indicators (error metrics) that can be used to evaluate the prediction capability (say, ...
- 121
1
vote
1
answer
368
views
Overall AUC higher than all "stratified" AUCs
For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox. The AUC on the test set as a whole is 0.8.
Gender is one of the model's features. So I decided to ...
- 147