Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [metric]

A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.

Filter by
Sorted by
Tagged with
1 vote
0 answers
8 views

Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?

I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
user avatar
1 vote
1 answer
17 views

the error value of two model is different, one has smallest MAE but another one have smallest MSE

I have two machine learning model, the model result different error value in MAE and MSE. M1 have smallest MAE, but M2 have smallest MSE. Can anyone explain to me, why this happen? I also add my ...
user avatar
  • 21
0 votes
0 answers
14 views

How do I compute the Weighted average ROC Curve?

So i have a multiclass problem and successfully computed the micro and macro average curves, how do I calculate the weighted value for each TPR and FPR?
user avatar
0 votes
1 answer
20 views

What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
user avatar
  • 320
1 vote
1 answer
20 views

specificity for 3 class

I was reading an answer in qoura to calculate the specificity of a 3 class classifier from a confusion matrix. In the below answer https://www.quora.com/How-do-I-get-specificity-and-sensitivity-from-a-...
user avatar
1 vote
0 answers
8 views

Normalization and Denormalization

I have few queries. 1) Is normalization required for ANN / CNN /LSTM ? 2) If we normalize the data with MinMax Scaler, then in that case how to denormalize it and when to denormalize it so that we ...
user avatar
1 vote
0 answers
13 views

How can I calculate de AUC PR of my classifiers in a multiclass scenario?

I'm developing image classifiers in a context with 25k images and 50 classes. The dataset is imbalanced. Some papers recommend AUC PR for comparing the performance of my classifiers in this setting. ...
user avatar
  • 127
2 votes
0 answers
18 views

Why is NDCG high even for wrongly ranked predictions?

The NDCG (Normalized Discounted Cumulative Gain) metric for ranking is defined as DCG/IDCG, where IDCG is the ideal DCG and is said to take values in [0, 1]. However, since the DCG will always be ...
user avatar
  • 21
0 votes
1 answer
16 views

how to interpret precsion recall value in binary classification of scikit-learn

I am working with binary classification and my classification report generated through scikit-learn looks like the image below. I am confused I have two precision-recall values one for class 0 and the ...
user avatar
  • 175
0 votes
0 answers
9 views

Soft-clustering evaluation with multiple labels

I have an article clustering problem ( the articles are encoded with T5 so I technically have vectors) where each one can have multiple topics as labels(the set of labels is unbounded). I did soft ...
user avatar
0 votes
0 answers
10 views

How to generate a penalty matrix after KMeans clustering

Let say we have 9 observations that can be grouped into 3 groups using K-means clustering: ...
user avatar
  • 101
2 votes
3 answers
79 views

Measuring performance of customer purchase predictions

My goal is to develop a model that predicts next customer purchases in USD (Update: During the time period of the dataset, if no purchase was made by the customer, the next purchase label is set to ...
user avatar
0 votes
0 answers
36 views

Which error metric is best for financial returns?

I am trying to predict price change i.e most of the time values around zero (+ and -). In my backtest I predict only one period during test. I would like to know in each iteration which model was the ...
user avatar
1 vote
1 answer
33 views

Computing precision in case of multi-label classification

When evaluating a multi-label model for precision by averaging the precision of each sample, would it be appropriate to a) ignore those samples where no prediction is being made? Or is it more ...
user avatar
  • 11
1 vote
0 answers
28 views

How can the F_beta score with a search for the best threshold be implemented as a metric in tf.keras?

I'm trying to implement a custom metric in TensorFlow using the Keras API. The metric will be used on a binary classifier with one single output node. On a single evaluation, the metric is supposed to ...
user avatar
0 votes
1 answer
35 views

What happens to auc when true positive rate grows

How does change in true positive rate affects AUC? Does increase of TPR lead to increase of AUC as well?
user avatar
  • 133
-1 votes
1 answer
20 views

How to show combined overall accuracy for a multi-ouput model in Keras?

I have a model of the following structure. It has 6 outputs. Given an image, the model predicts classes of 6 different components from the image. The metrics I used are: As you can see it outputs an ...
user avatar
  • 1
0 votes
0 answers
13 views

Drop in Accuracy and Precision of a Classifier on Validation set?

We have seen a significant drop in Accuracy by 20% and in Precision by 45%, after running Classifier on the validation set. During training, we had used 5cv and throughout our tests, we haven't seen ...
user avatar
  • 21
0 votes
0 answers
29 views

What's an appropriate clustering quality estimate / metric for precomputed distance in HDBSCAN?

HBDSCAN supports estimation of clusters from precomputed distances. However, the python implementation of HDBSCAN (scikit-contrib) doesn't create minimum spanning trees in the absence of raw data when ...
user avatar
  • 1
0 votes
1 answer
30 views

Is it bad to use "coefficient of determination" for recommendation?

This is a general question about recommendation: Is it a bad idea to use "coefficient of determination"($R^2$) as metrics for recommendation? I am building a model of recommendation and ...
user avatar
  • 3
0 votes
1 answer
21 views

Specificity over 100

I am construction a deep neural network for a classification task, when I look at the metrics, I have a specificity of 1.04. Is it possible to have this metric over 100 ? How do you interpret it? ...
user avatar
1 vote
1 answer
29 views

Average of metrics using 10-fold

I'm working with 10 k-fold cross validation and I'm wanting to average the metrics, but I'm not getting it with sklearn. This is the way I am doing it and the metrics are being printed by fold. ...
user avatar
4 votes
1 answer
125 views

Choose ROC/AUC vs. precision/recall curve?

I am trying to get a clear understanding on various classification metrics, including knowing when to choose ROC/AUC as opposed to opting for the Precision/Recall curve. I am reading Aurélien Géron's ...
user avatar
  • 259
0 votes
1 answer
691 views

How could we interpret a SI Scatter Index and RMSE?

SI is RMSE divided by the average value of the observed values (or the predicted values? am confused)? is SI = 25% acceptable? (is the model good enough? )
user avatar
  • 1
0 votes
1 answer
542 views

Which Keras metric for multiclass classification

I have a multiclass classification data where the target has 11 classes. I am trying to build a Neural Net using Keras. I am using softmax as activation function ...
user avatar
  • 1,241
0 votes
0 answers
19 views

How can i adapt accuracy metric for multiclass classification?

I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
user avatar
  • 466
1 vote
1 answer
34 views

Should I apply Softmax before calculating metrics Precision or similar?

I am using PyTorch Lightning (there is no tag for this and I don't have enough reputation to create one) and am facing a multi classification problem. My loss function is ...
user avatar
  • 63
0 votes
1 answer
206 views

val_sparse_categorical_accuracy

I know the metric sparse_categorical_accuracy ...
user avatar
  • 3
2 votes
0 answers
216 views

How to identify precision, recall, IoU, and mAP in these results for my trained Tensorflow model?

I have trained a Single Shot Detector model (using Tensorflow), and have run the evaluation metrics. However, I am not entirely sure what to make of them. Doing a computer vision literature search, ...
user avatar
  • 141
0 votes
0 answers
7 views

Why does the summaryFunction data only returns 10 rows with custom metric (caret trControl)

I was trying to generate my own F1 metric, however I am wondering why I only get 10 rows for my prediction in the data parameter. Can somebody please clarify were it doesn't return me all predictions ...
user avatar
0 votes
0 answers
15 views

What are the evaluation metrics for evaluating testing data

I am doing a project using T5 Transformer. I have read documentations related to T5 Transformer model. The aim of the project is to generate Job Description based on Job_Role and Skills. My concern ...
user avatar
  • 33
0 votes
1 answer
91 views

Is sensitivity the same as recall in multiclass classification?

In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
user avatar
1 vote
0 answers
11 views

Similarity between binary vector with hierarchal structure

I have dataset of binary vectors, where each vector composed from several small vector coming from a different parent category. Each of those categories has a different size e.g. ...
user avatar
  • 111
0 votes
0 answers
13 views

Creating a metric to compare models based on score and time

I have several models from which I need to choose the 'best' one. I am trying to find a metric that can define mathematically what 'best' means. The two parameters to be considered are ...
user avatar
  • 11
1 vote
3 answers
89 views

RMSE vs R-squared

Question: Which is a better metric to compare different models RMSE or R-squared ? I searched a bit usually all the blogs say both metrics explain a different idea, R-squared is a measure of how much ...
user avatar
0 votes
0 answers
8 views

Is there a way to add custom (virtually all popular) metrics to BinaryClassificationEvaluator from pyspark MFlow library?

I am trying to make runs with mlflow from pyspark and I would like to obtain multiple metrics from that (not only ones suggested in the documentation which are areaUnderROC and areaUnderPR).
user avatar
1 vote
1 answer
14 views

'Collision' resolution for precision in object detection

For object detection we often use metrics based on precision/recall. My question is what is generally the process of matching the prediction and ground truth bound boxes, when there are multiple ...
user avatar
0 votes
0 answers
25 views

Model comparison: how to explain worse (lower) dice scores but better (lower) Hausdorff distances

I have two segmentation models (U-Net-like architectures): an original model, and an experimental model. I use the dice score and 95% Hausdorff distance to evaluate their performance. Using the first ...
user avatar
  • 101
0 votes
0 answers
9 views

Use A Distribution To Measure Customer Success

I am working at a marketplace startup and one of our success metrics is average number of transactions. To better represent customer success I suggested that we leverage a distribution to measure the ...
user avatar
0 votes
0 answers
8 views

How to incorporate non-present classes when calculating Micro Precision for Non-binary Classification Tasks

I have conducted multiple experiments with my algorithms for non-binary classification. In the most datasets, not all classes in which the algorithm can put the instances are present but only around ...
user avatar
0 votes
0 answers
27 views

How to interpret different coherence values

For an experiment with topic models, I have calculated four coherence values using Gensim's implementation: c_v u_mass c_uci c_npmi From this paper, I know that c_v correlates mostly with human ...
user avatar
  • 189
1 vote
0 answers
12 views

No Recall metric in MLEval library in Python

I am exploring different AutoML libraries in Python. Found MLEval from Alteryx. When I try to use this tutorial, I have an interesting result. I was trying to add ...
user avatar
1 vote
1 answer
26 views

How to interpret the Precision Recall AUC

The ROC AUC has an intuitive interpretation: the probability that the score of a randomly sampled 1-labeled item will be higher than a randomly sampled 0-labeled item. Is there a similar ...
user avatar
  • 111
0 votes
0 answers
72 views

Testing metrics and How to test MLM Bert Models?

I am doing a project with MLM models with bert where i mask parts of sentences and try to predict them. And then i try to check the similarity of the word with the original word. The sentences are ...
user avatar
1 vote
2 answers
126 views

If we dont specify any distance in KNN model, how is n_neighbors parameter calculated?

If we don’t specify the distance, how is the n_neighbors calculated?
user avatar
1 vote
0 answers
25 views

Recall, Precision and f2 compute in python tensorflow

How can I compute precision,f2 and recall values in my code? Please find my code below: <...
user avatar
0 votes
1 answer
56 views

What metric should I use to achieve perfect score when choosing all possible results?

A guy told me that he can predict which player I would choose from Greece's Euro 2004 Champion football team. Assume my choice was random. He then goes ahead and names all the players of the team. He ...
user avatar
  • 291
0 votes
1 answer
15 views

How to compare error metrics for model with and without Seasonality?

I am aiming to guage the difference in my model performance from using data with and without Sesonality removal. My approach to Seasonality removal is taking the log of the column data and then ...
user avatar
0 votes
0 answers
78 views

Recall metric per class in Keras

For evaluation of an image classifier and to monitor the model's performance during training I want to compute (among other metrics) the recall for each class. Following the Keras metrics ...
user avatar
1 vote
1 answer
20 views

Using Z-test score to evaluate model performance

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model? I have a binary model that ...
user avatar

1
2 3 4 5