Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [metric]

A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.

Filter by
Sorted by
Tagged with
1 vote
1 answer
54 views

Average of metrics using 10-fold

I'm working with 10 k-fold cross validation and I'm wanting to average the metrics, but I'm not getting it with sklearn. This is the way I am doing it and the metrics are being printed by fold. ...
0 votes
0 answers
11 views

How to choose segment in Grouped AUC metric?

Background In Binary Classification, AUC is a common metric. However, Group-AUC performs better in some scenario, such as we use AUC grouped by user in recommendation systems. In the below examples, I ...
1 vote
1 answer
188 views

About the Evaluation method of the Market 1501 ReID dataset

The market 1501 dataset has train, query and gallery folders, each containing multiple views of people from multiple cameras. I would like to understand how to evaluate a model (trained with triplet ...
1 vote
2 answers
2k views

Can Precision-Recall be improved for imbalanced sample?

I tried out a few models on a highly imbalanced sample (~2:100) where I can get decent AUC from ROC (test sample). But when I plot precision-recall (test sample), it looks horrible. Kind of like the ...
1 vote
1 answer
305 views

How to explain a stable NDCG@K in extreme multilabel recommender model

I am working in a multilabel recommender project and I try to evaluate it as a ranking problem. I calculate recall@k and precision@k which both looks quite well. Recall increases and Precision ...
1 vote
1 answer
377 views

Click Through Rate calculation (CTR) calculation problem

So I'm doing a use case for a company interview and one of the questions is to calculate the CTR for a sorting algorithm. My question would be: Should I remove the operations where there were no ...
2 votes
1 answer
1k views

Metrics for presenting RNN/LSTM result

I am working on two different architectures based on the LSTM model to predict the user's next action based on the previous actions. I am wondering, what is the best way to present the result? Is it ...
0 votes
1 answer
675 views

Is sensitivity the same as recall in multiclass classification?

In Wikipedia, it is stated "In binary classification, recall is called sensitivity" under the Recall section. Are they both different in case of multi-class classification?
0 votes
0 answers
10 views

Keras siamese model history is empty

I am making a siamese neural network with triplet loss using keras, and have encountered an odd problem. I tried saving my history twice: once in a callback (saved as a dictionary), and once after ...
0 votes
0 answers
12 views

A/B test question - How to test significance for metrics that are not the unit of randomization

We're runnning an AB test on an ecommerce website. The feature being launched is not for the "users" that come to buy products on the website but is rather for "suppliers" who add ...
1 vote
2 answers
137 views

How to measure accuracy of a route prediction

I developed a new route prediction algorithm and I am trying to find a metric that informs on how well a prediction was. This metric is meant to be used offline, meaning that the goal is not to ...
2 votes
1 answer
254 views

How to Monitor ML Classification Models in production?

I've often heard of measures like Population Stability Index and Characteristic Stability Index. I might be mistaken, but these seem to be more applicable towards looking at the changes in univariate ...
1 vote
1 answer
72 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
0 votes
0 answers
18 views

What are the most important evaluation metrics for anomaly segmentation?

When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
1 vote
1 answer
258 views

Which error metric is good for measuring accuracy

I am estimating water depth with satellite data (predicted value) and would like to validate my result using bathymetry lidar data collected on the field and believed to be more accurate (observed ...
1 vote
1 answer
348 views

Loss and Metrics for COCO Keypoints

I am using Keras to train different models on the COCO keypoints dataset. All of the models I am working with are used for image segmentation, so they output heatmaps corresponding to the labels. All ...
0 votes
1 answer
186 views

What methods are available to evaluate similarity between different clustering algorithms?

I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure ...
1 vote
1 answer
78 views

Using Z-test score to evaluate model performance

I think I know the answer to this question but I am looking for a sanity check here: Is it appropriate to use z-test scores in order to evaluate the performance of my model? I have a binary model that ...
0 votes
1 answer
107 views

Standard metric for distance between two clusters

Let $A=\{A_1,A_2,\cdots,A_m\}$ and $B=\{B_1,B_2,\cdots,B_n\}$ be two sets of points in $k$-dimensional Euclidean space. Each points $A_i$ or $B_i$ can be thought of as a feature vector of a data ...
1 vote
1 answer
263 views

Metric (other than RMSE, MSE, etc.) to select the best model to better detect peaks in weather forecasting

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, the second model (RMSE = 0.15) was better able ...
2 votes
1 answer
379 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
0 votes
1 answer
220 views

How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
0 votes
0 answers
10 views

Commonly used metric in NLP literature to compare ranked weighted results with variable importance for top-k results

I have two different search engines that always return the same results but in different orders. The results consist of websites along with confidence scores, which range from 100 to 10,000. The ...
1 vote
1 answer
343 views

YOLO : why does changing the confidence threshold change the [email protected]?

I trained a YOLOv7 model for a detection task. I have only one class, which is the object I want to detect. I ran test.py with --conf-thresh to 0.001 (default) and a second time with --conf-thresh to ...
44 votes
6 answers
63k views

What is the relationship between the accuracy and the loss in deep learning?

I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following: ...
0 votes
1 answer
296 views

MPE (Most Probable Explanation) vs. MAP (Maximum A Posteriori)

What is MPE? How do MPE and MAP differ? Any example of when they would produce different results?
0 votes
1 answer
120 views

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why. ...
1 vote
1 answer
260 views

Average Precision if Target Class is Not in Evaluation

Suppose I have 5 classes, denoted by 1, 2, 3, 4, and 5, and this is used in object detection. When evaluating an object detection performance, suppose I have classes 1, 2, and 3 present, but classes 4 ...
0 votes
0 answers
18 views

How to make my validation plots more stable and improve R2 metric?

I'm working on predicting 4 numeric values basing on signal spectrum (spectrum is represented as an array of 800 numeric values in scale 0 to 1). The input values are scaled by using StandardScaler. ...
0 votes
1 answer
1k views

How to correctly measure the inference time and FLOPs of a model?

For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
0 votes
1 answer
316 views

How to compare performance between SVM and Keras models

I applied both SVM and CNN (using Keras) on a dataset. Now, I want to compare the performance of both models. Keras model.evaluate function predicts the output for the given input and then computes ...
0 votes
1 answer
2k views

SKLEARN Metrics report "Number of classes, 28, does not match size of target_names, 35. Try specifying the labels parameter"

What's the proper way to define the labels or target names of classification_report? I have the report like this: ...
8 votes
3 answers
719 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
1 vote
0 answers
51 views

Bad metrics results by strong class imbalance in Credit card classification

Hi i'm currently in the process of writing my bachelor's thesis and stuck at a some steps. I've developed a few ML-Model (XGBoost, (Balanced) Random Forest, ElasticNet,...) on an extreme imbalanced ...
0 votes
0 answers
10 views

Which analog of F1 score metrics can I use in this case?

I am training a cnn segmentation model and I need some analog of F1 score So, we have GT as red rectangles (called "red") and Pred as blue rectangles (called "blue"). It is clear ...
0 votes
0 answers
22 views

How to read the "predicted_true" Metric of an Azure ML experiment?

I followed along to Explore Automated Machine Learning in Azure Machine Learning which had me create a regression experiment using data from https://aka.ms/bike-rentals (731 samples; 12 features; 1 ...
0 votes
0 answers
16 views

Calculating Readmission Metrics in Python

I need to compute some Hospital Readmission Variables using Python. I would need to compute the following metrics: Simple Readmission: Compute variables for different periods 3, 7, 14 30 and 45 days ...
0 votes
0 answers
7 views

How to label a dataset of text pairs to use it as a universal one for calculating the precision@k metric for different models?

I am facing a semantic search problem. I am fine tuning different NLU models and i want to use precision@k as my main metric. Is it possible to label a dataset of text pairs to use it as a universal ...
4 votes
0 answers
222 views

Fast PR / ROC curves and corespondings AUPR / AUROC

I find myself in a position of calculating numerous PR / ROC curves and their associated area under the PR curves (AUPR) / area under the ROC curve (AUROC). Its is quite easy to perform those ...
0 votes
0 answers
54 views

Is this the appropriate way to calculate a multiclass reliability diagram for model calibration?

I'm trying to generalize reliability diagrams [1] to a multiclass classifier and implement that using pytorch and pytorch-metrics. So far so good but I'm somewhat confused about the definition of ...
14 votes
3 answers
20k views

MAD vs RMSE vs MAE vs MSLE vs R²: When to use which?

In regression problems, you can use various different metrics to check how well your model is doing: Mean Absolute Deviation (MAD): In $[0, \infty)$, the smaller the better Root Mean Squared Error (...
1 vote
1 answer
30 views

Is it bad to average several MAEs calculated from chunks of a big test dataset?

In my regression problem, I am using Mean Absolute Error (MAE) as a metric for my network. My test dataset is too big to fit in memory, so I am reading the test dataset in chunks and then Keras' ...
1 vote
1 answer
102 views

Why is the sprase categorical accuracy decreasing every epoch and predictions are always NaN?

Problem Summary My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a ...
9 votes
5 answers
14k views

Cosine similarity vs The Levenshtein distance

I wanted to know what is the difference between them and in what situations they work best? As per my understanding: Cosine similarity is a measure of similarity between two non-zero vectors of an ...
0 votes
0 answers
15 views

Survival analysis metric on time series data

I created a model that estimates the probability of failure of an asset (based on Weibull CDF, value between 0 and 1). I have a data point every minute. I want to measure the model's success based on ...
0 votes
0 answers
96 views

Custom loss and metric functions including additional parameter in Keras

The following example is based on this approach. Similar to that approach, I am wanting to pass an additional parameter with y_true for my custom metric, as both will be used in the computation of ...
0 votes
1 answer
68 views

When is Recall@k useful for a classifier with softmax-like output?

If a 3-class classifier returns a length-3 vector of probabilities, e.g. [0.1, 0.85, 0.05] for classes A, B, and C respectively (strongly indicating B), does it ...
0 votes
0 answers
30 views

Expected Calibration Error vs Cross-Entropy Loss

When would I use Expected Calibration Error over Cross Entropy Loss. I think I understand when to use Cross Entropy Loss i.e. it is easy to optimize Cross Entropy Loss during training. But can I not ...
1 vote
0 answers
108 views

Is there an elegant way to quantify the mix shift effect?

Imagine there is an online auction business selling 1000+ varieties of fruits. The demand for these fruits change over time, and the prices of these fruits is set by the seller. Supposed I am ...
0 votes
1 answer
254 views

Best metrics to evaluate the performance of a regression model?

I've just started with machine learning and I have a lot to learn but one of the recent problems I'm facing is evaluating the performance of a regression model. I know about MSE, RMSE, MAE ...

1
2 3 4 5