Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [metric]

A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

How to choose segment in Grouped AUC metric?

Background In Binary Classification, AUC is a common metric. However, Group-AUC performs better in some scenario, such as we use AUC grouped by user in recommendation systems. In the below examples, I ...
Travis's user avatar
  • 111
0 votes
0 answers
10 views

Keras siamese model history is empty

I am making a siamese neural network with triplet loss using keras, and have encountered an odd problem. I tried saving my history twice: once in a callback (saved as a dictionary), and once after ...
Rotem Ton's user avatar
0 votes
0 answers
12 views

A/B test question - How to test significance for metrics that are not the unit of randomization

We're runnning an AB test on an ecommerce website. The feature being launched is not for the "users" that come to buy products on the website but is rather for "suppliers" who add ...
helloworld's user avatar
0 votes
0 answers
18 views

What are the most important evaluation metrics for anomaly segmentation?

When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
Mosh Geb's user avatar
0 votes
0 answers
10 views

Commonly used metric in NLP literature to compare ranked weighted results with variable importance for top-k results

I have two different search engines that always return the same results but in different orders. The results consist of websites along with confidence scores, which range from 100 to 10,000. The ...
hanugm's user avatar
  • 157
0 votes
0 answers
18 views

How to make my validation plots more stable and improve R2 metric?

I'm working on predicting 4 numeric values basing on signal spectrum (spectrum is represented as an array of 800 numeric values in scale 0 to 1). The input values are scaled by using StandardScaler. ...
mkow93's user avatar
  • 1
1 vote
0 answers
51 views

Bad metrics results by strong class imbalance in Credit card classification

Hi i'm currently in the process of writing my bachelor's thesis and stuck at a some steps. I've developed a few ML-Model (XGBoost, (Balanced) Random Forest, ElasticNet,...) on an extreme imbalanced ...
user159373's user avatar
0 votes
0 answers
10 views

Which analog of F1 score metrics can I use in this case?

I am training a cnn segmentation model and I need some analog of F1 score So, we have GT as red rectangles (called "red") and Pred as blue rectangles (called "blue"). It is clear ...
sixtytrees's user avatar
0 votes
0 answers
22 views

How to read the "predicted_true" Metric of an Azure ML experiment?

I followed along to Explore Automated Machine Learning in Azure Machine Learning which had me create a regression experiment using data from https://aka.ms/bike-rentals (731 samples; 12 features; 1 ...
joseville's user avatar
  • 143
0 votes
0 answers
16 views

Calculating Readmission Metrics in Python

I need to compute some Hospital Readmission Variables using Python. I would need to compute the following metrics: Simple Readmission: Compute variables for different periods 3, 7, 14 30 and 45 days ...
Carmen Morales's user avatar
0 votes
0 answers
7 views

How to label a dataset of text pairs to use it as a universal one for calculating the precision@k metric for different models?

I am facing a semantic search problem. I am fine tuning different NLU models and i want to use precision@k as my main metric. Is it possible to label a dataset of text pairs to use it as a universal ...
Ir8_mind's user avatar
  • 183
0 votes
0 answers
54 views

Is this the appropriate way to calculate a multiclass reliability diagram for model calibration?

I'm trying to generalize reliability diagrams [1] to a multiclass classifier and implement that using pytorch and pytorch-metrics. So far so good but I'm somewhat confused about the definition of ...
Nirro's user avatar
  • 101
1 vote
1 answer
30 views

Is it bad to average several MAEs calculated from chunks of a big test dataset?

In my regression problem, I am using Mean Absolute Error (MAE) as a metric for my network. My test dataset is too big to fit in memory, so I am reading the test dataset in chunks and then Keras' ...
ihavenoidea's user avatar
0 votes
0 answers
15 views

Survival analysis metric on time series data

I created a model that estimates the probability of failure of an asset (based on Weibull CDF, value between 0 and 1). I have a data point every minute. I want to measure the model's success based on ...
rvdinter's user avatar
1 vote
1 answer
102 views

Why is the sprase categorical accuracy decreasing every epoch and predictions are always NaN?

Problem Summary My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a ...
Joachim Rives's user avatar
0 votes
0 answers
96 views

Custom loss and metric functions including additional parameter in Keras

The following example is based on this approach. Similar to that approach, I am wanting to pass an additional parameter with y_true for my custom metric, as both will be used in the computation of ...
David's user avatar
  • 1
0 votes
1 answer
68 views

When is Recall@k useful for a classifier with softmax-like output?

If a 3-class classifier returns a length-3 vector of probabilities, e.g. [0.1, 0.85, 0.05] for classes A, B, and C respectively (strongly indicating B), does it ...
Alex Shroyer's user avatar
0 votes
1 answer
120 views

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why. ...
Adnan Ali's user avatar
  • 101
0 votes
0 answers
30 views

Expected Calibration Error vs Cross-Entropy Loss

When would I use Expected Calibration Error over Cross Entropy Loss. I think I understand when to use Cross Entropy Loss i.e. it is easy to optimize Cross Entropy Loss during training. But can I not ...
RAbraham's user avatar
  • 187
1 vote
0 answers
108 views

Is there an elegant way to quantify the mix shift effect?

Imagine there is an online auction business selling 1000+ varieties of fruits. The demand for these fruits change over time, and the prices of these fruits is set by the seller. Supposed I am ...
tanvach's user avatar
  • 111
0 votes
1 answer
254 views

Best metrics to evaluate the performance of a regression model?

I've just started with machine learning and I have a lot to learn but one of the recent problems I'm facing is evaluating the performance of a regression model. I know about MSE, RMSE, MAE ...
Harshal R's user avatar
2 votes
1 answer
327 views

Implementation of spBLEU

I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
Prithvi's user avatar
  • 33
0 votes
0 answers
16 views

Does scikit learns implementation of silhouette score support parallelization and will benefit from multiple CPUs?

I wish to use the silhuette score to get the optimum number of clusters. I know kmeans implementation in scikit learn supports parallelization. But I am unsure whether the same is true for silhouette ...
Ali Raheel's user avatar
0 votes
1 answer
107 views

Standard metric for distance between two clusters

Let $A=\{A_1,A_2,\cdots,A_m\}$ and $B=\{B_1,B_2,\cdots,B_n\}$ be two sets of points in $k$-dimensional Euclidean space. Each points $A_i$ or $B_i$ can be thought of as a feature vector of a data ...
govindah's user avatar
0 votes
1 answer
296 views

MPE (Most Probable Explanation) vs. MAP (Maximum A Posteriori)

What is MPE? How do MPE and MAP differ? Any example of when they would produce different results?
randomvariable's user avatar
0 votes
0 answers
89 views

Binary classification metrics for one-hot label encoding in Tensorflow

I run a binary classification using different CNN versions in Tensorflow. When I label samples from each class using 0 and 1, I select a sigmoid output in the last layer of the CNN, like ...
GKH's user avatar
  • 101
0 votes
0 answers
37 views

Name of metric: percentage of K for 100% recall

I have a recommendation system problem where full recall is important. Thus, the standard metrics of recall@k is insufficient. Rather, what I want to measure is how much of the recommendations must be ...
Siddharth Bhat's user avatar
0 votes
1 answer
21 views

Best measure to inform how predicted value can differ from real one

I have trained a regression model and obtained a pandas series of the predicted values. I am working on a "calculator" that will be able to return a predicted value after entering an input ...
Paulina's user avatar
  • 15
2 votes
2 answers
1k views

Some simple questions about confusion matrix and metrics in general

I will first tell you about the context then ask my questions. The model detects hate speech and the training and testing datasets are imbalanced (NLP). My questions: Is this considered a good model? ...
Maxi's user avatar
  • 89
4 votes
3 answers
823 views

Which metric to use for imbalanced data in TensorFlow/Keras

I am doing a binary classification task with Keras and my model directly outputs either 0 or 1. Typically I compile the model like something below: ...
D.J. Elkind's user avatar
0 votes
1 answer
1k views

How to correctly measure the inference time and FLOPs of a model?

For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
Shadow_fiend's user avatar
1 vote
1 answer
144 views

Questions on reproducibility of TimeGAN results

I am playing the timeGAN model, using the example code from ydata-synthetic repo. To train the model, we used synth.train(stock_data, train_steps=50000) to ...
TripleH's user avatar
  • 157
0 votes
0 answers
89 views

Performance metrics for outlier/anomaly detection

I am currently seeking for metrics i can use to evaluate a model for outlier/anomaly detection without ground truth. The only thing i came up with for now is to use the scores/probas returned by my ...
Raphael's user avatar
1 vote
1 answer
49 views

Order of preproccesing, avoiding leakage and metrics

I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess. This is what I did: Performed an EDA which ...
pustelnikk's user avatar
1 vote
1 answer
104 views

Why does BLEU score for ignite, torchmetrics and nltk differs?

Here is the example : ...
amine ammor's user avatar
0 votes
1 answer
35 views

How to aggregate the metrics from two different regression problems?

I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a ...
Mehran's user avatar
  • 277
1 vote
1 answer
343 views

YOLO : why does changing the confidence threshold change the [email protected]?

I trained a YOLOv7 model for a detection task. I have only one class, which is the object I want to detect. I ran test.py with --conf-thresh to 0.001 (default) and a second time with --conf-thresh to ...
Quintino's user avatar
0 votes
1 answer
37 views

Get dependant probabilities in multiclassification

After training my CatBoostClassifier model I call get_proba function which returns me list of probabilities. The problem starts from an another point... I transfer that data into dataframe then to ...
Master_Sniffer's user avatar
1 vote
2 answers
152 views

How to evaluate Natural Question-Answer Generation pairs?

I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to ...
Aaditya ura's user avatar
2 votes
1 answer
414 views

Why is accuracy not a useful measure for information retrieval problems?

I have been studying about information retrieval and recommender systems. While reading about it I found that accuracy not a useful measure in information retrieval. I understand that, accuracy might ...
ilved17's user avatar
  • 41
0 votes
1 answer
39 views

Success metric of database migration using row counts

Description I have a problem where I'm tasked to successfully transform and repurpose data from one SQL server to another. Call the source $\text{src}$ and the target database $\text{tgt}$. In order ...
NoVariation's user avatar
3 votes
1 answer
4k views

Accuracy vs Categorical Accuracy

I was running a DNN model that uses ResNet50 for Transfer Learning. While fitting the training data on my model to check the initial trend (would run for more epochs if initial trend seems right), I ...
Harsh Khare's user avatar
3 votes
1 answer
89 views

What does precision-recall curve and ROC curve tell us abouth threshold invariance

Consider a binary classification problem. Intuitively, a value for the area under the curve (for both curves) very close to 1, shows that the curve is almost L-shaped. Thus, this means that the value ...
liakoyras's user avatar
  • 636
-1 votes
2 answers
208 views

Classification model- which metric to choose to evaluate

If my project is "Music Genre Classification" Which metric method I need to choose to evaluate, and why ? Thanks a lot
yuvi's user avatar
  • 9
1 vote
1 answer
188 views

About the Evaluation method of the Market 1501 ReID dataset

The market 1501 dataset has train, query and gallery folders, each containing multiple views of people from multiple cameras. I would like to understand how to evaluate a model (trained with triplet ...
Anamaki's user avatar
  • 111
0 votes
0 answers
60 views

My overfitted decision tree regressor gives better result than pre-pruned tree?

I create a decision tree regressor without giving any parameters. The resulting tree has 6255 leaf nodes (out of 6348 entries of train set) and depth of 39. Most probably it has overfitted. But its ...
Akrobeto's user avatar
1 vote
0 answers
103 views

Calculationg perplexity (in natural language processing) manually

I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following ...
Piskator's user avatar
  • 135
0 votes
1 answer
16 views

Statistic methods to compute the average of a list of metric results

I have a Machine Learning model that fits and predicts many time series at once, so, for each time series I have a metric result, for example, MAE. What I need is to generate an unique value for that ...
Gabriel Caldas's user avatar
0 votes
0 answers
60 views

Am I calculating LogAUC / pAUC correctly?

Hope you are well. I was wondering how one would calculate logAUC? I have an implementation but I don't think it's correct. I'm trying to recreate the metric in this manuscript. See figure 2. Any help ...
James Arthur's user avatar
2 votes
1 answer
195 views

What makes an ROC curve a curve and why do the values change?

I have a problem. I am currently looking at a classifier and I would like to examine this using an ROC curve as a metric. However, questions have arisen to which I can not find an answer. A ROC curve ...
Test's user avatar
  • 89

1
2 3 4 5