Questions tagged [metric]

A metric is a way to evaluate the performance of a machine learning model. Depending on the task, different metrics may be used.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Survival analysis metric on time series data

I created a model that estimates the probability of failure of an asset (based on Weibull CDF, value between 0 and 1). I have a data point every minute. I want to measure the model's success based on ...
rvdinter's user avatar
0 votes
1 answer
31 views

Why is the sprase categorical accuracy decreasing every epoch and predictions are always NaN?

Problem Summary My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a ...
Joachim Rives's user avatar
0 votes
0 answers
49 views

Custom loss and metric functions including additional parameter in Keras

The following example is based on this approach. Similar to that approach, I am wanting to pass an additional parameter with y_true for my custom metric, as both will be used in the computation of ...
David's user avatar
  • 1
0 votes
1 answer
22 views

When is Recall@k useful for a classifier with softmax-like output?

If a 3-class classifier returns a length-3 vector of probabilities, e.g. [0.1, 0.85, 0.05] for classes A, B, and C respectively (strongly indicating B), does it ...
Alex Shroyer's user avatar
0 votes
1 answer
30 views

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why. ...
Adnan Ali's user avatar
  • 101
0 votes
0 answers
16 views

Expected Calibration Error vs Cross-Entropy Loss

When would I use Expected Calibration Error over Cross Entropy Loss. I think I understand when to use Cross Entropy Loss i.e. it is easy to optimize Cross Entropy Loss during training. But can I not ...
RAbraham's user avatar
  • 187
1 vote
0 answers
44 views

Is there an elegant way to quantify the mix shift effect?

Imagine there is an online auction business selling 1000+ varieties of fruits. The demand for these fruits change over time, and the prices of these fruits is set by the seller. Supposed I am ...
tanvach's user avatar
  • 111
0 votes
1 answer
75 views

Best metrics to evaluate the performance of a regression model?

I've just started with machine learning and I have a lot to learn but one of the recent problems I'm facing is evaluating the performance of a regression model. I know about MSE, RMSE, MAE ...
Harshal R's user avatar
2 votes
1 answer
92 views

Implementation of spBLEU

I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
Prithvi's user avatar
  • 33
0 votes
0 answers
12 views

Does scikit learns implementation of silhouette score support parallelization and will benefit from multiple CPUs?

I wish to use the silhuette score to get the optimum number of clusters. I know kmeans implementation in scikit learn supports parallelization. But I am unsure whether the same is true for silhouette ...
Ali Raheel's user avatar
0 votes
1 answer
65 views

Standard metric for distance between two clusters

Let $A=\{A_1,A_2,\cdots,A_m\}$ and $B=\{B_1,B_2,\cdots,B_n\}$ be two sets of points in $k$-dimensional Euclidean space. Each points $A_i$ or $B_i$ can be thought of as a feature vector of a data ...
govindah's user avatar
0 votes
1 answer
88 views

MPE (Most Probable Explanation) vs. MAP (Maximum A Posteriori)

What is MPE? How do MPE and MAP differ? Any example of when they would produce different results?
randomvariable's user avatar
0 votes
0 answers
57 views

Binary classification metrics for one-hot label encoding in Tensorflow

I run a binary classification using different CNN versions in Tensorflow. When I label samples from each class using 0 and 1, I select a sigmoid output in the last layer of the CNN, like ...
GKH's user avatar
  • 101
0 votes
0 answers
31 views

Name of metric: percentage of K for 100% recall

I have a recommendation system problem where full recall is important. Thus, the standard metrics of recall@k is insufficient. Rather, what I want to measure is how much of the recommendations must be ...
Siddharth Bhat's user avatar
0 votes
1 answer
19 views

Best measure to inform how predicted value can differ from real one

I have trained a regression model and obtained a pandas series of the predicted values. I am working on a "calculator" that will be able to return a predicted value after entering an input ...
Paulina's user avatar
  • 15
2 votes
2 answers
427 views

Some simple questions about confusion matrix and metrics in general

I will first tell you about the context then ask my questions. The model detects hate speech and the training and testing datasets are imbalanced (NLP). My questions: Is this considered a good model? ...
FjkgB's user avatar
  • 89
0 votes
0 answers
19 views

torchmetrics BinaryMatthewsCorrCoef outputs 0 if target and prediction contains only one case either positive or negative case

I stared using MCC(Matthew's correlation coefficient) metric. But getting unexpected values when the given target and pred contains only one case either positive or negative (case - 1), The output of ...
lokesh's user avatar
  • 1
4 votes
3 answers
350 views

Which metric to use for imbalanced data in TensorFlow/Keras

I am doing a binary classification task with Keras and my model directly outputs either 0 or 1. Typically I compile the model like something below: ...
D.J. Elkind's user avatar
0 votes
1 answer
647 views

How to correctly measure the inference time and FLOPs of a model?

For some reason, I can’t find built-in solutions (not really?) in keras and tensorflow, while on the site https://keras.io/api/applications/ they provide Time (ms) per inference step (CPU), but for ...
Shadow_fiend's user avatar
1 vote
1 answer
110 views

Questions on reproducibility of TimeGAN results

I am playing the timeGAN model, using the example code from ydata-synthetic repo. To train the model, we used synth.train(stock_data, train_steps=50000) to ...
TripleH's user avatar
  • 157
0 votes
0 answers
17 views

Finding Look alike customers

I have a set of customers and their attributes (say spend/balance/etc.) per month. At month X something happen for some customers, but not all (call this set A). So, now I want to find customers who ...
user's user avatar
  • 1,973
0 votes
0 answers
25 views

SemEval-2016 Task 5 Subtask 2 Evaluation

Hi, I read SEMEVAL task 5 from this website (https://alt.qcri.org/semeval2016/task5/) but the evaluation method for subtask 2 is not clear. I really appreciate it if you tell me the method. This is ...
mansoor sh's user avatar
0 votes
0 answers
76 views

Performance metrics for outlier/anomaly detection

I am currently seeking for metrics i can use to evaluate a model for outlier/anomaly detection without ground truth. The only thing i came up with for now is to use the scores/probas returned by my ...
Raphael's user avatar
0 votes
0 answers
33 views

Do we have any range of values for MAP@K, which are considered as good to evaluate a recommendation system

I am using MAP@K as metric for the recommendation system I am building. Currently, I am seeing a value of 0.16, which I am not sure if can be considered as a good value. Any suggestions on what range ...
Nik's user avatar
  • 1
1 vote
1 answer
42 views

Order of preproccesing, avoiding leakage and metrics

I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess. This is what I did: Performed an EDA which ...
pustelnikk's user avatar
1 vote
1 answer
66 views

Why does BLEU score for ignite, torchmetrics and nltk differs?

Here is the example : ...
amine ammor's user avatar
0 votes
0 answers
27 views

Reporting and/or evaluation of metrics in deep learning

While I was trying to write a custom training script, I encountered the following doubt. I see that the loss is evaluated at the end of every forward pass (i.e., a step or with a particular batch of ...
Rajesh Nakka's user avatar
0 votes
0 answers
8 views

help with comparison between the prediction of a bayesian neural network and an analytical model

i am in a weird situation where i have a bayesian neural network used for regression and a polynomial model $f(a,b,c,d) that depends on 4 parameters and that is fitted through monte carlo methods. i ...
Alucard's user avatar
  • 111
0 votes
0 answers
11 views

Which F-metric to use in scientific paper?

Often, academic papers refer to F1-score, but don't explain which one they use (macro, micro, weighted). What is the common F-metric in academia?
CLRW97's user avatar
  • 101
0 votes
1 answer
29 views

How to aggregate the metrics from two different regression problems?

I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a ...
Mehran's user avatar
  • 267
0 votes
0 answers
14 views

Why are decision trees driven by the Gini impurity as opposed to the accuracy? [duplicate]

It seems that most implementations of decision trees use the Gini impurity as their partitioning criterion. Why isn't accuracy used instead, since it's a more widespread metric across different ...
Tfovid's user avatar
  • 203
0 votes
0 answers
67 views

How to calculate the gradient with triplet loss learning?

I have a CNN (convolutional neural network) that I train for face recognition. When teaching, I choose 3 images: Anchor, Positive, Negative. I pass each of them through CNN. Then I calculate the ...
Jan300385a's user avatar
0 votes
0 answers
7 views

Ages deviation in custom metric

I'm making custom metric for evaluation of genetic algorithm made to form teams of most appropriate people. I.e. I'd wanted to be people from one city, similar <...
taciturno's user avatar
  • 137
1 vote
1 answer
245 views

YOLO : why does changing the confidence threshold change the [email protected]?

I trained a YOLOv7 model for a detection task. I have only one class, which is the object I want to detect. I ran test.py with --conf-thresh to 0.001 (default) and a second time with --conf-thresh to ...
Quintino's user avatar
0 votes
1 answer
34 views

Get dependant probabilities in multiclassification

After training my CatBoostClassifier model I call get_proba function which returns me list of probabilities. The problem starts from an another point... I transfer that data into dataframe then to ...
Master_Sniffer's user avatar
1 vote
2 answers
66 views

How to evaluate Natural Question-Answer Generation pairs?

I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to ...
Aaditya ura's user avatar
2 votes
1 answer
266 views

Why is accuracy not a useful measure for information retrieval problems?

I have been studying about information retrieval and recommender systems. While reading about it I found that accuracy not a useful measure in information retrieval. I understand that, accuracy might ...
ilved17's user avatar
  • 31
0 votes
0 answers
61 views

When using custom metrics, num_threads setting is not working in lightgbm?

When I train a lightgbm model with my own custom metrics, I find that model is trained in single thread, though I set "num_threads": 16. ...
JisongXie's user avatar
0 votes
1 answer
35 views

Success metric of database migration using row counts

Description I have a problem where I'm tasked to successfully transform and repurpose data from one SQL server to another. Call the source $\text{src}$ and the target database $\text{tgt}$. In order ...
ArnoV's user avatar
  • 101
2 votes
1 answer
3k views

Accuracy vs Categorical Accuracy

I was running a DNN model that uses ResNet50 for Transfer Learning. While fitting the training data on my model to check the initial trend (would run for more epochs if initial trend seems right), I ...
Harsh Khare's user avatar
3 votes
1 answer
55 views

What does precision-recall curve and ROC curve tell us abouth threshold invariance

Consider a binary classification problem. Intuitively, a value for the area under the curve (for both curves) very close to 1, shows that the curve is almost L-shaped. Thus, this means that the value ...
liakoyras's user avatar
  • 636
-1 votes
2 answers
163 views

Classification model- which metric to choose to evaluate

If my project is "Music Genre Classification" Which metric method I need to choose to evaluate, and why ? Thanks a lot
yuvi's user avatar
  • 9
0 votes
1 answer
100 views

About the Evaluation method of the Market 1501 ReID dataset

The market 1501 dataset has train, query and gallery folders, each containing multiple views of people from multiple cameras. I would like to understand how to evaluate a model (trained with triplet ...
Anamaki's user avatar
  • 101
0 votes
0 answers
45 views

My overfitted decision tree regressor gives better result than pre-pruned tree?

I create a decision tree regressor without giving any parameters. The resulting tree has 6255 leaf nodes (out of 6348 entries of train set) and depth of 39. Most probably it has overfitted. But its ...
Akrobeto's user avatar
1 vote
0 answers
84 views

Calculationg perplexity (in natural language processing) manually

I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following ...
Piskator's user avatar
  • 125
0 votes
1 answer
16 views

Statistic methods to compute the average of a list of metric results

I have a Machine Learning model that fits and predicts many time series at once, so, for each time series I have a metric result, for example, MAE. What I need is to generate an unique value for that ...
Gabriel Caldas's user avatar
0 votes
0 answers
38 views

Am I calculating LogAUC / pAUC correctly?

Hope you are well. I was wondering how one would calculate logAUC? I have an implementation but I don't think it's correct. I'm trying to recreate the metric in this manuscript. See figure 2. Any help ...
James Arthur's user avatar
2 votes
1 answer
140 views

What makes an ROC curve a curve and why do the values change?

I have a problem. I am currently looking at a classifier and I would like to examine this using an ROC curve as a metric. However, questions have arisen to which I can not find an answer. A ROC curve ...
Test's user avatar
  • 89
0 votes
1 answer
74 views

Why compare multiple machine learning algorithms and then decide which algorithm to use for fine tuning?

I have a problem. There is a dataset A, which deals with a classification problem. And for this dataset, several different baseline algorithms have been defined and computed. In addition, three models ...
Test's user avatar
  • 89
0 votes
0 answers
27 views

What if many different models reach the same maximum metrics

I am talking about trying different algorithms, different parameters, different stacking configurations that all improve upon previous baselines, and yet a lot of them have exactly the same values on ...
liakoyras's user avatar
  • 636

1
2 3 4 5