Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Simple score function with 4 different indicators

I want to create a function, which returns a value between (0,1) or (-1,1). The result of this function is then used for a boolean decision. E.g. if the value is closer to 0 decision ...
2
votes
1answer
23 views

How to measure the accuracy of an NLP paraphrasing model?

I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with ...
0
votes
0answers
12 views

ML Evaluation Metrics In C++ [closed]

Are there any C++ libraries (preferably header-only) that implement equivalents of sklearn's metrics section?
0
votes
0answers
20 views

repeated train/test splitting and assessing performance variability

I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset). ...
0
votes
0answers
16 views

Calculating effect of entity recognition on a relation extraction system

How can we calculate/formulate the effectiveness of named entity linking (based on P/R/F1 or any other evaluation metrics) on a relation extraction system which accepts the output of ER as its input? ...
0
votes
0answers
20 views

Combine several performance metrics from several datasets

We are developing and evaluating a multi knee/elbow point detection algorithm. For our evaluation, we have 200 sequences of real data. These sequences were annotated manually. For each algorithm and ...
1
vote
1answer
34 views

In general, what are precision, recall, F1 that are reported in papers?

I used classification_report in sklearn library And, the picture below shows evaluation on my model (anomaly detector) In general, what are precision, recall, F1 ...
0
votes
1answer
12 views

How to interprete the feature significance and the evaluation metrics in classification predictive model?

Consider a experiment to predict the Google-Play apps rating using a Random-Forest classifier with scikit-learn in Python. Three attributes 'Free', 'Size' and 'Category' are utilized to predict the ...
1
vote
1answer
33 views

Comparing Multiclass classifiers with "No Answer"-Class

I have three classifiers to classify some words into four classes. Every word that does not fit into any of these four classes gets classified as "No Answer". I would like to compare the ...
1
vote
3answers
23 views

What to do about the failed portion of trained dataset?

I've seen lots of tutorials and papers about this or that model getting some great accuracy score. In this case, let's say 85%. But what I never see is what you are supposed to do with the remaining ...
0
votes
1answer
31 views

Evaluation metric for imbalanced data

Hi I'm a CS graduate student I have a question for AI or data experts. I'm writing a paper My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6% you can see the ...
1
vote
0answers
14 views

Criteria for assessing difficulty of a question

I have a list of questions and how many times they have been answered correctly and incorrectly. Based on this, I applied the formula: ...
1
vote
1answer
35 views

How to test unsupervised learning methods for anomaly detection?

How to test unsupervised learning methods for anomaly detection? I am looking for a test strategy to evaluate my result of my anomaly detection technique? what is your offer more than evaluate with ...
1
vote
1answer
30 views

Evaluation metric for time-series anomaly detection

I have a question for AI or data experts. I'm writing a paper My dataset are time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one ...
2
votes
1answer
126 views

How to choose the right threshold for binary classification?

I am currently working on the titanic dataset from Kaggle. The data set is imbalanced with almost 61.5 % negative and 38.5 positive class. I divided my training dataset into 85% train and 15% ...
0
votes
0answers
4 views

Can we use the origional text documnet (which we sumerized) as a reference in ROUGE?

Traditionally, for evaluation, the reference in ROUGE is human generated text (summary) which we compare with system generated text (summary). So consider this, if we generate summaries with different ...
0
votes
0answers
7 views

Search / Multiple Choice System evaluation

I have a DB with N items. My system can output an ID for the item or say N/A (not found). What are different ways to evaluate the performance of such system, and what are the characteristics/tradeoffs ...
1
vote
1answer
56 views

How to construct a test set for an active learning project?

With active learning I hope to keep the annotation effort to a minimum, yet building still a good classifier. My initial starting point is that I have about 20k images which can belong to ten ...
0
votes
0answers
20 views

Evaluate scoring algorithm

I have several large datasets (100k samples) with label 1 / 0 and scoring algorithms that should score each sample, after the scoring of the dataset I sort it by the score. I would like to evaluate ...
1
vote
1answer
13 views

ZeroR as performance baseline for binary classfication model?

It is known that ZeroR model is used predict the majority class in a given data set. Having said that, is ZeroR a suitable performance baseline provided one has a balanced data set (50/50)? If not, ...
2
votes
1answer
26 views

Is it better to use F1 score or AUC metric for imbalanced data classification?

I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?
1
vote
0answers
30 views

Permutation importance of features [closed]

This agnostic-model is not well addressed in research papers. I read articles where it was used to test the accuracy of the models, trying to understand the importance of individual features and their ...
0
votes
0answers
19 views

Scoring function for transformers (BERT etc)

While using BERT / transformers for NLP tasks, a major problem faced by us was to detect if the answer returned by model is correct or not, or what is the confidence level of the answer. The ...
0
votes
0answers
21 views

Evaluating a model with different target class distributions between training and testing data

I'm having a bit of an argument about when class imbalances matter when training a classifier, so I was hoping to get some help on understanding a specific concept. Say I have a problem where I want ...
0
votes
0answers
11 views

How to make fair comparison of multi-task RL models if I have unlimited test data?

The data comes from a simulator hence I have the possibility to generate unlimited data. The reward is 0 (no success) and success(1) if episode is successful. Now, the question is what metric to use ...
1
vote
1answer
22 views

How to interpret the rec curve for a regression task?

I am using forest fire dataset and applied neural network model. I tried to generate REC curve, this is how it looks like. Pretty weird!!! I have also applied XGBoost but the REC curve is almost ...
0
votes
0answers
19 views

How to optimize AUPRC for imbalanced data given a precision or recall bias?

My general understanding is that when optimizing a model in an imbalanced class case with a small preferred target class one should optimize first for a model with the best AUPRC (assuming one doesn't ...
3
votes
1answer
132 views

How to create a confusion matrix for one node of a decision tree?

I am doing past papers for my data science exam and was curious about one of the questions. They ask us to create a confusion matrix by hand for one node of a decision tree. I understand how to create ...
2
votes
0answers
26 views

Best way to evaluate interlaced recommendation system results while reducing bias

I already asked this question but I worded it in such a way that it was a completely different question to the one I want to ask. I have not deleted the old question in case someone finds it useful. ...
1
vote
0answers
33 views

ROC and AUC curve for CNN multi-class classification problem

I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My ...
1
vote
1answer
42 views

How to explain a relationship between Accuracy and F1 Score / F-Measure?

I am building a CNN model for pitch estimation using a song recording. Pitch estimation is done by inputting spectrogram to CNN model and make the CNN predict pitch sequence (250 pitch values per ...
0
votes
0answers
11 views

Understanding the process for generating Random Forest model in caret

This is not so much a problem, as it is me making sure I understand what's happening with my Random Forest algorithm. Below, I've set a few parameters. Am I right in thinking that this is the stages: ...
0
votes
0answers
20 views

Margin of error for imbalanced discrete set

I'm evaluating the performance of a classifier regarding its false negatives. The classifier performed over 9090 samples, from which 9000 were labeled as negative. I randomly chose 800 samples (out of ...
1
vote
0answers
45 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
5
votes
1answer
66 views

Suitable metric choice for imbalanced multi-class dataset (classes have equal importance)

What type of metrics I should use to evaluate my classification models, given that I have two imbalanced multi-class datasets (21 and 16 classes, respectively) where all classes have equal importance? ...
0
votes
0answers
37 views

How to evaluate pix2pix?

As far as I know, to evaluate synthesized images it is proposed to use: human scoring, "Inception score", where in the second case the quality is rated based on a pre-trained Inception ...
1
vote
0answers
17 views

How to interpret high loss value from model.evaluate() on test data

I'm collecting some metrics for my model's performance using: ...
2
votes
1answer
958 views

precision@k and recall@k

Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems. I ...
0
votes
0answers
18 views

TF Estimator stop when overfitting?

With TF 1.x is there a way to stop training, when evaluation loss starts increasing? Or is there another way to detect overlearning? I know about the stop_if_no_decrease_hook, but it doesn't consider ...
0
votes
0answers
42 views

Can you use precision/recall/f1 scores in a balanced data context?

I want to evaluate two classifiers. The data is perfectly balanced and there is the same amount of data per class. Since it is balanced I am using accuracy. I am interested in assessing which ...
0
votes
0answers
7 views

Test set is representative of population. Is the evaluation of the ML on the test set represents absolute truth how model will behave in real world?

The question is more theoretic. Lets assume that the test set is perfect representation of the population. If I will evaluate the machine learning predictive model on the test set, can we call the ...
0
votes
0answers
52 views

Time series imputation benchmark

In a work, I have to benchmark different algorithms to fill in missing values in time series. I insist on the fact that this is imputation and not forecasting. In my case, I have access to 15 years of ...
0
votes
1answer
105 views

IterativeImputer Evaluation

I am having a hard time evaluating my model of imputation. I used an iterative imputer model to fill in the missing values in all four columns. For the model on the iterative imputer, I am using a ...
5
votes
2answers
404 views

Can Micro-Average Roc Auc Score be larger than Class Roc Auc Scores

I'm working with an imbalanced data set. There are 11567 negative and 3737 positive samples in train data. There are 2892 negative and 935 positive samples in validation data. It is a binary ...
1
vote
0answers
53 views

Can you estimate average precision from log loss?

I am doing my final thesis in the field of Deepfakes and their detection. The final outcome is to have a binary classifier which could predict which video was updated and which was not. In other words,...
1
vote
0answers
15 views

What are the business metrics I should track to evaluate a recommender model deployed on an e-commerce website? [closed]

Can you suggest some google analytics metrics such as (click or impressions etc) to evaluate a recommender model deployed on an e-commerce website.
1
vote
0answers
25 views

(Graph Convolutional Network (GCN) based recommender system maintenance issue [closed]

I have built an item-item recommender model using (Graph Convolutional Network (GCN) for an E-commerce website. Could you please help me with the maintenance of the model. How often should I retrain ...
1
vote
0answers
12 views

What are the motion planning benchmarks?

Suppose I wanted to try and improve on existing motion planning algorithms. What benchmarks should I be trying to beat? Papers with code site has no motion planning benchmarks. I searched online and ...
1
vote
1answer
19 views

Output of evaluation metric for XGBoost - is it cumulative?

On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
0
votes
0answers
88 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...

1
2 3 4 5