People who code: we want your input. Take the Survey

Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Search / Multiple Choice System evaluation

I have a DB with N items. My system can output an ID for the item or say N/A (not found). What are different ways to evaluate the performance of such system, and what are the characteristics/tradeoffs ...
1
vote
0answers
18 views

How to construct a test set for an active learning project?

With active learning I hope to keep the annotation effort to a minimum, yet building still a good classifier. My initial starting point is that I have about 20k images which can belong to ten ...
0
votes
0answers
17 views

Evaluate scoring algorithm

I have several large datasets (100k samples) with label 1 / 0 and scoring algorithms that should score each sample, after the scoring of the dataset I sort it by the score. I would like to evaluate ...
1
vote
1answer
8 views

ZeroR as performance baseline for binary classfication model?

It is known that ZeroR model is used predict the majority class in a given data set. Having said that, is ZeroR a suitable performance baseline provided one has a balanced data set (50/50)? If not, ...
2
votes
1answer
23 views

Is it better to use F1 score or AUC metric for imbalanced data classification?

I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?
1
vote
0answers
22 views

Permutation importance of features [closed]

This agnostic-model is not well addressed in research papers. I read articles where it was used to test the accuracy of the models, trying to understand the importance of individual features and their ...
0
votes
0answers
16 views

Scoring function for transformers (BERT etc)

While using BERT / transformers for NLP tasks, a major problem faced by us was to detect if the answer returned by model is correct or not, or what is the confidence level of the answer. The ...
0
votes
0answers
16 views

Evaluating a model with different target class distributions between training and testing data

I'm having a bit of an argument about when class imbalances matter when training a classifier, so I was hoping to get some help on understanding a specific concept. Say I have a problem where I want ...
0
votes
0answers
11 views

How to make fair comparison of multi-task RL models if I have unlimited test data?

The data comes from a simulator hence I have the possibility to generate unlimited data. The reward is 0 (no success) and success(1) if episode is successful. Now, the question is what metric to use ...
1
vote
1answer
19 views

How to interpret the rec curve for a regression task?

I am using forest fire dataset and applied neural network model. I tried to generate REC curve, this is how it looks like. Pretty weird!!! I have also applied XGBoost but the REC curve is almost ...
0
votes
0answers
15 views

How to optimize AUPRC for imbalanced data given a precision or recall bias?

My general understanding is that when optimizing a model in an imbalanced class case with a small preferred target class one should optimize first for a model with the best AUPRC (assuming one doesn't ...
3
votes
1answer
102 views

How to create a confusion matrix for one node of a decision tree?

I am doing past papers for my data science exam and was curious about one of the questions. They ask us to create a confusion matrix by hand for one node of a decision tree. I understand how to create ...
2
votes
0answers
26 views

Best way to evaluate interlaced recommendation system results while reducing bias

I already asked this question but I worded it in such a way that it was a completely different question to the one I want to ask. I have not deleted the old question in case someone finds it useful. ...
1
vote
0answers
23 views

ROC and AUC curve for CNN multi-class classification problem

I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My ...
1
vote
1answer
34 views

How to explain a relationship between Accuracy and F1 Score / F-Measure?

I am building a CNN model for pitch estimation using a song recording. Pitch estimation is done by inputting spectrogram to CNN model and make the CNN predict pitch sequence (250 pitch values per ...
0
votes
0answers
11 views

Understanding the process for generating Random Forest model in caret

This is not so much a problem, as it is me making sure I understand what's happening with my Random Forest algorithm. Below, I've set a few parameters. Am I right in thinking that this is the stages: ...
0
votes
0answers
20 views

Margin of error for imbalanced discrete set

I'm evaluating the performance of a classifier regarding its false negatives. The classifier performed over 9090 samples, from which 9000 were labeled as negative. I randomly chose 800 samples (out of ...
0
votes
0answers
24 views

Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
5
votes
1answer
61 views

Suitable metric choice for imbalanced multi-class dataset (classes have equal importance)

What type of metrics I should use to evaluate my classification models, given that I have two imbalanced multi-class datasets (21 and 16 classes, respectively) where all classes have equal importance? ...
0
votes
0answers
24 views

How to evaluate pix2pix?

As far as I know, to evaluate synthesized images it is proposed to use: human scoring, "Inception score", where in the second case the quality is rated based on a pre-trained Inception ...
1
vote
0answers
12 views

How to interpret high loss value from model.evaluate() on test data

I'm collecting some metrics for my model's performance using: ...
2
votes
1answer
281 views

precision@k and recall@k

Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems. I ...
0
votes
0answers
14 views

TF Estimator stop when overfitting?

With TF 1.x is there a way to stop training, when evaluation loss starts increasing? Or is there another way to detect overlearning? I know about the stop_if_no_decrease_hook, but it doesn't consider ...
0
votes
0answers
29 views

Can you use precision/recall/f1 scores in a balanced data context?

I want to evaluate two classifiers. The data is perfectly balanced and there is the same amount of data per class. Since it is balanced I am using accuracy. I am interested in assessing which ...
0
votes
0answers
6 views

Test set is representative of population. Is the evaluation of the ML on the test set represents absolute truth how model will behave in real world?

The question is more theoretic. Lets assume that the test set is perfect representation of the population. If I will evaluate the machine learning predictive model on the test set, can we call the ...
0
votes
0answers
52 views

Time series imputation benchmark

In a work, I have to benchmark different algorithms to fill in missing values in time series. I insist on the fact that this is imputation and not forecasting. In my case, I have access to 15 years of ...
0
votes
1answer
73 views

IterativeImputer Evaluation

I am having a hard time evaluating my model of imputation. I used an iterative imputer model to fill in the missing values in all four columns. For the model on the iterative imputer, I am using a ...
4
votes
2answers
189 views

Can Micro-Average Roc Auc Score be larger than Class Roc Auc Scores

I'm working with an imbalanced data set. There are 11567 negative and 3737 positive samples in train data. There are 2892 negative and 935 positive samples in validation data. It is a binary ...
1
vote
0answers
52 views

Can you estimate average precision from log loss?

I am doing my final thesis in the field of Deepfakes and their detection. The final outcome is to have a binary classifier which could predict which video was updated and which was not. In other words,...
1
vote
0answers
15 views

What are the business metrics I should track to evaluate a recommender model deployed on an e-commerce website? [closed]

Can you suggest some google analytics metrics such as (click or impressions etc) to evaluate a recommender model deployed on an e-commerce website.
1
vote
0answers
23 views

(Graph Convolutional Network (GCN) based recommender system maintenance issue [closed]

I have built an item-item recommender model using (Graph Convolutional Network (GCN) for an E-commerce website. Could you please help me with the maintenance of the model. How often should I retrain ...
1
vote
0answers
12 views

What are the motion planning benchmarks?

Suppose I wanted to try and improve on existing motion planning algorithms. What benchmarks should I be trying to beat? Papers with code site has no motion planning benchmarks. I searched online and ...
1
vote
1answer
16 views

Output of evaluation metric for XGBoost - is it cumulative?

On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
0
votes
0answers
87 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
1
vote
1answer
31 views

Evaluation of recommendation systems

I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
3
votes
1answer
39 views

Evaluation metric for Information retrieval system

I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to maximize ...
0
votes
0answers
50 views

How can realize the evaluation/validation of unsupervised models through unlabeled data?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental ...
2
votes
1answer
435 views

Micro Average vs Macro Average for Class Imbalance

I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional ...
0
votes
0answers
22 views

Using accuracy metric during training for unbalanced multiclass classification

I am training a convolutional neural network and the sensitivity and precision of the minority class is what is most important to me. I am using 10-Fold cross validation, and the test fold is ...
0
votes
1answer
65 views

How to estimate missing values when calculating NDCG

I would like to compare recommendations methods using NDCG metric on MovieLens dataset. In ranking problem, the goal is to rank items based on their relevance for user. Ranking models can be learned ...
1
vote
1answer
324 views

Macro and micro average for imbalanced binary classes

Micro and macro averaging are metrics for multi-class classification. However, for binary classification when data are imbalanced, it seems that micro and macro precision have different results. My ...
1
vote
1answer
202 views

Proper evaluation method for recommendation system with implicit feedback?

I am trying to implement a recommendation system for a live-streaming website. Here "users" are simply the website users and "items" are streamers that they should watch. I ...
0
votes
0answers
14 views

Appropriate naive benchmark for class recall in binary classification for unbalanced dataset

I have an unbalanced dataset with 3969 rows of customer data. The labels are whether or not they subscribed for a loan (yes or no). There are 3618 no cases (91.2%) and 351 yes cases (8.8%). I am more ...
1
vote
1answer
236 views

What am I supposed to see on tensorboard images tab?

I'm training an object detection model with Tensorflow and monitor the training task with tensorboard. I was expecting in the Images tab of tensorboard that displayed images would show a bounding box (...
2
votes
0answers
32 views

Assessing model performance on different sub-segments [closed]

I am currently working on a credit risk related project where i built a binary logistic regression model for an imbalanced dataset. According to the regulations i have to prove that the model performs ...
0
votes
1answer
87 views

Machine Learning validation data returns 100% accuracy [closed]

I'm Testing a Machine Learning model with validation data returns that return 100% correct answers, is it overfitting or the model works extremely well, do I need to continue training on more data? I'...
1
vote
1answer
74 views

Time series analysis model evaluation performance metrics integration in time series application

After study in time series analysis, I recognized RMSE and MAPE are the best evaluation metrics for used model in real time series application. But my queries are below as this is my first practice ...
1
vote
1answer
170 views

Match between objective function and evaluation metric

Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process? For example, can a XGBoost model be ...
0
votes
0answers
8 views

Get latest Item by Date for a Recommender System

I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would ...
1
vote
1answer
70 views

How do you identify whether your RMSE score is good or not?

Im building a XGBoost regression model to predict the values in the range of -3 to 3. Im using Root Mean Squared Error to evaluate the model. With hyper-parameter tuning and everything the best scores ...

1
2 3 4 5