Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
131
votes
6answers
140k views

Micro Average vs Macro average Performance in a Multiclass classification setting

I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the ...
21
votes
4answers
17k views

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see ...
20
votes
8answers
21k views

When is precision more important over recall?

Can anyone give me some examples where precision is important and some examples where recall is important ?
14
votes
1answer
8k views

How many features to sample using Random Forests

The Wikipedia page which quotes "The Elements of Statistical Learning" says: Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$ features are used in each split. ...
11
votes
1answer
22k views

How to define a custom performance metric in Keras?

I tried to define a custom metric fuction (F1-Score) in Keras (Tensorflow backend) according to the following: ...
11
votes
3answers
8k views

Neural Networks - Loss and Accuracy correlation

I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the ...
9
votes
2answers
3k views

Difference between using RMSE and nDCG to evaluate Recommender Systems

What kind of error measures do RMSE and nDCG give while evaluating a recommender system, and how do I know when to use one over the other? If you could give an example of when to use each, that would ...
9
votes
3answers
5k views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
8
votes
2answers
7k views

Train/Test Split after perform SMOTE

I am dealing with a highly unbalanced data, so I used the SMOTE algorithm to resample the dataset. After SMOTE resampling, I splitted the resampled dataset to training/testing sets, using the ...
8
votes
2answers
639 views

Why is the F-measure preferred for classification tasks?

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks? The F-measure is the ...
7
votes
3answers
3k views

How do you evaluate ML model already deployed in production?

so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production. ...
6
votes
1answer
193 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
5
votes
2answers
110 views

Is the F1 Score sensitive to the threshold?

Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?
5
votes
1answer
4k views

When do I have to use aucPR instead of auROC? (and vice versa)

I'm wondering if sometimes, to validate a model, it's not better to use aucPR instead of aucROC? Do these cases only depend on the "domain & business understanding" ? Especially, I'm thinking ...
5
votes
1answer
59 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
5
votes
2answers
51 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
4
votes
4answers
317 views

How can conclusions be drawn from recommendation systems evaluation?

From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many ...
4
votes
1answer
2k views

Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

I have a binary classification problem which in the test set, the number of data in both classes are equal (the test number of class 0 and class 1 are equal). Since we know that the number of samples ...
4
votes
1answer
696 views

Difference between learning_curve and validation_curve

What is the difference between these two curves: learning_curve and validation_curve ?
4
votes
3answers
380 views

Evaluating new features

How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?
4
votes
1answer
37 views

How is “relevance” defined in information retrieval outside the context of systems with user feedback?

I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant". What can you do if you do not have user ...
4
votes
1answer
75 views

Why are there currently no content-based evaluation metrics for information retrieval?

Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem ...
4
votes
1answer
153 views

How can RL agents be monitored?

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on ...
4
votes
1answer
244 views

Class leaking on validation set

I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements ...
4
votes
1answer
52 views

Assessing significance / confidence of a crossvalidated performance measure

I have a prediction model $P$ and I use some performance measure $I$ to measure $P$'s accuracy. The distribution of $I$ is unknown (it's a custom metric, which is somehow similar to the precision ...
4
votes
1answer
87 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
4
votes
0answers
366 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
4
votes
0answers
1k views

In XGBoost, how to change eval function and keeping same objective?

I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows. ...
3
votes
3answers
1k views

Evaluation methods for multi-class classification

I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ...
3
votes
3answers
481 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
3
votes
2answers
27 views

Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

It is common to define the F-measure as a function of precision and recall, as mentioned in [1]: $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$ However I came across some other cases, another ...
3
votes
1answer
845 views

Evaluating Logistic Regression Model in Tensorflow

Following this tutorial, I have a doubt about the evaluation part in: ...
3
votes
3answers
80 views

Statistical test for machine learning

I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to ...
3
votes
1answer
62 views

Evaluating the result of topic modeling in a way that time matters

I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh ...
3
votes
1answer
2k views

Irregular Precision-Recall Curve

I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting ...
3
votes
1answer
2k views

roc_auc score GridSearch

I am experimenting with xgboost. I ran GridSearchCV with score='roc_auc' on xgboost. The best classificator scored ~0.935 (this is what I read from GS output). But now when I run best classificator ...
3
votes
1answer
101 views

What is the efficiency difference between different cost functions in case of neural networks?

I'm studying the theory behind neural nets and I wonder if there is any actual difference between using different lost/cost functions? Let's say I could use either MAE or MSE for backpropagating loss;...
3
votes
2answers
57 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
3
votes
1answer
200 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
3
votes
1answer
371 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
3
votes
1answer
303 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
3
votes
1answer
212 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
3
votes
0answers
114 views

Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

When comparing several regression models in terms of quality, it seems like most have agreed on the MSE. There are also papers comparing "variance" and "variance accounted for (VAF)". However, there ...
3
votes
0answers
852 views

How to represent ROC curve when using Cross-Validation

I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold. My desired output is one ROC curve with a ...
2
votes
2answers
3k views

XGBoost increase the error when changing evaluation function

I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does ...
2
votes
3answers
7k views

Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

I am evaluating a Credit Risk model that predicts the estimated likelihood of customers defaulting on their mortgage accounts. The model is a Logistic Regression estimator and was built by another ...
2
votes
1answer
227 views

How to evaluate sequence to sequence models?

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$ $Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$ $\hat{Y} = [[...
2
votes
2answers
2k views

Why exactly using a test set for model evaluation is a bad idea?

I don't understand why using the test set for model evaluation is a bad idea. I completely understand why you should not use your test set to train your model (because in that case, you would be ...
2
votes
1answer
871 views

how to evaluate top n recommendation system with movie lens dataset?

Based on my research a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And I'm currently ...
2
votes
1answer
47 views

How to correctly calculate average F1 score, precision and recall of a Named Entity Recognition system?

My Named Entity Recognition (NER) pipeline built with Apache uimaFIT and DKPro recognizes named entities (called datatypes for now) in texts (e.g. persons, locations, organizations and many more). I ...