Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
1answer
73 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
123
votes
6answers
126k views

Micro Average vs Macro average Performance in a Multiclass classification setting

I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the ...
2
votes
0answers
24 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
1
vote
0answers
12 views

matplotlib survey data evaluation

I would need some help on how to display data properly from a survey, since I am relatively new to it I feel a little lost on how to work it the right way and was wondering if someone could give me ...
1
vote
1answer
20 views

Need of Weighted Mean Squared Error

We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix. Below is the WMSE formula: Can ...
2
votes
0answers
15 views

FFR and FAR calculating for multiclasss biometric face recognition system

I am implementing a face recognition system using facenet and svc Ml algorithm i have like 20 classes or more and I'm getting 98% accuracy im trying to calculate the FAR and FRR and the EER I'm ...
3
votes
1answer
176 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
4
votes
1answer
264 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
0
votes
1answer
894 views

How to measure F1 score and NMI for clustering task?

I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section....
1
vote
1answer
35 views

XGBoost Feature Importance, Permutation Importance, and Model Evaluation Criteria

I have built an XGBoost classification model in Python on an imbalanced dataset (~1 million positive values and ~12 million negative values), where the features are binary user interaction with web ...
2
votes
1answer
19 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
3
votes
1answer
283 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
3
votes
2answers
52 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
3
votes
1answer
68 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
10
votes
3answers
7k views

Neural Networks - Loss and Accuracy correlation

I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the ...
1
vote
1answer
51 views

How to Maximize recall for Minority class?

I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am ...
1
vote
1answer
27 views

Evaluation method for multi-class classification problem modeled as binary classification problem

I should mention that even though I have some basic knowledge regarding ML, it is the first big ML project I am working on and for the proposal of my research project I need to suggest an evaluation ...
2
votes
1answer
29 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
0
votes
0answers
16 views

taking np.argmax while evaluating the model

I'm studying a code for a task of Music Genre Classification and I'm stuck at understanding a few line of codes that come after the model has been built. Basically it concerns the valuation of your ...
5
votes
1answer
51 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
4
votes
1answer
34 views

How is “relevance” defined in information retrieval outside the context of systems with user feedback?

I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant". What can you do if you do not have user ...
0
votes
1answer
78 views

What is Continuous Ranked Probability Score (CRPS)?

I came across some evolution metric at Kaggle: Continuous Ranked Probability Score (CRPS): Mathematically, $C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$ where P ...
0
votes
1answer
26 views

Evaluating information extraction from structured documents

I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that: ...
2
votes
1answer
47 views

Evaluate clustering by using decision tree unsupervised learning

I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion ...
4
votes
0answers
334 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
0
votes
1answer
22 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while ...
2
votes
2answers
58 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
90 views

How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
2
votes
1answer
55 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
0
votes
0answers
12 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
3
votes
1answer
274 views

Difference between learning_curve and validation_curve

What is the difference between these two curves: learning_curve and validation_curve ?
0
votes
0answers
8 views

How to proceed after tuning hyperparameters?

As I am still on the journey to understand what when and how to use, I am now at the point how to proceed after finding the best hyperparameters: Define Model (NN) Split Data into ...
0
votes
1answer
41 views

Purpose of test data in binary classification

I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as <...
0
votes
2answers
36 views

Validate via predict() or via fit()?

There are several possibilites to evaluate a model: hist = model.fit(x_train, y_train, (...) validation_data=(x_test, y_test)) or to use <...
0
votes
1answer
34 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
0
votes
0answers
22 views

How do I compare more than 20 deep learning models?

I have to compare several deep learning models (CNNs) based on the same dataset. For estimating the model skill's I use the train_test_split instead of ...
0
votes
0answers
12 views

Threshold for overfitted models

It's common knowledge in DS that overfitted models perform well on training data and poorly on test data. But how do you decide if a model is really overfitting? I have nowhere (books, online courses, ...
1
vote
1answer
718 views

Evaluation of semantic segmentation network with mAP

I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an ...
0
votes
0answers
7 views

Alternative ranking evaluation metrics for biased data

What are some alternative evaluation metrics for ranking problems, that could help when evaluation is done on heavily biased data? Example - if we sort items by their price and want to evaluate ...
0
votes
0answers
48 views

How to evaluate unsupervised KNN?

I'm creating a recommender system using an unsupervised nearest neighbors model to suggest similar publishers for a given publisher, advertiser combination. I'm wondering how to evaluate the model I ...
16
votes
7answers
17k views

When is precision more important over recall?

Can anyone give me some examples where precision is important and some examples where recall is important ?
0
votes
1answer
93 views

Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
1
vote
0answers
89 views

Evaluating Random Forest regression model that predicts low values for skewed dependent variable

Background I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values ...
0
votes
0answers
60 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
1
vote
1answer
25 views

Validity of cross-validation for model performance estimation

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure,...
0
votes
1answer
44 views

Evaluate imbalanced classification model on balanced testing sample

Why it would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?
8
votes
3answers
4k views

Macro- or micro-average for imbalanced class problems

The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time. Some googling shows that many bloggers tend to say that micro-average is the preferred way ...
3
votes
0answers
159 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
0
votes
2answers
77 views

A robust metric in the presence of class imbalance

When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose? I read a paper that states: "Average ...
1
vote
2answers
141 views

Doubt to use accuracy or macro f1 measure in an unbalanced classification task

I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure. The provided data is unbalanced, and I don't have an idea about the test ...