Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
1answer
27 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
0
votes
2answers
29 views

Validate via predict() or via fit()?

There are several possibilites to evaluate a model: hist = model.fit(x_train, y_train, (...) validation_data=(x_test, y_test)) or to use <...
0
votes
0answers
19 views

How do I compare more than 20 deep learning models?

I have to compare several deep learning models (CNNs) based on the same dataset. For estimating the model skill's I use the train_test_split instead of ...
0
votes
0answers
6 views

Threshold for overfitted models

It's common knowledge in DS that overfitted models perform well on training data and poorly on test data. But how do you decide if a model is really overfitting? I have nowhere (books, online courses, ...
0
votes
0answers
6 views

Alternative ranking evaluation metrics for biased data

What are some alternative evaluation metrics for ranking problems, that could help when evaluation is done on heavily biased data? Example - if we sort items by their price and want to evaluate ...
0
votes
0answers
19 views

How to evaluate unsupervised KNN?

I'm creating a recommender system using an unsupervised nearest neighbors model to suggest similar publishers for a given publisher, advertiser combination. I'm wondering how to evaluate the model I ...
0
votes
0answers
41 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
1
vote
1answer
22 views

Validity of cross-validation for model performance estimation

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure,...
0
votes
1answer
43 views

Evaluate imbalanced classification model on balanced testing sample

Why it would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?
3
votes
0answers
36 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
1
vote
0answers
84 views

Evaluating Random Forest regression model that predicts low values for skewed dependent variable

Background I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values ...
0
votes
1answer
45 views

Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
1
vote
0answers
20 views

How mAP is unfair evaluation metric for Object Detection?

The following figure is from the last page in YOLOv3 paper highlighting how mAP is unfair metric for evaluating Object Detectors: The figure shows two hypothetical Object Detector results which the ...
0
votes
1answer
22 views

How do I interpret the given classification report?

The given classification report was obtained from running a Random Forest binary classifier on the test data. There is huge class imbalance in the training data. How do I interpret the given ...
0
votes
0answers
161 views

How to compute Frechet Inception Score for MNIST GAN?

I'm starting out with GANs and I am training a DC-GAN on MNIST dataset. The two metrics that are used to evaluate GANs are Inception Score (IS) and Frechet Inception Distance (FID). Since Inception ...
3
votes
1answer
42 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
0
votes
0answers
29 views

Evaluating accuracy of TensorFlow object detection model

I've just finished training and testing my TensorFlow object detection model, and would like to create a presentation displaying various statistics that demonstrate the overall accuracy and utility of ...
1
vote
0answers
83 views

What's the difference between these two custom sparse categorical accuracy functions?

I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer. I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head ...
2
votes
1answer
24 views

How can I get an algorithm to have an evalutation metric based on aggregate predictions?

Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual ...
1
vote
0answers
29 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
2
votes
1answer
139 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
2
votes
3answers
222 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
2
votes
1answer
29 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
19 views

Which metric should be used to select best binary pixel-wise classifier for segmentation task?

I am doing a semantic segmentation task using a supervised algorithm to classify image pixels into one class or the other (binary classification). I am trying several classifiers and feature ...
0
votes
2answers
47 views

A robust metric in the presence of class imbalance

When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose? I read a paper that states: "Average ...
0
votes
1answer
62 views

Normalized metric for comparing regression models performance

I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It ...
0
votes
0answers
16 views

PageRank computation with dumping (scaling) factor

it is not clear to me how to calculate two iterations of PageRank computation on the following network with dumping (scaling) factor s = 1.how can they calculate it correctly?
0
votes
1answer
19 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while ...
0
votes
0answers
50 views

How can I evaluate my sequence prediction model?

I want to evaluate the performance of my prediction model , which is an VED (Variational Encoder Decoder) used for sequences prediction (it predicts the next sequence knowing the actual) I want to ...
1
vote
2answers
128 views

Do precision-recall curves have a constant shape/pattern?

I know ROC curve always looks like a stair shape and that I can evaluate AUC of ROC. And I know I can compute AUC of ROC curve to compare which model is better. What I wonder is: Does precision-...
4
votes
2answers
95 views

Is the F1 Score sensitive to the threshold?

Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?
1
vote
0answers
22 views

Error-analysis and evaluation of a model using Python?

My method of evaluating a model is the following : Split the training data set and do cross validation to obtain an accuracy of my model on my cross validation data set. Use the parameters that gave ...
1
vote
1answer
42 views

Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity)

I am new to Machine Learning and have been doing some practice on Logistic Regression. To evaluate the models, I've been trying to create some ROC plots. The package that i used is pROC. The model ...
0
votes
0answers
17 views

Original References for Micro and Macro Averages

When evaluating group results, micro and macro averages are commonly used. They are explained in multiple text books in detail. However, I wonder whether there is an "original" inventor to those ...
1
vote
1answer
27 views

How is the linear regression cost function evolved?

A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. ...
0
votes
0answers
64 views

Dummy/baseline models for time series forecasting

I am working on an evaluation of time series forecasting models in Python, more specifically with statsmodels, scikit-learn and <...
0
votes
1answer
57 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
2
votes
1answer
35 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...
1
vote
0answers
99 views

Can McNemar's test be applied to evaluate multiclass models?

Full Disclosure: I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other. ...
2
votes
0answers
44 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
1
vote
2answers
51 views

Classifier performance evaluation

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me. so when building a classifier model such ...
6
votes
1answer
160 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
2
votes
0answers
38 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
1
vote
0answers
65 views

What does the ratio of precision/recall to their ideal values mean, and why are they equal?

When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score. What happens if we take the same idea to Precision (Pre) ...
2
votes
1answer
1k views

Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

I have a binary classification problem which in the test set, the number of data in both classes are equal (the test number of class 0 and class 1 are equal). Since we know that the number of samples ...
1
vote
1answer
320 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
5
votes
2answers
43 views

Does a precision score increasing with a higher number of folds mean the model will improve with more data?

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about ...
2
votes
1answer
49 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
0
votes
1answer
40 views

Purpose of test data in binary classification

I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as <...
0
votes
0answers
68 views

multi armed bandit offline evaluation

I have read those papers: Counterfactual Estimation and Optimization of Click Metrics for Search Engines Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation ...