Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
2
votes
1answer
7 views

Calculating Expected Percentile Ranking for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
0
votes
0answers
11 views

taking np.argmax while evaluating the model

I'm studying a code for a task of Music Genre Classification and I'm stuck at understanding a few line of codes that come after the model has been built. Basically it concerns the valuation of your ...
4
votes
1answer
32 views

How is “relevance” defined in information retrieval outside the context of systems with user feedback?

I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant". What can you do if you do not have user ...
5
votes
1answer
47 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
0
votes
1answer
24 views

Evaluating information extraction from structured documents

I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that: ...
2
votes
1answer
43 views

Evaluate clustering by using decision tree unsupervised learning

I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion ...
0
votes
1answer
27 views

What is Continuous Ranked Probability Score (CRPS)?

I came across some evolution metric at Kaggle: Continuous Ranked Probability Score (CRPS): Mathematically, $C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$ where P ...
1
vote
1answer
26 views

Evaluation method for multi-class classification problem modeled as binary classification problem

I should mention that even though I have some basic knowledge regarding ML, it is the first big ML project I am working on and for the proposal of my research project I need to suggest an evaluation ...
0
votes
0answers
43 views

How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
0
votes
0answers
11 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
3
votes
1answer
166 views

Difference between learning_curve and validation_curve

What is the difference between these two curves: learning_curve and validation_curve ?
0
votes
0answers
7 views

How to proceed after tuning hyperparameters?

As I am still on the journey to understand what when and how to use, I am now at the point how to proceed after finding the best hyperparameters: Define Model (NN) Split Data into ...
0
votes
1answer
33 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
0
votes
2answers
32 views

Validate via predict() or via fit()?

There are several possibilites to evaluate a model: hist = model.fit(x_train, y_train, (...) validation_data=(x_test, y_test)) or to use <...
0
votes
0answers
21 views

How do I compare more than 20 deep learning models?

I have to compare several deep learning models (CNNs) based on the same dataset. For estimating the model skill's I use the train_test_split instead of ...
0
votes
0answers
9 views

Threshold for overfitted models

It's common knowledge in DS that overfitted models perform well on training data and poorly on test data. But how do you decide if a model is really overfitting? I have nowhere (books, online courses, ...
0
votes
0answers
6 views

Alternative ranking evaluation metrics for biased data

What are some alternative evaluation metrics for ranking problems, that could help when evaluation is done on heavily biased data? Example - if we sort items by their price and want to evaluate ...
0
votes
0answers
35 views

How to evaluate unsupervised KNN?

I'm creating a recommender system using an unsupervised nearest neighbors model to suggest similar publishers for a given publisher, advertiser combination. I'm wondering how to evaluate the model I ...
0
votes
0answers
51 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
1
vote
1answer
23 views

Validity of cross-validation for model performance estimation

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure,...
0
votes
1answer
43 views

Evaluate imbalanced classification model on balanced testing sample

Why it would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?
3
votes
0answers
101 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
1
vote
0answers
89 views

Evaluating Random Forest regression model that predicts low values for skewed dependent variable

Background I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values ...
0
votes
1answer
68 views

Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
1
vote
0answers
23 views

How mAP is unfair evaluation metric for Object Detection?

The following figure is from the last page in YOLOv3 paper highlighting how mAP is unfair metric for evaluating Object Detectors: The figure shows two hypothetical Object Detector results which the ...
0
votes
1answer
30 views

How do I interpret the given classification report?

The given classification report was obtained from running a Random Forest binary classifier on the test data. There is huge class imbalance in the training data. How do I interpret the given ...
0
votes
0answers
286 views

How to compute Frechet Inception Score for MNIST GAN?

I'm starting out with GANs and I am training a DC-GAN on MNIST dataset. The two metrics that are used to evaluate GANs are Inception Score (IS) and Frechet Inception Distance (FID). Since Inception ...
3
votes
1answer
54 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
0
votes
0answers
46 views

Evaluating accuracy of TensorFlow object detection model

I've just finished training and testing my TensorFlow object detection model, and would like to create a presentation displaying various statistics that demonstrate the overall accuracy and utility of ...
1
vote
0answers
141 views

What's the difference between these two custom sparse categorical accuracy functions?

I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer. I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head ...
2
votes
1answer
25 views

How can I get an algorithm to have an evalutation metric based on aggregate predictions?

Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual ...
1
vote
0answers
37 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
2
votes
1answer
217 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
2
votes
3answers
243 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
2
votes
2answers
43 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
21 views

Which metric should be used to select best binary pixel-wise classifier for segmentation task?

I am doing a semantic segmentation task using a supervised algorithm to classify image pixels into one class or the other (binary classification). I am trying several classifiers and feature ...
0
votes
2answers
58 views

A robust metric in the presence of class imbalance

When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose? I read a paper that states: "Average ...
0
votes
1answer
77 views

Normalized metric for comparing regression models performance

I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It ...
0
votes
0answers
17 views

PageRank computation with dumping (scaling) factor

it is not clear to me how to calculate two iterations of PageRank computation on the following network with dumping (scaling) factor s = 1.how can they calculate it correctly?
0
votes
1answer
21 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while ...
0
votes
0answers
59 views

How can I evaluate my sequence prediction model?

I want to evaluate the performance of my prediction model , which is an VED (Variational Encoder Decoder) used for sequences prediction (it predicts the next sequence knowing the actual) I want to ...
1
vote
2answers
165 views

Do precision-recall curves have a constant shape/pattern?

I know ROC curve always looks like a stair shape and that I can evaluate AUC of ROC. And I know I can compute AUC of ROC curve to compare which model is better. What I wonder is: Does precision-...
4
votes
2answers
99 views

Is the F1 Score sensitive to the threshold?

Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?
1
vote
0answers
27 views

Error-analysis and evaluation of a model using Python?

My method of evaluating a model is the following : Split the training data set and do cross validation to obtain an accuracy of my model on my cross validation data set. Use the parameters that gave ...
1
vote
1answer
48 views

Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity)

I am new to Machine Learning and have been doing some practice on Logistic Regression. To evaluate the models, I've been trying to create some ROC plots. The package that i used is pROC. The model ...
0
votes
0answers
18 views

Original References for Micro and Macro Averages

When evaluating group results, micro and macro averages are commonly used. They are explained in multiple text books in detail. However, I wonder whether there is an "original" inventor to those ...
1
vote
1answer
28 views

How is the linear regression cost function evolved?

A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. ...
0
votes
0answers
69 views

Dummy/baseline models for time series forecasting

I am working on an evaluation of time series forecasting models in Python, more specifically with statsmodels, scikit-learn and <...
0
votes
1answer
64 views

Metric (rather than RMSE, MSE, etc.) to choose the best model in terms of the ability to detect peaks better

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
2
votes
1answer
41 views

Conditional Entropy and Mutual Information - Clustering evaluation

First of all, I am doing clustering and I have the true labels for my data. For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with ...