Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
1
vote
1answer
21 views

what metrics to evaluate rank order results?

I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem). ...
0
votes
0answers
7 views

How to select checkpoint for model evaluation?

I have trained a deep convolutional neural network for image similarity classification. The network returns whether the images are the same or different. I trained the network for 20 epochs and save ...
0
votes
1answer
21 views

Manual way to draw accuracy/loss graphs

During the training process of the convolutional neural network, the network outputs the training/validation accuracy/loss after each epoch as shown below: ...
2
votes
1answer
31 views

How to correctly calculate average F1 score, precision and recall of a Named Entity Recognition system?

My Named Entity Recognition (NER) pipeline built with Apache uimaFIT and DKPro recognizes named entities (called datatypes for now) in texts (e.g. persons, locations, organizations and many more). I ...
2
votes
0answers
27 views

Confused about the MSE ERROR

I am created a random forest regressor and calculate my own error. I want also to calculate MAE, MSE and RMSE to compare my results to similar usecases. But the results of the MAE, MSE, RMSE are ...
1
vote
1answer
48 views

How to keep the test data from leaking into the training process of a machine learning algorithm?

I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to ...
1
vote
0answers
28 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
1
vote
1answer
27 views

Need of Weighted Mean Squared Error

We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix. Below is the WMSE formula: Can ...
2
votes
0answers
17 views

FFR and FAR calculating for multiclasss biometric face recognition system

I am implementing a face recognition system using facenet and svc Ml algorithm i have like 20 classes or more and I'm getting 98% accuracy im trying to calculate the FAR and FRR and the EER I'm ...
2
votes
1answer
21 views

evaluation metrics for multiple values per session

I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based ...
1
vote
1answer
68 views

XGBoost Feature Importance, Permutation Importance, and Model Evaluation Criteria

I have built an XGBoost classification model in Python on an imbalanced dataset (~1 million positive values and ~12 million negative values), where the features are binary user interaction with web ...
1
vote
1answer
70 views

How to Maximize recall for Minority class?

I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am ...
3
votes
2answers
56 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
2
votes
1answer
44 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
0
votes
0answers
18 views

taking np.argmax while evaluating the model

I'm studying a code for a task of Music Genre Classification and I'm stuck at understanding a few line of codes that come after the model has been built. Basically it concerns the valuation of your ...
4
votes
1answer
35 views

How is “relevance” defined in information retrieval outside the context of systems with user feedback?

I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant". What can you do if you do not have user ...
5
votes
1answer
54 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
0
votes
1answer
28 views

Evaluating information extraction from structured documents

I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that: ...
2
votes
1answer
47 views

Evaluate clustering by using decision tree unsupervised learning

I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion ...
0
votes
1answer
264 views

What is Continuous Ranked Probability Score (CRPS)?

I came across some evolution metric at Kaggle: Continuous Ranked Probability Score (CRPS): Mathematically, $C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$ where P ...
1
vote
1answer
27 views

Evaluation method for multi-class classification problem modeled as binary classification problem

I should mention that even though I have some basic knowledge regarding ML, it is the first big ML project I am working on and for the proposal of my research project I need to suggest an evaluation ...
0
votes
1answer
99 views

How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
0
votes
0answers
15 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
4
votes
1answer
479 views

Difference between learning_curve and validation_curve

What is the difference between these two curves: learning_curve and validation_curve ?
0
votes
0answers
8 views

How to proceed after tuning hyperparameters?

As I am still on the journey to understand what when and how to use, I am now at the point how to proceed after finding the best hyperparameters: Define Model (NN) Split Data into ...
0
votes
1answer
35 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
0
votes
2answers
38 views

Validate via predict() or via fit()?

There are several possibilites to evaluate a model: hist = model.fit(x_train, y_train, (...) validation_data=(x_test, y_test)) or to use <...
0
votes
0answers
22 views

How do I compare more than 20 deep learning models?

I have to compare several deep learning models (CNNs) based on the same dataset. For estimating the model skill's I use the train_test_split instead of ...
0
votes
0answers
15 views

Threshold for overfitted models

It's common knowledge in DS that overfitted models perform well on training data and poorly on test data. But how do you decide if a model is really overfitting? I have nowhere (books, online courses, ...
0
votes
0answers
7 views

Alternative ranking evaluation metrics for biased data

What are some alternative evaluation metrics for ranking problems, that could help when evaluation is done on heavily biased data? Example - if we sort items by their price and want to evaluate ...
0
votes
0answers
61 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
1
vote
1answer
25 views

Validity of cross-validation for model performance estimation

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure,...
0
votes
1answer
45 views

Evaluate imbalanced classification model on balanced testing sample

Why it would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?
3
votes
1answer
195 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
1
vote
0answers
92 views

Evaluating Random Forest regression model that predicts low values for skewed dependent variable

Background I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values ...
0
votes
1answer
111 views

Choice of f1 score for highly imbalanced dataset?

I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation. Given my dataset is highly imbalanced(600:100000)
1
vote
0answers
28 views

How mAP is unfair evaluation metric for Object Detection?

The following figure is from the last page in YOLOv3 paper highlighting how mAP is unfair metric for evaluating Object Detectors: The figure shows two hypothetical Object Detector results which the ...
0
votes
1answer
64 views

How do I interpret the given classification report?

The given classification report was obtained from running a Random Forest binary classifier on the test data. There is huge class imbalance in the training data. How do I interpret the given ...
1
vote
0answers
407 views

How to compute Frechet Inception Score for MNIST GAN?

I'm starting out with GANs and I am training a DC-GAN on MNIST dataset. The two metrics that are used to evaluate GANs are Inception Score (IS) and Frechet Inception Distance (FID). Since Inception ...
4
votes
1answer
76 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
1
vote
0answers
197 views

What's the difference between these two custom sparse categorical accuracy functions?

I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer. I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head ...
2
votes
1answer
25 views

How can I get an algorithm to have an evalutation metric based on aggregate predictions?

Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual ...
1
vote
0answers
43 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
3
votes
1answer
322 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
3
votes
2answers
350 views

In k-fold-cross-validation, why do we compute the mean of the metric of each fold

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric. Source : https://scikit-learn.org/stable/...
2
votes
2answers
88 views

Metrics for Name Entity Recognition

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime ...
0
votes
0answers
29 views

Which metric should be used to select best binary pixel-wise classifier for segmentation task?

I am doing a semantic segmentation task using a supervised algorithm to classify image pixels into one class or the other (binary classification). I am trying several classifiers and feature ...
0
votes
2answers
91 views

A robust metric in the presence of class imbalance

When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose? I read a paper that states: "Average ...
0
votes
1answer
106 views

Normalized metric for comparing regression models performance

I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It ...