A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [evaluation]

To evaluate is to score or rate the performance of a model, most commonly with a metric like accuracy.

57 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
264 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
4
votes
0answers
336 views

Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, ...
3
votes
2answers
55 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
3
votes
0answers
159 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
3
votes
1answer
69 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
3
votes
1answer
284 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
3
votes
1answer
177 views

Chi-square as evaluation metrics for nonlinear machine learning regression models

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are ...
3
votes
0answers
108 views

Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

When comparing several regression models in terms of quality, it seems like most have agreed on the MSE. There are also papers comparing "variance" and "variance accounted for (VAF)". However, there ...
3
votes
0answers
1k views

In XGBoost, how to change eval function and keeping same objective?

I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows. ...
2
votes
0answers
26 views

How to estimate the accuracy on a large dataset?

Given that I have a deep learning model(handover from former colleague). For some reason, the train/dev set was missing. In my situation, I want to classify my dataset into 100 categories. The ...
2
votes
0answers
16 views

FFR and FAR calculating for multiclasss biometric face recognition system

I am implementing a face recognition system using facenet and svc Ml algorithm i have like 20 classes or more and I'm getting 98% accuracy im trying to calculate the FAR and FRR and the EER I'm ...
2
votes
0answers
47 views

Is AUC a good metric for evaluating the performance of a multi-class classification?

Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?
2
votes
0answers
48 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
2
votes
1answer
55 views

Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby ...
2
votes
0answers
42 views

What would be the main and essential criteria for evaluating auto-sklearn library ?

I m running experiments using benchmark datasets with auto-sklearn to see how its performance is different to the standard sklearn library, Since automl does an exhaustive search over parameters and ...
2
votes
0answers
21 views

Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?

For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the ...
2
votes
0answers
806 views

How to represent ROC curve when using Cross-Validation

I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold. My desired output is one ROC curve with a ...
1
vote
0answers
13 views

matplotlib survey data evaluation

I would need some help on how to display data properly from a survey, since I am relatively new to it I feel a little lost on how to work it the right way and was wondering if someone could give me ...
1
vote
1answer
27 views

Evaluation method for multi-class classification problem modeled as binary classification problem

I should mention that even though I have some basic knowledge regarding ML, it is the first big ML project I am working on and for the proposal of my research project I need to suggest an evaluation ...
1
vote
0answers
89 views

Evaluating Random Forest regression model that predicts low values for skewed dependent variable

Background I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values ...
1
vote
0answers
26 views

How mAP is unfair evaluation metric for Object Detection?

The following figure is from the last page in YOLOv3 paper highlighting how mAP is unfair metric for evaluating Object Detectors: The figure shows two hypothetical Object Detector results which the ...
1
vote
0answers
359 views

How to compute Frechet Inception Score for MNIST GAN?

I'm starting out with GANs and I am training a DC-GAN on MNIST dataset. The two metrics that are used to evaluate GANs are Inception Score (IS) and Frechet Inception Distance (FID). Since Inception ...
1
vote
0answers
170 views

What's the difference between these two custom sparse categorical accuracy functions?

I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer. I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head ...
1
vote
0answers
42 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
1
vote
0answers
29 views

Error-analysis and evaluation of a model using Python?

My method of evaluating a model is the following : Split the training data set and do cross validation to obtain an accuracy of my model on my cross validation data set. Use the parameters that gave ...
1
vote
0answers
161 views

Can McNemar's test be applied to evaluate multiclass models?

Full Disclosure: I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other. ...
1
vote
0answers
78 views

What does the ratio of precision/recall to their ideal values mean, and why are they equal?

When evaluating rankings, Normalized Discounted Cumulative Gain (NDCG) normalizes the score to a [0,1] range by dividing by the ideal score. What happens if we take the same idea to Precision (Pre) ...
1
vote
1answer
721 views

Evaluation of semantic segmentation network with mAP

I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an ...
1
vote
0answers
152 views

NDCG score is greater than 1

I'm solving a problem of ranking classes for each unique id based on the utilization quantity. I have 6 unique classes in the training and test data. My neural net mode predicts the utilization ...
1
vote
0answers
24 views

What does AR(max=1) mean?

http://cocodataset.org/#detection-eval "ARmax=1: AR given 1 detection per image" "AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs." Above link,...
1
vote
0answers
14 views

Validating performance of panel data based models

I'm wondering from a theoretical/general practice perspective, what is the best way to evaluate performance of regression models derived from panel data (i.e. a time series of cross sectional data). ...
1
vote
0answers
84 views

very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and ...
1
vote
0answers
89 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
1
vote
0answers
39 views

What does NIST information weights refer to?

NIST is a metric used to measure the goodness of translation. In the paper, Doddington (2002) introduce the notion of "Information weights" Information weights were computed using N-gram counts ...
1
vote
0answers
2k views

Can tuning individual precision and recall classification thresholds improve deep learning models?

I learned that Keras doesn't have a built-in way to set a threshold for precision and accuracy when building a classifier. Courtesy of a solution here, I wanted to see what would happen when I fit a ...
1
vote
0answers
239 views

Using tensorflow to test a variable amount of correct labels

I'm using a neural network to analyze item choices made by players in a computer game. In the game players can choose between 0 and 7 items. Right now I'm struggling with how I can evaluate my data. ...
1
vote
0answers
230 views

Estimating precision & recall

Given a very large population, is there an accepted recall estimation method to use when one is validating a trained model against the population? In my case, a trained model was tested on a ...
0
votes
0answers
16 views

taking np.argmax while evaluating the model

I'm studying a code for a task of Music Genre Classification and I'm stuck at understanding a few line of codes that come after the model has been built. Basically it concerns the valuation of your ...
0
votes
0answers
91 views

How to define quadratic weighted kappa as eval_metric in catboost classifier

I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric. Catboost already has WKappa as an eval_metric but it is linearly weighted ...
0
votes
0answers
12 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
0
votes
0answers
8 views

How to proceed after tuning hyperparameters?

As I am still on the journey to understand what when and how to use, I am now at the point how to proceed after finding the best hyperparameters: Define Model (NN) Split Data into ...
0
votes
0answers
22 views

How do I compare more than 20 deep learning models?

I have to compare several deep learning models (CNNs) based on the same dataset. For estimating the model skill's I use the train_test_split instead of ...
0
votes
0answers
13 views

Threshold for overfitted models

It's common knowledge in DS that overfitted models perform well on training data and poorly on test data. But how do you decide if a model is really overfitting? I have nowhere (books, online courses, ...
0
votes
0answers
7 views

Alternative ranking evaluation metrics for biased data

What are some alternative evaluation metrics for ranking problems, that could help when evaluation is done on heavily biased data? Example - if we sort items by their price and want to evaluate ...
0
votes
0answers
48 views

How to evaluate unsupervised KNN?

I'm creating a recommender system using an unsupervised nearest neighbors model to suggest similar publishers for a given publisher, advertiser combination. I'm wondering how to evaluate the model I ...
0
votes
0answers
60 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
0
votes
0answers
26 views

Which metric should be used to select best binary pixel-wise classifier for segmentation task?

I am doing a semantic segmentation task using a supervised algorithm to classify image pixels into one class or the other (binary classification). I am trying several classifiers and feature ...
0
votes
0answers
20 views

PageRank computation with dumping (scaling) factor

it is not clear to me how to calculate two iterations of PageRank computation on the following network with dumping (scaling) factor s = 1.how can they calculate it correctly?
0
votes
1answer
22 views

Evaluating performance of classifier on lopsided dataset

I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while ...
0
votes
0answers
63 views

How can I evaluate my sequence prediction model?

I want to evaluate the performance of my prediction model , which is an VED (Variational Encoder Decoder) used for sequences prediction (it predicts the next sequence knowing the actual) I want to ...