Questions tagged [model-evaluations]

This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.

Filter by
Sorted by
Tagged with
1
vote
1answer
27 views

n_jobs=-1 or n_jobs=1?

I am confused regarding the n_jobs parameter used in some models and for CV. I know it is used for parallel computing, where it includes the number of processors specified in n_jobs parameter. So if I ...
0
votes
0answers
12 views

What could be reasons for higher MAPE?

I built two models on a dataset where data for independent variables (X) being the same and dependent variable (Y) changes for each model eg : Y is price value for a particular region. Y values change ...
1
vote
1answer
16 views

Performance measurement of an event extraction system

I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
0
votes
1answer
11 views

Evaluation Metrics for Multiclass

How to obtain the Accuracy, Detection_Rate, False_Positive_Rate and ...
1
vote
1answer
28 views

Quantitative measure of the smoothness of learning curves

$\DeclareMathOperator{\loss}{loss}$ $\DeclareMathOperator{\AvgVar}{AvgVar}$ Lat's say we have some deep learning task. We have our model and two sets of hyperparameters $A$ and $B$. We train both ...
0
votes
0answers
24 views

Linear regression to find differences between model performances

For one of my projects I needed to create classification models for each of many products. In order to see which classifier performs best, I created one SVM, RandomForest and Naive Bayes model for ...
2
votes
0answers
25 views

Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps: obtain prediction on test data using ...
0
votes
0answers
9 views

how to evaluate the performance of a recommender system with single recommendation

Say we have a recommender system in production which recommends 1 our of N items according to some internal algorithm f given inputs Xi for each user i, let's assume f is a black box model. We have ...
0
votes
0answers
29 views

How to use mean IoU for RGB mask (keras implementation)?

I am training pix2pix GAN for converting SAR satellite images to segmentation mask. But I am not aware about how to use the mean IoU to evaluate my model. My output is of ...
2
votes
1answer
570 views

precision@k and recall@k

Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems. I ...
0
votes
0answers
7 views

Test set is representative of population. Is the evaluation of the ML on the test set represents absolute truth how model will behave in real world?

The question is more theoretic. Lets assume that the test set is perfect representation of the population. If I will evaluate the machine learning predictive model on the test set, can we call the ...
2
votes
3answers
91 views

Comparing ML models to baselines

When comparing ML models with baseline or "dummy" models, are there best practices for building and comparing baselines? I'm doing a binary classification task where 40% of the samples are ...