Questions tagged [model-evaluations]

This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.

Filter by
Sorted by
Tagged with
4 votes
2 answers
70 views

Evaluation of a model of imbalanced data

I've created a model with Random Forest algorithm. There are 45k observations, where 1s I have 12% and the rest are 0s. As far as I know ROC AUC is not the best evaluation metric in such a case. I ...
0 votes
0 answers
13 views

Weighted Classification Metric for Multi class classification

I have a multi-class classification problem with the classes X-Small, Small, Medium and <...
0 votes
0 answers
24 views

Is r-squared a good metric for nonlinear models?

I come to ask if the r-squared metric is correct for assessing the quality of nonlinear models? I found very interesting thread on reddit Is r-squared useless with proofs from a statistics professor, ...
  • 1
0 votes
0 answers
9 views

Proper way of updating model metrics through time with MLflow?

I am new to MLflow and I am using it for model performance tracking. I want to keep track of how model metrics decay through time. My model predicts probability of events in future. Every month when I ...
1 vote
1 answer
32 views

What is the proper way to evaluate ML model when training on time-dependent data?

I have a task where I predict a probability of an event happening every month, for which I am using LGBM model and MLflow for model performance tracking. My dataset consists of historical data. In my ...
1 vote
1 answer
20 views

Evaluating Models that Return Percentage Present of Multiple Classes

If there is a model that returns a vector of the amount of different classes present in the data as percentages, what would be a good way to evaluate it (with charts and/or statistics)? Say, for ...
  • 13
3 votes
2 answers
122 views

Binary classifier high overall ROC AUC but low in different bins

I'm trying to analyze the performance of a binary classifier on the test set on different ranges of the predictions. the classifier has a .97 ROC AUC on the test. Then I binarize the test set ...
0 votes
1 answer
20 views

Is it good practice for Keras/TensorFlow users to rely on the validation set for testing?

Some sources consider a test/train split, such as with sklearn, to be expected practice, and validation is more or less reserved for k-fold validation. However, Keras has a somewhat different approach ...
  • 11
0 votes
1 answer
16 views

Validating classification results

I created a model for only 2 classes and the classification report was: Although accuracy looks good, I don't think this model is good. The original data has 522 records of class 1 and 123 of class 2....
  • 21
0 votes
0 answers
42 views

Song playlist recommendation system

I want to build a recommender system to suggest similar songs to continue a playlist (similar to what Spotify does by recommending similar songs at the end of a playlist). I want to build two models: ...
  • 1
4 votes
2 answers
81 views

Meaningfully compare target vs observed TPR & FPR

Suppose I have a binary classifier $f$ which acts on an input $x$. Given a threshold $t$, the predicted binary output is defined as: $$ \widehat{y} = \begin{cases} 1, & f(x) \geq t \\ 0, &...
1 vote
1 answer
25 views

Bias-variance trade-off and model evaluation

Suppose that we have train a model (as defined by its hyperparameters) and we evaluated it on a test set using some performance metric (say $R^2$). If we now train the same model (as defined by its ...
  • 151
0 votes
1 answer
18 views

How is model evaluation and re-training done after deployment without ground truth labels?

Suppose I deployed a model by manual labeling the ground truth labels with my training data, as the use case is such that there's no way to get the ground truth labels without humans. Once the model ...
  • 133
2 votes
0 answers
159 views

Is there a Mean Average Recall for Item Retrieval/ Recommendation Systems?

Mean Average Precision for Information retrieval is computed using Average Precision @ k (AP@k). AP@k is measured by first computing Precision @ k (P@k) and then averaging the P@k only for the k's ...
  • 21
2 votes
1 answer
92 views

Is data leakage giving me misleading results? Independent test set says no!

TLDR: I evaluated a classification model using 10-fold CV with data leakage in the training and test folds. The results were great. I then solved the data leakage and the results were garbage. I then ...
  • 21
0 votes
0 answers
46 views

Repeatability tests for machine learning models (in the sense of measurement system analysis)

For analyzing a machine learning model, we usually calculate the model performance metrics (such as accuracy...) and during validation step make sure that the model has not overfitted. We can consider ...
  • 1
0 votes
0 answers
57 views

Comparison of performance of regression models for multi-regression tasks

I have a sample time-series dataset (23, 14291) a pivot table count for 24hrs for some users. After pre-processing, I have a dataset with (23, 200) shape. I filtered some of the columns/features which ...
  • 389
3 votes
2 answers
609 views

Uncertainty about shape of ROC curve

I am working on a binary classification and the plotted ROC curves that I am using for evaluation together with AUC, have seemed strange to me. Here is an example. I understand that ROC is a visual ...
  • 289
1 vote
0 answers
26 views

Baseline result is much better than state-of-the-art model

I am researching about Deep Learning based Intrusion Detection System. I found a paper on a well-known journal, which is considered as a state-of-the-art method in this research area, because it got ...
  • 11
0 votes
1 answer
11 views

Metrics in pediction different than evaluation

In general when you have already evaluated your model on unseen data (test set) and its RMSE is different than predictions RMSE, is it ok ? How much difference is fine and how to know that ?
1 vote
2 answers
126 views

Evaluation Metric for Imbalanced and Ordinal Classification

I'm looking for an ML evaluation metric that would work well with imbalanced and ordinal multiclass datasets: Imagine you want to predict the severity of a disease that has 4 grades of severity where ...
4 votes
1 answer
227 views

Choose ROC/AUC vs. precision/recall curve?

I am trying to get a clear understanding on various classification metrics, including knowing when to choose ROC/AUC as opposed to opting for the Precision/Recall curve. I am reading Aurélien Géron's ...
  • 289
0 votes
0 answers
47 views

How to evaluate model accuracy at tail of empirical distribution?

I am making a nonlinear regression on stationary dependent variable and I want to precisely forecast extreme values of this variable. So when my model predicts extreme values I want them to be highly ...
0 votes
0 answers
21 views

How can i adapt accuracy metric for multiclass classification?

I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
  • 486
0 votes
1 answer
293 views

Cluster Evaluation with Jaccard and Rand Index

I've clusterized my data according to 3 criteria in 3 groups. I used kmeans to obtain those cluster so the label for each cluster is random and changes at each script run. To evaluate the consistency ...
  • 1
0 votes
0 answers
57 views

How to calculate mAP for multi-label classification using output predictions?

I have a model which predicts the actions happening in a video clip. Once I get these predictions, I use some rules(set of if-else conditions) to come up with composite labels for eg. ...
0 votes
0 answers
79 views

Assess feature importance in Keras for one-hot-encoded categorical features

An important aspect of tuning a model is assessing feature importance. In Keras, how to assess the importance of a categorical feature which is one-hot encoded? E.g. if a categorical feature is ...
  • 165
0 votes
1 answer
23 views

Evaluate Text-to-speech without Human Involved?

I've explored text-to-speech evaluation matrices and they seem to used Mean Opinion Score (MOS) to evaluate a particular model. This matrice required humans to help to judge the model based on a scale ...
0 votes
2 answers
93 views

I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]

I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
  • 73
0 votes
1 answer
30 views

Comparing RMSEs of multiple test sets having different sizes

The data I have is a time series data (stock returns), and I am training a Random Forest Regressor on it. Total observations = 2499 To better evaluate the performance, I have implemented rolling ...
1 vote
1 answer
74 views

n_jobs=-1 or n_jobs=1?

I am confused regarding the n_jobs parameter used in some models and for CV. I know it is used for parallel computing, where it includes the number of processors specified in n_jobs parameter. So if I ...
  • 1,293
1 vote
1 answer
21 views

Performance measurement of an event extraction system

I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
0 votes
1 answer
29 views

Evaluation Metrics for Multiclass

How to obtain the Accuracy, Detection_Rate, False_Positive_Rate and ...
  • 45
1 vote
1 answer
39 views

Quantitative measure of the smoothness of learning curves

$\DeclareMathOperator{\loss}{loss}$ $\DeclareMathOperator{\AvgVar}{AvgVar}$ Lat's say we have some deep learning task. We have our model and two sets of hyperparameters $A$ and $B$. We train both ...
0 votes
0 answers
25 views

Linear regression to find differences between model performances

For one of my projects I needed to create classification models for each of many products. In order to see which classifier performs best, I created one SVM, RandomForest and Naive Bayes model for ...
  • 101
2 votes
0 answers
45 views

Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps: obtain prediction on test data using ...
  • 31
3 votes
1 answer
7k views

precision@k and recall@k

Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems. I ...
  • 131
2 votes
3 answers
508 views

Comparing ML models to baselines

When comparing ML models with baseline or "dummy" models, are there best practices for building and comparing baselines? I'm doing a binary classification task where 40% of the samples are ...