Questions tagged [model-evaluations]

This tag is meant to be used for questions related to how to evaluate a model performance, not only based on standard metrics, but also in the context of real use case applications. What is a good model might depend on many factors to take into account, to eventually get really useful data science applications.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Song playlist recommendation system

I want to build a recommender system to suggest similar songs to continue a playlist (similar to what Spotify does by recommending similar songs at the end of a playlist). I want to build two models: ...
user avatar
  • 1
4 votes
2 answers
76 views

Meaningfully compare target vs observed TPR & FPR

Suppose I have a binary classifier $f$ which acts on an input $x$. Given a threshold $t$, the predicted binary output is defined as: $$ \widehat{y} = \begin{cases} 1, & f(x) \geq t \\ 0, &...
user avatar
  • 181
1 vote
1 answer
17 views

Bias-variance trade-off and model evaluation

Suppose that we have train a model (as defined by its hyperparameters) and we evaluated it on a test set using some performance metric (say $R^2$). If we now train the same model (as defined by its ...
user avatar
  • 111
0 votes
1 answer
10 views

How is model evaluation and re-training done after deployment without ground truth labels?

Suppose I deployed a model by manual labeling the ground truth labels with my training data, as the use case is such that there's no way to get the ground truth labels without humans. Once the model ...
user avatar
  • 133
2 votes
0 answers
33 views

Is there a Mean Average Recall for Item Retrieval/ Recommendation Systems?

Mean Average Precision for Information retrieval is computed using Average Precision @ k (AP@k). AP@k is measured by first computing Precision @ k (P@k) and then averaging the P@k only for the k's ...
user avatar
  • 21
2 votes
1 answer
74 views

Is data leakage giving me misleading results? Independent test set says no!

TLDR: I evaluated a classification model using 10-fold CV with data leakage in the training and test folds. The results were great. I then solved the data leakage and the results were garbage. I then ...
user avatar
  • 21
0 votes
0 answers
26 views

Repeatability tests for machine learning models (in the sense of measurement system analysis)

For analyzing a machine learning model, we usually calculate the model performance metrics (such as accuracy...) and during validation step make sure that the model has not overfitted. We can consider ...
user avatar
  • 1
0 votes
0 answers
13 views

Comparison of performance of regression models for multi-regression tasks

I have a sample time-series dataset (23, 14291) a pivot table count for 24hrs for some users. After pre-processing, I have a dataset with (23, 200) shape. I filtered some of the columns/features which ...
user avatar
  • 369
3 votes
2 answers
490 views

Uncertainty about shape of ROC curve

I am working on a binary classification and the plotted ROC curves that I am using for evaluation together with AUC, have seemed strange to me. Here is an example. I understand that ROC is a visual ...
user avatar
  • 257
1 vote
0 answers
24 views

Baseline result is much better than state-of-the-art model

I am researching about Deep Learning based Intrusion Detection System. I found a paper on a well-known journal, which is considered as a state-of-the-art method in this research area, because it got ...
user avatar
  • 11
0 votes
1 answer
11 views

Metrics in pediction different than evaluation

In general when you have already evaluated your model on unseen data (test set) and its RMSE is different than predictions RMSE, is it ok ? How much difference is fine and how to know that ?
user avatar
1 vote
2 answers
48 views

Evaluation Metric for Imbalanced and Ordinal Classification

I'm looking for an ML evaluation metric that would work well with imbalanced and ordinal multiclass datasets: Imagine you want to predict the severity of a disease that has 4 grades of severity where ...
user avatar
4 votes
1 answer
121 views

Choose ROC/AUC vs. precision/recall curve?

I am trying to get a clear understanding on various classification metrics, including knowing when to choose ROC/AUC as opposed to opting for the Precision/Recall curve. I am reading Aurélien Géron's ...
user avatar
  • 257
0 votes
0 answers
47 views

How to evaluate model accuracy at tail of empirical distribution?

I am making a nonlinear regression on stationary dependent variable and I want to precisely forecast extreme values of this variable. So when my model predicts extreme values I want them to be highly ...
user avatar
0 votes
0 answers
18 views

How can i adapt accuracy metric for multiclass classification?

I have a problem which is multiclass e.g. That is 4 classes. I would like a custom metric to assess the model where only if class 3 is predicted as class 2 and class 2 is predicted as class 3 (i.e. ...
user avatar
  • 466
0 votes
1 answer
121 views

Cluster Evaluation with Jaccard and Rand Index

I've clusterized my data according to 3 criteria in 3 groups. I used kmeans to obtain those cluster so the label for each cluster is random and changes at each script run. To evaluate the consistency ...
user avatar
  • 1
0 votes
0 answers
21 views

How to calculate mAP for multi-label classification using output predictions?

I have a model which predicts the actions happening in a video clip. Once I get these predictions, I use some rules(set of if-else conditions) to come up with composite labels for eg. ...
user avatar
0 votes
0 answers
57 views

Assess feature importance in Keras for one-hot-encoded categorical features

An important aspect of tuning a model is assessing feature importance. In Keras, how to assess the importance of a categorical feature which is one-hot encoded? E.g. if a categorical feature is ...
user avatar
  • 165
0 votes
1 answer
19 views

Evaluate Text-to-speech without Human Involved?

I've explored text-to-speech evaluation matrices and they seem to used Mean Opinion Score (MOS) to evaluate a particular model. This matrice required humans to help to judge the model based on a scale ...
user avatar
0 votes
2 answers
81 views

I am attempting to implement k-folds cross validation in python3. What is the best way to implement this? Is it preferable to use Pandas or Numpy? [closed]

I am attempting to create a script to implement cross validation in data. However, the splits cannot randomly take any records, so the training and testing can be done on equal data splits for each ...
user avatar
  • 73
0 votes
1 answer
18 views

Comparing RMSEs of multiple test sets having different sizes

The data I have is a time series data (stock returns), and I am training a Random Forest Regressor on it. Total observations = 2499 To better evaluate the performance, I have implemented rolling ...
user avatar
1 vote
1 answer
49 views

n_jobs=-1 or n_jobs=1?

I am confused regarding the n_jobs parameter used in some models and for CV. I know it is used for parallel computing, where it includes the number of processors specified in n_jobs parameter. So if I ...
user avatar
  • 1,241
0 votes
0 answers
12 views

What could be reasons for higher MAPE?

I built two models on a dataset where data for independent variables (X) being the same and dependent variable (Y) changes for each model eg : Y is price value for a particular region. Y values change ...
user avatar
  • 21
1 vote
1 answer
20 views

Performance measurement of an event extraction system

I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
user avatar
0 votes
1 answer
19 views

Evaluation Metrics for Multiclass

How to obtain the Accuracy, Detection_Rate, False_Positive_Rate and ...
user avatar
  • 45
1 vote
1 answer
34 views

Quantitative measure of the smoothness of learning curves

$\DeclareMathOperator{\loss}{loss}$ $\DeclareMathOperator{\AvgVar}{AvgVar}$ Lat's say we have some deep learning task. We have our model and two sets of hyperparameters $A$ and $B$. We train both ...
user avatar
0 votes
0 answers
25 views

Linear regression to find differences between model performances

For one of my projects I needed to create classification models for each of many products. In order to see which classifier performs best, I created one SVM, RandomForest and Naive Bayes model for ...
user avatar
  • 101
2 votes
0 answers
45 views

Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps: obtain prediction on test data using ...
user avatar
  • 31
3 votes
1 answer
5k views

precision@k and recall@k

Normally, I am familiar with precision and recall evaluation metrics but as you know recall@k and precision@k are different things and used in ranking evaluations especially recommendation systems. I ...
user avatar
  • 131
2 votes
3 answers
313 views

Comparing ML models to baselines

When comparing ML models with baseline or "dummy" models, are there best practices for building and comparing baselines? I'm doing a binary classification task where 40% of the samples are ...
user avatar