Skip to main content
5 votes
Accepted

For cross validation should I use training set, or whole dataset?

It's generally recommended to use the training set for cross-validation to avoid data leakage. The primary role of the test set is to provide an unbiased evaluation of a model's performance. If you ...
Rodrigo Ferro's user avatar
3 votes
Accepted

Xgboost model predicting extreme values for events and non-events | Overfitting

This is not necessarily overfitting, but it may indicate data leakage i.e you are passing information to the model that is not supposed to be there it may be: Information that is generated after the ...
Multivac's user avatar
  • 2,999
2 votes
Accepted

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

If you want to avoid annotate the data yourself, you can try to evaluate the model on one or more benchmark datasets that are available for sentiment analysis. This may work unless the model is very ...
Luca Anzalone's user avatar
2 votes
Accepted

Implementation of spBLEU

spBLEU was introduced in the Flores-101 article: [...] we propose to use BLEU over text tokenized with a single language-agnostic and publicly available fixed SentencePiece subword model. We call ...
noe's user avatar
  • 27k
2 votes

For cross validation should I use training set, or whole dataset?

The test and train set should be different. The Cross Validate set comes in training set. The test data should be totally unseen by the model. While there is no hard and fast rule for making test ...
Banarasi Vaibhav's user avatar
1 vote

How to measure different models' feature importance using a generic and common standard?

Permutation importance is a relatively simply model-agnostic approach. You train and score a model the usual way to get a reference score. Then you take each feature in turn, and score the model after ...
MuhammedYunus's user avatar
1 vote

How do I work with time-series data of temperature?

Following Andrew Ng's approach to machine learning, "make a quick & dirty model." When you're unsure on how to solve some prediction task (and there's no guiding literature on the ...
F-said's user avatar
  • 71
1 vote

Reduce mode searching behaviour of VAE

VAE objective is to maximise the ELBO: $$ \int_x p(x) \int_z q(z|x) \big\{ \log p(x|z) - \log \tfrac{q(z|x)}{p(z)} \big\} $$ the first term reconstructs the data, the second term maximises the ...
Carl's user avatar
  • 396
1 vote
Accepted

How to interpret RMSE to evaluate a regression model

You didn’t tell us about the use case or business domain for your problem. For example, if you were modeling battery energy consumption in noise canceling headphones, root mean squared error would be ...
J_H's user avatar
  • 1,130
1 vote

Topic modeling evaluation

Evaluating unsupervised learning methods is always an interesting question. There are typically two main ways to evaluate clusters. Explicit evaluation Qualitative analysis First of all, you should ...
Valentin Calomme's user avatar
1 vote

Understand and compute confidence interval and coefficient of variation for regression model

Don't think xgbregressor is that similar to linear regression, so you might be better with understanding the latter first. You took quotes from wiki, but do you understand how it works in practice? ...
Cryo's user avatar
  • 553
1 vote

One class classification using Exclusively Positive Examples

Sure, you can use a OneClassSVM model or an IsolationForest (read also this). Basically, you feed them the data you have (regardless of the class label) and the model scores your data (according to an ...
Luca Anzalone's user avatar
1 vote

Xgboost model predicting extreme values for events and non-events | Overfitting

A high score on the test set does not indicate overfitting. See Why 100% accuracy on test data is not good? ; you're not quite reaching perfect performance, but you're quite close, and in that seeing ...
Ben Reiniger's user avatar
  • 12k
1 vote

Different accuracy scores with sklearn roc_auc_score on same model using sklearn.metrics

There is a fundamental difference between .predict() and .predict_proba(). The former does ...
Luca Anzalone's user avatar
1 vote

eval_metric of XGBoost // ML model in general

Which metric to choose is not related to the model, but to the problem to be solved. If you are unsure, go back and think about the objective - why we need to build the model, and what the model needs ...
lpounng's user avatar
  • 1,094
1 vote
Accepted

How to compare test vs train model performance

When comparing models, the main objective is often to choose the one that performs well on unseen data, that is, the model that has a good generalization ability. This means you'd typically prefer the ...
Dipanwita Mallick's user avatar
1 vote

Which is the best binary classification model? Train and Test Accuracy are similar

Unfortunately, there is not golden rule which model is the best. It always depends on the use case and what is done with the model. In general, the metrics allow to compare different aspects, but it ...
Broele's user avatar
  • 1,535
1 vote
Accepted

Can I use GridSearchCV.best_score_ for evaluation of model performance?

Yes, the GridSearchCV.best_score_ should not be used as a final measure of model performance. The reason is that this score is optimistic, it is the best score obtained on the validation set during ...
Harshad Patil's user avatar
1 vote
Accepted

Why do residuals of linear regression model need to be normally distributed?

Let's take a step back and think: if my predictions are good, how should they compare to the ground truth? The predictions should be close to the ground truth (well, which is why we do prediction in ...
lpounng's user avatar
  • 1,094
1 vote

Why do residuals of linear regression model need to be normally distributed?

If you want a statistically significant result, I suggest you to verify the normality of the residuals distribution using the QQ-plot - you can find more informations about that here - or the Shapiro-...
PurpleCat's user avatar
1 vote

Appropriate objective function and evaluation metric when I DO care about outliers?

How to use least squares with weight matrix - use weights ...
JeeyCi's user avatar
  • 133

Only top scored, non community-wiki answers of a minimum length are eligible