Questions tagged [model-selection]

Model selection is the process of comparing several models and their respective results to choose the model is best according to some evaluation metric.

Filter by
Sorted by
Tagged with
0
votes
2answers
27 views

Selecting Transforms with sklearn pipelines

So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do : ...
1
vote
1answer
102 views

Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA). I scaled both the input values ...
0
votes
3answers
917 views

Is it a good practice to evaluate a model on the training set

Is it a good practice to evaluate a model on the training set (i.e. train a model on training set and evaluate the regression error/accuracy on the same training set) and compare the evaluation result ...
6
votes
6answers
1k views

Is there any way to explicitly measure the complexity of a Machine Learning Model in Python

I'm interested in model debugging and one of the points that it mentions is to compare your model with a "less complex" one in order to check if the performance is substantially better on ...
0
votes
1answer
22 views

Determining which categorical data is beneficial in predictive modelling

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
1
vote
1answer
44 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
0
votes
1answer
245 views

Population stability Index vs Population Accuracy Index

Can anyone explain to me the difference between Population Stability Index(PSI) and Population Accuracy Index(PAI)?
0
votes
1answer
48 views

How do I know what the best number of layers is required to achieve the highest accuracy

I'm learning from Udacity using this video. I saw this piece of code: ...
0
votes
3answers
210 views

how to use predictions on a single value?

I am comfortable using Machine learning on my train data and test data and validate it. But the question here is if I want to predict a single variable how do I do it? Let's suppose I have done ...
1
vote
1answer
35 views

What probability distribution would be more appropriate for monthly rate of going to the store?

Part of a model I am making includes the frequency with which people go shopping for a given good (e.g. people on average go to the supermarket some n times a month). I am trying to figure out what ...
1
vote
1answer
59 views

is it possible get a overfit underfit comparation between models, with this chart? (homework) [closed]

I am trying to interpret this chart. I am not sure how to interpret this, because, I think that the fact of the for examples LGBM Validation error, is wide and similar to train boxplot, there arent ...
1
vote
2answers
65 views

How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?

I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
1
vote
1answer
47 views

Propensity model with Only Positive Data

Is it possible to build a propensity model (i.e., the likelihood that a user will buy an item) using only positive values. For example, I have a bunch of data about Customers (people that bought stuff)...
2
votes
1answer
48 views

What supervised machine learning model can be used to generate a scorecard-like result?

A scorecard is typically used in Credit Application. One very common model for developing a credit scorecard is logistic regression since it has well-defined probabilities. Apart from logistic ...
3
votes
4answers
206 views

ML: Classification Model Comparison

Given is a dataset that I need to use for a classification and I want to compare the performance of different classification models. Let's assume, I want to look at logistic regression (with ...
-1
votes
1answer
128 views

Proper workflow for model selection and hyperparameter tuning using cross validation

I have been trying to teach myself about machine learning and wanted to make sure I had the right idea about model selection, hyperparameter tuning, and cross validation. So given a data set, my ...
0
votes
1answer
209 views

How to compare performance between SVM and Keras models

I applied both SVM and CNN (using Keras) on a dataset. Now, I want to compare the performance of both models. Keras model.evaluate function predicts the output for the given input and then computes ...
1
vote
0answers
57 views

Learning based on Vector Inputs and Scalar Output

I'm trying to fit a function of the following form to some time series data that I have... Z = X/(a*Y+b) a and ...
0
votes
1answer
285 views

Effecient way to decompose multiple time series in a data frame and compare the fit of additive and multiplicative models?

I have a data frame in R that contains time series data of 7 variables that were taken on several hundred different individuals. I want to know if it would be more appropriate to use an additive model ...
1
vote
0answers
697 views

Which of the following is a consequence of Selecting model complexity on test data? (multiple right answers)

Selecting model complexity on test data (choose all that apply): A. Allows you to avoid issues of overfitting to training data B. Provides an overly optimistic assessment of performance of the ...
1
vote
0answers
27 views

What to do when feature engineering and parameter tuning don't add to the base model performance

I've been working on using LogisticRegression from scikit to try the Titanic Kaggle comp. I've found something interesting, and that is that no amount of feature engineering and paramater tuning is ...
1
vote
0answers
19 views

What is the better model architecture and setting when using merge layers?

I am building a deep learning model with dense, dropout, and merge layers. The inputs will be N sentences' feature encoded by BERT (768 dim) and then each will go into the same dense layer as the ...
1
vote
1answer
50 views

Model selection metric for validation phase in deep learning

I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This ...
1
vote
1answer
560 views

What is the selection criteria to choose between XGBoost and Random Forest

I am trying to understand - when would someone choose Random Forest over XGBoost and vice versa. All the articles out there highlights on the differences between both. I understand them. But when ...
1
vote
1answer
29 views

Asynchronous Hyperparameter Optimization - Dependency between iterations

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
2
votes
1answer
39 views

How to select the best model from validation/training/holdout accuracy score

I have made my own function to log all the attempts at hyperparamter tuning, the following information is gathered from a 10 fold cross validation. But I am struggling to work out which model is best....
1
vote
2answers
292 views

Same validation accuracy, different train accuracy for two neural networks models

I'm performing emotion classification over FER2013 dataset. I'm trying to measure different models performance, and when I checked ImageDataGenerator with a model I had already used I came up with the ...
1
vote
1answer
46 views

How to archive trained PyTorch models?

Currently, I am working on my thesis which is built on LSTM networks and I am using PyTorch library. However I am struggling to ...
2
votes
2answers
706 views

Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters?

I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, ...
3
votes
1answer
178 views

Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model?

I have experimented with a number of different machine learning models used for pose estimation. Most of them output a heatmap and offsets for the detected person(s) in the image. I really like the ...
3
votes
2answers
82 views

How can I approach this problem?

Let's say I have a dataset with pricing information for the same flight during the past year. So, for a flight departing on day D, I have the available price from D-130 up to D (departure day). Then ...
3
votes
2answers
1k views

Is k-means with Mahalanobis a valid option for clustering?

I want more info into if k-means with Mahalanobis distance is a mathematically/methodologically correct option for datasets with different variance clusters. The steps are: Create aggregate datasets (...
5
votes
1answer
3k views

difference between empirical risk minimization and structural risk minimization?

I understand the meaning of empirical risk minimization as separate topic and was reading about structural risk minimization, it is hard for me to understand the difference between these two. I read ...
2
votes
1answer
84 views

Machine learning solution approach to match loan repayments

I'm relatively new to the AI/ML space, but come from a programming background. The problem: I have a dataset of users transactions who have taken short-term loans from a single loan provider and I ...
3
votes
0answers
39 views

Improving a simple trig model

I have some data which I know is well approximated as a trig function, and I can fit it with scipy.optimize.curve_fit as follows: ...
2
votes
2answers
4k views

Need of Weighted Mean Squared Error

We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix. Below is the WMSE formula: Can ...
2
votes
0answers
25 views

Interpretable models apart from Logistic Regression

I am wondering about other interpretable models apart from logistic regression. I am looking for models that can interpret the effect on the target variable by unit change in any feature variable. I ...
1
vote
1answer
41 views

How can I compare my regressors?

I am trying to build a regressor for a dataset which gives info about students' school performance and the probability of getting admitted in the University of their choice. The first 5 observations ...
3
votes
3answers
91 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
2
votes
2answers
1k views

What is my training score the mean_train_score or mean_test_score?

I am using sklearn to train some models (random forest, decision tree). For the training I am using RandomsearchCV with Stratified k-fold as cross-validation. Then I make a predictions on the test set ...
2
votes
3answers
90 views

Can someone explain what batch size is doing in convolutional NNs?

I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was ...
1
vote
2answers
263 views

model with features of different sizes

I want to train a model (either classification or regression, doesn't matter) with features/inputs of different sizes, but I am not sure how to do it. For example, for each data-point, feature 1 and ...
1
vote
2answers
367 views

What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
1
vote
1answer
48 views

Association between features

Given the anonymized dataset of features below, where: "code" is a categorical variable. "x1" and "x2" are continuous variables. "x3" and "x4" are extracted features. They are the mean values of ...
1
vote
1answer
111 views

How to choose a model for this cross-validation curve?

I'm using GridSearchCV to tune hyperparameters for a Logistic Regression multiclass model. I read on Kaggle that you should choose the hyperparameter that results in the lowest discrepancy between ...
1
vote
1answer
230 views

Language modelling for Spell Checker

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use Word-Level modelling Character-Level modelling plus I am preferring neural networks over ...
1
vote
1answer
827 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
1
vote
1answer
35 views

Simplest way to build a semantic analyzer

I want to build a semantic analyzer i.e., to find how similar the meaning of two sentences are. For example- English: Birdie is washing itself in the water basin. English Paraphrase: The bird is ...
4
votes
1answer
4k views

svm.LinearSVC: larger max_iter number doesn't always increase the accuracy/precision/recall

Background: Supervised machine learning Data shape 10+ features, target = 1 or 0 only, 100,000+ samples (so should be no issue of over-sampling) 80% training, 20% testing train_test_split(X_train, ...