What's the deal with Deno? We talk with a major contributor to find out. Listen now.

Questions tagged [model-selection]

Model selection is the process of comparing several models and their respective results to choose the model is best according to some evaluation metric.

Filter by
Sorted by
Tagged with
1
vote
0answers
7 views

Model selection in active learning

I am dabbling in active learning and was wondering how to combine this in seeking out the best architecture for the network. In my understanding, active learning uses a heuristic for selecting the ...
1
vote
0answers
22 views

Comparing different machine learning methods over multiple test datasets with different number of samples [closed]

Say, I have an image dataset (for example, imagenet) and I am training two image recognition models on it. I train a resnet with 10 layers 3 times on it (each time with different random weight ...
0
votes
1answer
29 views

model selection in clustering

I am working on a mall customer segmentation dataset (5 features, 200 rows) using clustering. This dataset does not have any ground truth labels. I had a few doubts regarding clustering: Can I use ...
4
votes
3answers
197 views

Is autocorrelation of residuals a problem in machine learning?

Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem? As an example, let's assume I have two different random forest models, A and B, with a ...
0
votes
2answers
21 views

Selecting Transforms with sklearn pipelines

So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do : ...
1
vote
1answer
29 views

Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA). I scaled both the input values ...
-1
votes
3answers
56 views

Is it a good practice to evaluate a model on the training set

Is it a good practice to evaluate a model on the training set (i.e. train a model on training set and evaluate the regression error/accuracy on the same training set) and compare the evaluation result ...
5
votes
6answers
222 views

Is there any way to explicitly measure the complexity of a Machine Learning Model in Python

I'm interested in model debuggin and one of the points that it recommends is to compare your model with a "less complex" one in order to see if the performance is substantially better on the ...
0
votes
0answers
10 views

Model selection Tensorflow for custom dataset comparing panoramic images vs regular images (Image segmentation)

I have a question regarding the use of a specific model such as Deeplab and how to create a custom dataset for it. Background To give a bit of background info, I want to compare panoramically stitched ...
0
votes
1answer
19 views

Determining which categorical data is beneficial in predictive modelling

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
0
votes
1answer
18 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
0
votes
1answer
33 views

Population stability Index vs Population Accuracy Index

Can anyone explain to me the difference between Population Stability Index(PSI) and Population Accuracy Index(PAI)?
0
votes
1answer
41 views

How do I know what the best number of layers is required to achieve the highest accuracy

I'm learning from Udacity using this video. I saw this piece of code: ...
0
votes
3answers
30 views

how to use predictions on a single value?

I am comfortable using Machine learning on my train data and test data and validate it. But the question here is if I want to predict a single variable how do I do it? Let's suppose I have done ...
0
votes
0answers
5 views

Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
0
votes
0answers
8 views

Comparing models when their performance may depend on a continuous variable

I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form ...
1
vote
1answer
26 views

What probability distribution would be more appropriate for monthly rate of going to the store?

Part of a model I am making includes the frequency with which people go shopping for a given good (e.g. people on average go to the supermarket some n times a month). I am trying to figure out what ...
1
vote
1answer
37 views

is it possible get a overfit underfit comparation between models, with this chart? (homework) [closed]

I am trying to interpret this chart. I am not sure how to interpret this, because, I think that the fact of the for examples LGBM Validation error, is wide and similar to train boxplot, there arent ...
0
votes
0answers
16 views

Is recommendable look for high variance when your data is imbalance?

Hello I have a dataset with the following classes A, B, C, and this classes have the following representation of the dataset 60% 39% 1%. Is it a good idea try to get a model with high variance in this ...
0
votes
0answers
11 views

Vader vs TextBlob opposite outcome: why?

I've been studying for a Data Science course and yesterday I was challenged with a sentiment analysis, for which tons of material can be found online. So bear with me, ad I'm trying to get to the ...
1
vote
2answers
31 views

How do I select the “best” unsupervised machine learning algorithm to cluster my specific dataset?

I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
0
votes
1answer
15 views

Propensity model with Only Positive Data

Is it possible to build a propensity model (i.e., the likelihood that a user will buy an item) using only positive values. For example, I have a bunch of data about Customers (people that bought stuff)...
2
votes
1answer
23 views

What supervised machine learning model can be used to generate a scorecard-like result?

A scorecard is typically used in Credit Application. One very common model for developing a credit scorecard is logistic regression since it has well-defined probabilities. Apart from logistic ...
3
votes
4answers
85 views

ML: Classification Model Comparison

Given is a dataset that I need to use for a classification and I want to compare the performance of different classification models. Let's assume, I want to look at logistic regression (with ...
0
votes
0answers
4 views

{Thoughts on Model Design} Predicting Default Rate Time Series as a combination of lagged values of default rates and other exogenous time series

I have a time series that reflects the last twelve months default rate for every month-end since Jan 2001. I also have other exogenous variables like credit quality score etc. Before proceeding, my ...
-1
votes
1answer
30 views

Proper workflow for model selection and hyperparameter tuning using cross validation

I have been trying to teach myself about machine learning and wanted to make sure I had the right idea about model selection, hyperparameter tuning, and cross validation. So given a data set, my ...
0
votes
1answer
33 views

How to compare performance between SVM and Keras models

I applied both SVM and CNN (using Keras) on a dataset. Now, I want to compare the performance of both models. Keras model.evaluate function predicts the output for the given input and then computes ...
1
vote
0answers
53 views

Learning based on Vector Inputs and Scalar Output

I'm trying to fit a function of the following form to some time series data that I have... Z = X/(a*Y+b) a and ...
0
votes
1answer
26 views

Effecient way to decompose multiple time series in a data frame and compare the fit of additive and multiplicative models?

I have a data frame in R that contains time series data of 7 variables that were taken on several hundred different individuals. I want to know if it would be more appropriate to use an additive model ...
0
votes
0answers
339 views

Which of the following is a consequence of Selecting model complexity on test data? (multiple right answers)

Selecting model complexity on test data (choose all that apply): A. Allows you to avoid issues of overfitting to training data B. Provides an overly optimistic assessment of performance of the ...
1
vote
0answers
24 views

What to do when feature engineering and parameter tuning don't add to the base model performance

I've been working on using LogisticRegression from scikit to try the Titanic Kaggle comp. I've found something interesting, and that is that no amount of feature engineering and paramater tuning is ...
0
votes
0answers
11 views

Comparison of Time series models

How to compare two different forecasting models lets say one is the classical statistics-based model and the other is a machine learning-based. Also, let's say if I have to forecast the unit volume ...
0
votes
0answers
15 views

Which algorithm and/or libraries to use for prediction and probability matching of event (per second) data having numerical values?

Suggest algorithm to match input data with existing records and calculate matching probability. Based on matching probability predict the output similar to a matching record. Data contains per second ...
1
vote
0answers
16 views

What is the better model architecture and setting when using merge layers?

I am building a deep learning model with dense, dropout, and merge layers. The inputs will be N sentences' feature encoded by BERT (768 dim) and then each will go into the same dense layer as the ...
1
vote
1answer
17 views

Model selection metric for validation phase in deep learning

I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This ...
0
votes
1answer
127 views

What is the selection criteria to choose between XGBoost and Random Forest

I am trying to understand - when would someone choose Random Forest over XGBoost and vice versa. All the articles out there highlights on the differences between both. I understand them. But when ...
1
vote
1answer
25 views

Asynchronous Hyperparameter Optimization - Dependency between iterations

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
0
votes
0answers
16 views

Which machine learning technique can be used for predicting customer behavior? The description of the case is mentioned below

The customers apply request for getting an estimated amount for the service. Then the customer decides if they want the service or not. In this case there are multiple customers and multiple services ...
0
votes
0answers
17 views

Linear Mixed Model vs Variable Interaction

I am studying relationship between 'Y' and two continuous vars(X1, X2) + one categorical var (C). I think the categorical variable not only influence 'Y', but also changes coefficients of continuous ...
2
votes
1answer
24 views

How to select the best model from validation/training/holdout accuracy score

I have made my own function to log all the attempts at hyperparamter tuning, the following information is gathered from a 10 fold cross validation. But I am struggling to work out which model is best....
1
vote
2answers
81 views

Same validation accuracy, different train accuracy for two neural networks models

I'm performing emotion classification over FER2013 dataset. I'm trying to measure different models performance, and when I checked ImageDataGenerator with a model I had already used I came up with the ...
1
vote
1answer
21 views

How to archive trained PyTorch models?

Currently, I am working on my thesis which is built on LSTM networks and I am using PyTorch library. However I am struggling to ...
2
votes
2answers
173 views

Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters?

I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, ...
3
votes
1answer
93 views

Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model?

I have experimented with a number of different machine learning models used for pose estimation. Most of them output a heatmap and offsets for the detected person(s) in the image. I really like the ...
2
votes
2answers
61 views

How can I approach this problem?

Let's say I have a dataset with pricing information for the same flight during the past year. So, for a flight departing on day D, I have the available price from D-130 up to D (departure day). Then ...
2
votes
2answers
139 views

Is k-means with Mahalanobis a valid option for clustering?

I want more info into if k-means with Mahalanobis distance is a mathematically/methodologically correct option for datasets with different variance clusters. The steps are: Create aggregate datasets (...
3
votes
1answer
711 views

difference between empirical risk minimization and structural risk minimization?

I understand the meaning of empirical risk minimization as separate topic and was reading about structural risk minimization, it is hard for me to understand the difference between these two. I read ...
2
votes
1answer
58 views

Machine learning solution approach to match loan repayments

I'm relatively new to the AI/ML space, but come from a programming background. The problem: I have a dataset of users transactions who have taken short-term loans from a single loan provider and I ...
3
votes
0answers
26 views

Improving a simple trig model

I have some data which I know is well approximated as a trig function, and I can fit it with scipy.optimize.curve_fit as follows: ...