Questions tagged [model-selection]

Model selection is the process of comparing several models and their respective results to choose the model is best according to some evaluation metric.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

{Thoughts on Model Design} Predicting Default Rate Time Series as a combination of lagged values of default rates and other exogenous time series

I have a time series that reflects the last twelve months default rate for every month-end since Jan 2001. I also have other exogenous variables like credit quality score etc. Before proceeding, my ...
-1
votes
1answer
19 views

Proper workflow for model selection and hyperparameter tuning using cross validation

I have been trying to teach myself about machine learning and wanted to make sure I had the right idea about model selection, hyperparameter tuning, and cross validation. So given a data set, my ...
0
votes
1answer
21 views

How to compare performance between SVM and Keras models

I applied both SVM and CNN (using Keras) on a dataset. Now, I want to compare the performance of both models. Keras model.evaluate function predicts the output for the given input and then computes ...
1
vote
0answers
49 views

Learning based on Vector Inputs and Scalar Output

I'm trying to fit a function of the following form to some time series data that I have... Z = X/(a*Y+b) a and ...
0
votes
1answer
21 views

Effecient way to decompose multiple time series in a data frame and compare the fit of additive and multiplicative models?

I have a data frame in R that contains time series data of 7 variables that were taken on several hundred different individuals. I want to know if it would be more appropriate to use an additive model ...
0
votes
0answers
70 views

Which of the following is a consequence of Selecting model complexity on test data? (multiple right answers)

Selecting model complexity on test data (choose all that apply): A. Allows you to avoid issues of overfitting to training data B. Provides an overly optimistic assessment of performance of the ...
1
vote
0answers
24 views

What to do when feature engineering and parameter tuning don't add to the base model performance

I've been working on using LogisticRegression from scikit to try the Titanic Kaggle comp. I've found something interesting, and that is that no amount of feature engineering and paramater tuning is ...
0
votes
0answers
10 views

Comparison of Time series models

How to compare two different forecasting models lets say one is the classical statistics-based model and the other is a machine learning-based. Also, let's say if I have to forecast the unit volume ...
0
votes
0answers
14 views

Which algorithm and/or libraries to use for prediction and probability matching of event (per second) data having numerical values?

Suggest algorithm to match input data with existing records and calculate matching probability. Based on matching probability predict the output similar to a matching record. Data contains per second ...
1
vote
0answers
13 views

What is the better model architecture and setting when using merge layers?

I am building a deep learning model with dense, dropout, and merge layers. The inputs will be N sentences' feature encoded by BERT (768 dim) and then each will go into the same dense layer as the ...
1
vote
1answer
16 views

Model selection metric for validation phase in deep learning

I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This ...
0
votes
1answer
54 views

What is the selection criteria to choose between XGBoost and Random Forest

I am trying to understand - when would someone choose Random Forest over XGBoost and vice versa. All the articles out there highlights on the differences between both. I understand them. But when ...
1
vote
1answer
23 views

Asynchronous Hyperparameter Optimization - Dependency between iterations

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
0
votes
0answers
16 views

Which machine learning technique can be used for predicting customer behavior? The description of the case is mentioned below

The customers apply request for getting an estimated amount for the service. Then the customer decides if they want the service or not. In this case there are multiple customers and multiple services ...
0
votes
0answers
16 views

Linear Mixed Model vs Variable Interaction

I am studying relationship between 'Y' and two continuous vars(X1, X2) + one categorical var (C). I think the categorical variable not only influence 'Y', but also changes coefficients of continuous ...
2
votes
1answer
24 views

How to select the best model from validation/training/holdout accuracy score

I have made my own function to log all the attempts at hyperparamter tuning, the following information is gathered from a 10 fold cross validation. But I am struggling to work out which model is best....
0
votes
1answer
46 views

Same validation accuracy, different train accuracy for two neural networks models

I'm performing emotion classification over FER2013 dataset. I'm trying to measure different models performance, and when I checked ImageDataGenerator with a model I had already used I came up with the ...
1
vote
0answers
16 views

A Productive way to archive trained models?

Currently, I am working on my thesis which is built on LSTM networks and I am using PyTorch library. However I am struggling to ...
2
votes
1answer
100 views

Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters?

I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, ...
2
votes
1answer
83 views

Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model?

I have experimented with a number of different machine learning models used for pose estimation. Most of them output a heatmap and offsets for the detected person(s) in the image. I really like the ...
1
vote
2answers
55 views

How can I approach this problem?

Let's say I have a dataset with pricing information for the same flight during the past year. So, for a flight departing on day D, I have the available price from D-130 up to D (departure day). Then ...
2
votes
2answers
49 views

Is k-means with Mahalanobis a valid option for clustering?

I want more info into if k-means with Mahalanobis distance is a mathematically/methodologically correct option for datasets with different variance clusters. The steps are: Create aggregate datasets (...
3
votes
1answer
258 views

difference between empirical risk minimization and structural risk minimization?

I understand the meaning of empirical risk minimization as separate topic and was reading about structural risk minimization, it is hard for me to understand the difference between these two. I read ...
2
votes
1answer
49 views

Machine learning solution approach to match loan repayments

I'm relatively new to the AI/ML space, but come from a programming background. The problem: I have a dataset of users transactions who have taken short-term loans from a single loan provider and I ...
3
votes
0answers
24 views

Improving a simple trig model

I have some data which I know is well approximated as a trig function, and I can fit it with scipy.optimize.curve_fit as follows: ...
1
vote
1answer
183 views

Need of Weighted Mean Squared Error

We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix. Below is the WMSE formula: Can ...
1
vote
1answer
29 views

How can I compare my regressors?

I am trying to build a regressor for a dataset which gives info about students' school performance and the probability of getting admitted in the University of their choice. The first 5 observations ...
3
votes
2answers
62 views

How best to show the best model over multiple labels?

I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on. I get a very messy plot when I naively try to place them one on top of the other. ...
2
votes
2answers
197 views

What is my training score the mean_train_score or mean_test_score?

I am using sklearn to train some models (random forest, decision tree). For the training I am using RandomsearchCV with Stratified k-fold as cross-validation. Then I make a predictions on the test set ...
2
votes
3answers
53 views

Can someone explain what batch size is doing in convolutional NNs?

I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was ...
1
vote
2answers
110 views

model with features of different sizes

I want to train a model (either classification or regression, doesn't matter) with features/inputs of different sizes, but I am not sure how to do it. For example, for each data-point, feature 1 and ...
0
votes
2answers
39 views

What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
1
vote
1answer
36 views

Association between features

Given the anonymized dataset of features below, where: "code" is a categorical variable. "x1" and "x2" are continuous variables. "x3" and "x4" are extracted features. They are the mean values of ...
1
vote
1answer
31 views

How to choose a model for this cross-validation curve?

I'm using GridSearchCV to tune hyperparameters for a Logistic Regression multiclass model. I read on Kaggle that you should choose the hyperparameter that results in the lowest discrepancy between ...
1
vote
1answer
32 views

Language modelling for Spell Checker

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use Word-Level modelling Character-Level modelling plus I am preferring neural networks over ...
1
vote
1answer
185 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
0
votes
1answer
24 views

Simplest way to build a semantic analyzer

I want to build a semantic analyzer i.e., to find how similar the meaning of two sentences are. For example- English: Birdie is washing itself in the water basin. English Paraphrase: The bird is ...
0
votes
0answers
6 views

Is there an ISO or other standard or best practice for structuring records related to organisations?

I'm working on a project which includes redesigning the way an application represents data about organisations. The current design is quite "flat" - meaning for example that if I want to add a new ...
2
votes
1answer
1k views

svm.LinearSVC: larger max_iter number doesn't always increase the accuracy/precision/recall

Background: Supervised machine learning Data shape 10+ features, target = 1 or 0 only, 100,000+ samples (so should be no issue of over-sampling) 80% training, 20% testing train_test_split(X_train, ...
0
votes
0answers
16 views

Model or algorithm for iterative optimization

Here is my problem : At every loop, I have new data that depends on the previous outputs. I need to approximate the function that optimizes (minimizes / maximizes) this new data on every iteration. ...
2
votes
2answers
74 views

What regression model can handle tiny amounts of data?

I'm trying to use machine learning to predict properties of a material during a crash test, but each data point requires physically crashing an expensive toy car, so I can only gather a few hundred ...
1
vote
1answer
62 views

Modeling strategy for predicting a day/hour based on my dataset

This is my first time posting here. I'm usually on SO. So I'm not sure if these kind of questions fit into DS stackexchange. I genuinely need opinions on this. What data do I have - ...
4
votes
1answer
79 views

help understanding nested cross validation

From what I read online, nested CV works as follows: I divide my whole data in k folds, where k-1 folds are the train set and one fold is the test set. There is the inner CV loop, where we may ...
0
votes
0answers
34 views

Do pseudo r squared metrics make sense for classifiers that aren't logistic regression?

I'm working with some domain scientists that are used to using logistic regression to predict a binary value. One of the ways they evaluate their logistic regression model is through the Nagelkerke $...
-1
votes
2answers
26 views

How to approach a problem when the information is in the relationship between the points, and not the points itself?

I am trying to analyze vehicular mobility models, where I am trying to learn how a particular vehicle moves and then detect similar patterns from the testing data. Here's what I have done for now: I ...
0
votes
0answers
31 views

Predicting house pricing given a dataset consisting of [ location: date of transaction: price ]

What would be the right way to tackle the problem of predicting median house pricing, given that the data I have for training consists in a big list of entries that have the following values: ...
2
votes
0answers
83 views

How to measure the stability of hyperparameter selection in a model-building procedure?

For my project I run several model-building-procedures. I use the mean and standard deviation of the test scores in the outer folds as an estimator for the generalizability of the model-building ...
3
votes
1answer
67 views

How to handle associated features in machine learning

I am working on a classification project in which some features are linked and I'm not sure how to handle them. I will simplify my project like that : There are different jobs, and multiple ...
0
votes
1answer
36 views

What model is recommended: I am using text features in a regression and want to interpret coefficients

I am using the text of comments on a forum to predict how many upvotes it will get. I want to be able to say, "Reviews with X, Y, Z words are more upvoted". So to do this, I want to use text features ...