Questions tagged [model-selection]
Model selection is the process of comparing several models and their respective results to choose the model is best according to some evaluation metric.
251 questions
2
votes
1
answer
28
views
Problem: Logistic Regression vs. Tree-based Model Performance on Small Imbalanced Dataset
I have a small dataset with 300 rows and an imbalanced class distribution (35% positives). When I fit a logistic regression model, it consistently performs worse than random chance. I’m using ...
- 121
0
votes
0
answers
23
views
Can I compare mean absolute shap values for different models?
I am comparing 3 different classifiers ANN, XG Boost and Random Forest in making predictions. I also used SHAP for feature importance. I am only interested in the top 10 features based on SHA. The 3 ...
0
votes
0
answers
10
views
Question regarding an implementation of OLS subset selection
On this website I found a Python implementation of Best Subset Selection, Forward Stepwise Selection etc. for choosing a set of predictors in an ordinary linear regression. It is supposed to be a ...
- 101
1
vote
0
answers
40
views
What is the standard ML pipeline for training and testing? [closed]
I have a dataframe containing 1324 rows and 28 columns and I'm kinda lost on which approach to go for when training regression models. Currently I perform a data split and run GridSearchCV to pick the ...
0
votes
0
answers
17
views
How to broadly structure a Record Linkage model for Emails on top of a Vector Embedding model for Semantic Search with entities?
Sorry the broad and naive question, but the structure I have in mind is as follows:
Extract the text from a large collection of Documents with varying types.
This part I plan to use Apache Tika and ...
- 101
0
votes
0
answers
31
views
deep learning performance stuck
Task
Multivariate time series forecast. I have one variable to forecast - and many explanatory variables. I want to train a model and then forecast the current period. This is for anomaly detection - ...
- 25
0
votes
0
answers
22
views
How to use cross validation to select/evaluate model with probability score as the output?
Initially I was evaluating my models using cross_val with out-of-pocket metrics such as precision, recall, f1 score, etc, or with my own metrics defined in ...
- 21
1
vote
0
answers
23
views
How to compare 55 models using AUC bootstrap confidence intervals?
I want to check if there is a difference in the confidence intervals of 55 models and select just one model. What should I do?
- 13
0
votes
0
answers
31
views
I have a poor understanding of nested cv and generalization
I'm not sure if I understand the purpose and generalization of 'nested cv' correctly.
I found information online that the purpose of nestd cv is to be able to correctly estimate generalization error. ...
- 13
1
vote
1
answer
55
views
How do I automate testing and comparison of the performance of models with different layer depths, layer types, and unit counts?
I am testing the effects of different layer counts/depths, unit counts, and layer types for natural language processing. I made a Kaggle notebook where I manually create different layers and then ...
- 153
0
votes
1
answer
182
views
Should I choose an ARIMA model (2,1,1) with a higher AIC value or an ARIMA model (6,1,8) with a lower AIC value?
I am trying to fit an ARIMA model to time series data. When I fit the model using auto.arima function in R, ...
- 103
0
votes
1
answer
49
views
Selecting optimal regression model using cross validation
I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
- 103
1
vote
1
answer
36
views
Aggregating decision criteria of different scales
Let's say I have a framework that performs a detection task on some dataset. In order to do so I use three different metrics (A, B, and C) as decision makers. A and B are probabilities, i.e., $ 0 \le ...
- 113
1
vote
1
answer
102
views
How to build a categorization system without a target variable?
The data
I have a large dataset containing execution logs from various tests conducted over several years. The logs can be noisy and often contain a plethora of messages detailing the ongoing ...
- 11
0
votes
1
answer
544
views
How to compare test vs train model performance
When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
- 755
0
votes
1
answer
201
views
How can I compare the accuracy of imputation models if there is already missing dataset in the file?
Let's say I have a dataset of 50,000 where about 2% were already missing from the beginning. From what I have learned, we need to use indicators to compare the imputation model with the ground truth ...
0
votes
1
answer
63
views
Sequence prediction in Parent - Child dataset
We have a large collection of documents (D), each accompanied by a set of metadata (M). Within this collection, some documents act as parent documents and have multiple child documents. Both parent ...
- 101
0
votes
1
answer
66
views
Is this the best method for comparing different approaches nd selecting the best model in machine learning?
My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these ...
0
votes
1
answer
294
views
Would time series input work in multiple polynomial regression model?
I am trying to do a side project to get a better understanding of the whole data science after completing my online course. Am now in an early stage of just laying out the project in general and was ...
- 3
0
votes
0
answers
55
views
Existence of a "three-point" machine learning model?
I may want to ask if there are studies that exist which utilize a "three-point machine learning model. What I mean by "three-point machine learning model is that it may use several ...
3
votes
2
answers
947
views
Group unstructured chat logs into conversations
I am new to ML/AI/NLP and am interested in tackling the following problem. I have a database of chat logs from a Discord server. The database contains the following labeled data: ...
- 31
0
votes
1
answer
70
views
Do I need to use always the same "Test" dataset to compare between different models?
I have two datasources A and B, and I want to check how several methods can affect the accuracy of my multi class models:
If I use cross-validation with validate dataset to obtain the best
hyper ...
0
votes
1
answer
293
views
How does GridSearchCV use Cross Validation to produce a Model's Score?
I understand Cross Validation in practice, but I'm not sure how SciKit-Learn's GridSearchCV uses it to produce an overall score/ metric for a model. For example, if ...
- 671
1
vote
1
answer
880
views
Good flowchart for which ML model to use given characteristics of the dataset?
There are so many ML models to choose from. Looking for a flow chart that someone may have created or come across that helps you decide which ML model(s) to use.
Here is some of the possible flow ...
- 111
0
votes
1
answer
270
views
Models that are good for long answer generation given context and question and what datasets would be the best for training?
Basically I am trying to create a context-needing question and long answer model and I was wondering what model would be best for such tasks, currently I am leaning towards T5, or GPT-NeoX-20B. ...
1
vote
1
answer
54
views
Can applying different models and seeing which one fits best be called brute forcing?
I have seen a tutorial which said that you have to try different models and see which fits best on your data.
Can this be considered brute force? I have searched this on google and the closest answer ...
1
vote
0
answers
103
views
Machine learning with mixed variables dataset (numerical, categorical and embeddings)
I'm working on a machine learning project where I'm trying to predict the revenue of a movie.
My dataset contains mixed data types. There are numerical features (rating, number of votes, release year,....
- 111
3
votes
2
answers
1k
views
Which is the final model from Nested Cross Validation: Accuracy or Frequency?
https://www.cnblogs.com/guo-xiang/p/8044624.html explains with a nice example the mechanics of Nested Cross Validation.
In the picture, the example shows how to use Nested CV for hyperparameter ...
- 541
0
votes
1
answer
158
views
Good models for predicting whether a customer would make a purchase given details like age, gender, ethnicity, salary, etc?
I have around 30,000 data points and for those data points I have some numerical fields like customer_age, ...
- 1
2
votes
0
answers
90
views
High overfitting, but test metrics are higher [duplicate]
Ml models must strike a balance between predictive power and generalization power. Therefore, I split the data into train/test and calculate metrics on both. Often I see instructions in someone else's ...
- 406
0
votes
1
answer
94
views
How to find significance for Gini coefficient changes?
I'm using the Gini coefficient to evaluate the performance of a model.
Making some changes (feature selection, hyperparameter tuning, etc.) I created variant models with different Gini coefficients.
...
- 604
0
votes
1
answer
23
views
Algorithm suggestion for correlated models
I'm looking for suggestions on how to proceed with predicting on separate but correlated models.
The example I will use is housing data. I have three inputs:
Latitude
Longitude
1-Google Street View ...
- 493
0
votes
1
answer
94
views
Kernel ridge regression (KRR), accuracy scale?
What does a good range for the accuracy score look like for the KRR model?
For example, RMSE produces a value between 0 and 1, where values closer to 0 represent better fitting models. What's the ...
- 103
1
vote
0
answers
19
views
What is the best way for me to classify this audio data?
I have a set of audio data. I would like to classify each audio file based on a half-second of data from a give time period. The audio data is given as counts as a function of time $s(t)$. Right now ...
- 11
0
votes
1
answer
202
views
additive or multiplicative model?
let's say if I have two scores $x_1^i$ and $x_2^i$ for each data point $i$, and I need to make a final score/loss function out of it.
Should I use a weighted sum $w_1 x_1^i + w_2 x_2^i$, or their ...
- 165
0
votes
0
answers
70
views
What are some "best practices" for discovering true patterns in data without relying on "scores" as measures of accuracy?
What are some "best practices" for discovering true patterns in data without knowing about them and without relying on "scores" as measures of accuracy?
In university I was always ...
- 416
0
votes
4
answers
63
views
Where to learn which ML task is most appropriate for a problem?
There is now tons of material available on how to do certain (most popular) ML tasks and what kind of output you can expect.
However I found that resources on how to select appropriate ML task/...
- 103
2
votes
1
answer
627
views
Nested Cross-Validation with Small dataset
I am currently working with a small dataset (only 175 samples, 45 features) and have been reading on the proper way to cross-validate my model. I had started with a basic cross-validation using a grid ...
- 21
2
votes
0
answers
25
views
Capping labels negatively impacts business metric
I have this deep neural network model with an integer label to predict. The label is heavily skewed so we cap the labels at some value (let's say 90 %ile).
Now when we build and run the model, it ...
- 131
0
votes
1
answer
219
views
0
votes
2
answers
36
views
do feature selection and model selection must share the same ratio between development set and test set?
As the title, after I performed a Feature Selection, is it mandatory to respect the same ratio (between development set and test set) in Model Selection?
1
vote
1
answer
29
views
How to choose Recursive Feature Elimination parameters
in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy).
But I can't figure out ...
0
votes
0
answers
14
views
How to check the validity of data? if There are repeated Y values against X data
I need to find the best-fitted curve for this data, I am not sure which model should I use? Can someone suggest me a model?
I also have doubts that this dataset is not valid. As there are multiple Y ...
- 101
0
votes
1
answer
90
views
Models: during training and during deployment
It's known that during the model training, we hold out the test-set. However, I actually find during deployment, that if to use a new model train on the entire dataset (train+test), actually yield ...
- 411
1
vote
1
answer
46
views
Feature selection with "overly important" features
I am very new to machine learning modeling, but I encountered a feature selection problem that I hope can get your insights on:
For example, I have A,B,C,D as my independent variables and y as my
...
- 13
2
votes
0
answers
141
views
Is data leakage giving me misleading results? Independent test set says no!
TLDR:
I evaluated a classification model using 10-fold CV with data leakage in the training and test folds. The results were great. I then solved the data leakage and the results were garbage. I then ...
- 21
0
votes
0
answers
131
views
Repeatability tests for machine learning models (in the sense of measurement system analysis)
For analyzing a machine learning model, we usually calculate the model performance metrics (such as accuracy...) and during validation step make sure that the model has not overfitted.
We can consider ...
- 1
0
votes
1
answer
45
views
Remove frame from background
I am having 400 images that look like the following:
I would like to remove the frame and only get the image in the middle:
I tried the MODNet model ...
- 187
1
vote
0
answers
71
views
What model to train to restore MNIST test dataset
I came across this problem, and not sure where to start. What model would work best for this problem and why?
Imagine the digits in the test set of the MNIST dataset
(http://yann.lecun.com/exdb/mnist/)...
1
vote
0
answers
25
views
How to go about predicting administrative fees?
We collect administrative fees from our customers based on many complex business rules albeit based on few variables. I have the history of fees colected through time (about 500 records for each ...
- 111