Questions tagged [model-selection]

Model selection is the process of comparing several models and their respective results to choose the model is best according to some evaluation metric.

Filter by
Sorted by
Tagged with
1 vote
1 answer
31 views

Aggregating decision criteria of different scales

Let's say I have a framework that performs a detection task on some dataset. In order to do so I use three different metrics (A, B, and C) as decision makers. A and B are probabilities, i.e., $ 0 \le ...
Minuano's user avatar
  • 113
1 vote
1 answer
40 views

How to build a categorization system without a target variable?

The data I have a large dataset containing execution logs from various tests conducted over several years. The logs can be noisy and often contain a plethora of messages detailing the ongoing ...
Mr Kartofel's user avatar
0 votes
1 answer
33 views

How to compare test vs train model performance

When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the ...
thereandhere1's user avatar
0 votes
0 answers
15 views

How do you figure out how much of a user rating to attribute to each step?

I've got a multi-step pipeline that produces output which is then (sometimes) rated by users. Something like this: Run sentiment analysis on input Run intent analysis on input Choose gating weights ...
TheEnvironmentalist's user avatar
0 votes
0 answers
9 views

Validation error less than training error

entire project link: Github Repository In a classification task using Neural Network, I computed the fraction of misclassification as an error. And I am getting a validation error less than the ...
Dipen Pandit's user avatar
0 votes
1 answer
102 views

How can I compare the accuracy of imputation models if there is already missing dataset in the file?

Let's say I have a dataset of 50,000 where about 2% were already missing from the beginning. From what I have learned, we need to use indicators to compare the imputation model with the ground truth ...
Amisha Dhimal's user avatar
0 votes
0 answers
46 views

Time Series: Why is my model giving predictions of lower variance?

I have fit an MA(1) model (based on examining ACF and PACF) to a de-trended time series, which is stationary. I get the following result. The fitted values are closer to mean but with lower variance. ...
Athos's user avatar
  • 1
0 votes
1 answer
33 views

Sequence prediction in Parent - Child dataset

We have a large collection of documents (D), each accompanied by a set of metadata (M). Within this collection, some documents act as parent documents and have multiple child documents. Both parent ...
6nagi9's user avatar
  • 101
0 votes
1 answer
39 views

Is this the best method for comparing different approaches nd selecting the best model in machine learning?

My objective is to experiment with various approaches for different algorithms, identify the best approach for each algorithm, and subsequently determine the best overall algorithm from among these ...
Salah Amani's user avatar
0 votes
1 answer
59 views

Would time series input work in multiple polynomial regression model?

I am trying to do a side project to get a better understanding of the whole data science after completing my online course. Am now in an early stage of just laying out the project in general and was ...
ENGSSG's user avatar
  • 3
0 votes
0 answers
15 views

What model should I use to get the best three predictions for one observation?

Sorry, I'm a noobie, the problem is: I have a dataset with cars available for sale in different locations, so I have attributes like year built, miles, brand, model, # of doors, # of seats, style, etc....
Cris Tina's user avatar
0 votes
0 answers
42 views

Existence of a "three-point" machine learning model?

I may want to ask if there are studies that exist which utilize a "three-point machine learning model. What I mean by "three-point machine learning model is that it may use several ...
Ralph Henry's user avatar
2 votes
2 answers
339 views

Group unstructured chat logs into conversations

I am new to ML/AI/NLP and am interested in tackling the following problem. I have a database of chat logs from a Discord server. The database contains the following labeled data: ...
imaginative's user avatar
0 votes
0 answers
19 views

Which CNN model to use for the classification(20 classes) of gemstones (diamonds, sapphire, ruby etc) based on digital photo images and huge data set?

Im trying to build CNN Model for the classification of precious stones (like diamonds, sapphire, ruby) based on digital images. So I have data set of labeled 150,000 gemstone certifications and the ...
Bob9710's user avatar
  • 101
0 votes
0 answers
220 views

How to make the model fitting to the inverse-square law in Python?

I am studying basics of physics. I would like to learn to make a graphical solution to the following problem: In a class room a lighting meter was used to measure the illuminance E at the distance r. ...
self-studying's user avatar
0 votes
1 answer
42 views

Do I need to use always the same "Test" dataset to compare between different models?

I have two datasources A and B, and I want to check how several methods can affect the accuracy of my multi class models: If I use cross-validation with validate dataset to obtain the best hyper ...
Just_4n0th3r_Pr0gr4mm3r's user avatar
0 votes
0 answers
57 views

LSTM input_shape returns value error

My time series dataset dimension are as follows: print(X_train.shape) = (1766, 4) i.e. 1,766 time steps and 4 features ...
user144958's user avatar
0 votes
1 answer
186 views

How does GridSearchCV use Cross Validation to produce a Model's Score?

I understand Cross Validation in practice, but I'm not sure how SciKit-Learn's GridSearchCV uses it to produce an overall score/ metric for a model. For example, if ...
Connor's user avatar
  • 617
1 vote
1 answer
239 views

Good flowchart for which ML model to use given characteristics of the dataset?

There are so many ML models to choose from. Looking for a flow chart that someone may have created or come across that helps you decide which ML model(s) to use. Here is some of the possible flow ...
Katsu's user avatar
  • 111
0 votes
1 answer
62 views

Models that are good for long answer generation given context and question and what datasets would be the best for training?

Basically I am trying to create a context-needing question and long answer model and I was wondering what model would be best for such tasks, currently I am leaning towards T5, or GPT-NeoX-20B. ...
Thamognya Kodi's user avatar
0 votes
0 answers
24 views

Image recognition model with CNN for face gestures is really bad

I have a dataset that contains facial expressions and their label, and I am trying to make a classification model for it. Unfortunatly, I can't manage to create a good model with CNN, as the highest ...
Arad's user avatar
  • 1
1 vote
1 answer
52 views

Can applying different models and seeing which one fits best be called brute forcing?

I have seen a tutorial which said that you have to try different models and see which fits best on your data. Can this be considered brute force? I have searched this on google and the closest answer ...
Vishwanath Upadhyay's user avatar
1 vote
0 answers
67 views

Machine learning with mixed variables dataset (numerical, categorical and embeddings)

I'm working on a machine learning project where I'm trying to predict the revenue of a movie. My dataset contains mixed data types. There are numerical features (rating, number of votes, release year,....
Mathieu Rousseau's user avatar
2 votes
2 answers
332 views

Which is the final model from Nested Cross Validation: Accuracy or Frequency?

https://www.cnblogs.com/guo-xiang/p/8044624.html explains with a nice example the mechanics of Nested Cross Validation. In the picture, the example shows how to use Nested CV for hyperparameter ...
Sm1's user avatar
  • 521
0 votes
0 answers
41 views

Am I calculating LogAUC / pAUC correctly?

Hope you are well. I was wondering how one would calculate logAUC? I have an implementation but I don't think it's correct. I'm trying to recreate the metric in this manuscript. See figure 2. Any help ...
James Arthur's user avatar
0 votes
0 answers
22 views

Predicting number of women who will give birth on a given day?

I have a set of data with information about women and their expected delivery dates for childbirth. I have more columns in the table but for simplicity let's just focus on the below and assume that my ...
Tom's user avatar
  • 1
0 votes
1 answer
36 views

Good models for predicting whether a customer would make a purchase given details like age, gender, ethnicity, salary, etc?

I have around 30,000 data points and for those data points I have some numerical fields like customer_age, ...
Tom's user avatar
  • 1
2 votes
0 answers
56 views

High overfitting, but test metrics are higher

Ml models must strike a balance between predictive power and generalization power. Therefore, I split the data into train/test and calculate metrics on both. Often I see instructions in someone else's ...
Andrew's user avatar
  • 396
0 votes
1 answer
43 views

How to find significance for Gini coefficient changes?

I'm using the Gini coefficient to evaluate the performance of a model. Making some changes (feature selection, hyperparameter tuning, etc.) I created variant models with different Gini coefficients. ...
Lafayette's user avatar
  • 604
0 votes
1 answer
23 views

Algorithm suggestion for correlated models

I'm looking for suggestions on how to proceed with predicting on separate but correlated models. The example I will use is housing data. I have three inputs: Latitude Longitude 1-Google Street View ...
nfmcclure's user avatar
  • 493
0 votes
1 answer
58 views

Kernel ridge regression (KRR), accuracy scale?

What does a good range for the accuracy score look like for the KRR model? For example, RMSE produces a value between 0 and 1, where values closer to 0 represent better fitting models. What's the ...
noor h's user avatar
  • 103
1 vote
0 answers
18 views

What is the best way for me to classify this audio data?

I have a set of audio data. I would like to classify each audio file based on a half-second of data from a give time period. The audio data is given as counts as a function of time $s(t)$. Right now ...
user3517167's user avatar
0 votes
1 answer
148 views

additive or multiplicative model?

let's say if I have two scores $x_1^i$ and $x_2^i$ for each data point $i$, and I need to make a final score/loss function out of it. Should I use a weighted sum $w_1 x_1^i + w_2 x_2^i$, or their ...
user900476's user avatar
0 votes
0 answers
53 views

What are some "best practices" for discovering true patterns in data without relying on "scores" as measures of accuracy?

What are some "best practices" for discovering true patterns in data without knowing about them and without relying on "scores" as measures of accuracy? In university I was always ...
mavavilj's user avatar
  • 416
0 votes
4 answers
53 views

Where to learn which ML task is most appropriate for a problem?

There is now tons of material available on how to do certain (most popular) ML tasks and what kind of output you can expect. However I found that resources on how to select appropriate ML task/...
Boppity Bop's user avatar
2 votes
1 answer
283 views

Nested Cross-Validation with Small dataset

I am currently working with a small dataset (only 175 samples, 45 features) and have been reading on the proper way to cross-validate my model. I had started with a basic cross-validation using a grid ...
Fritos121's user avatar
2 votes
0 answers
20 views

Capping labels negatively impacts business metric

I have this deep neural network model with an integer label to predict. The label is heavily skewed so we cap the labels at some value (let's say 90 %ile). Now when we build and run the model, it ...
aghd's user avatar
  • 131
0 votes
1 answer
187 views

How can I export the best classifier from my code to a model for real future usage?

...
Tempu's user avatar
  • 1
0 votes
2 answers
26 views

do feature selection and model selection must share the same ratio between development set and test set?

As the title, after I performed a Feature Selection, is it mandatory to respect the same ratio (between development set and test set) in Model Selection?
Giorgio Martinez's user avatar
1 vote
1 answer
22 views

How to choose Recursive Feature Elimination parameters

in my project I have >900 features and I thought to use Recursive Feature Elimination algorithm to reduce the dimensionality of my problem (in order to improve the accuracy). But I can't figure out ...
Giorgio Martinez's user avatar
0 votes
0 answers
13 views

How to check the validity of data? if There are repeated Y values against X data

I need to find the best-fitted curve for this data, I am not sure which model should I use? Can someone suggest me a model? I also have doubts that this dataset is not valid. As there are multiple Y ...
Adnan Ali's user avatar
  • 101
0 votes
1 answer
70 views

Models: during training and during deployment

It's known that during the model training, we hold out the test-set. However, I actually find during deployment, that if to use a new model train on the entire dataset (train+test), actually yield ...
Student's user avatar
  • 411
1 vote
1 answer
32 views

Feature selection with "overly important" features

I am very new to machine learning modeling, but I encountered a feature selection problem that I hope can get your insights on: For example, I have A,B,C,D as my independent variables and y as my ...
shadowrain's user avatar
2 votes
0 answers
119 views

Is data leakage giving me misleading results? Independent test set says no!

TLDR: I evaluated a classification model using 10-fold CV with data leakage in the training and test folds. The results were great. I then solved the data leakage and the results were garbage. I then ...
PeMADS's user avatar
  • 21
0 votes
0 answers
106 views

Repeatability tests for machine learning models (in the sense of measurement system analysis)

For analyzing a machine learning model, we usually calculate the model performance metrics (such as accuracy...) and during validation step make sure that the model has not overfitted. We can consider ...
abaghb's user avatar
  • 1
0 votes
1 answer
41 views

Remove frame from background

I am having 400 images that look like the following: I would like to remove the frame and only get the image in the middle: I tried the MODNet model ...
Carol.Kar's user avatar
  • 187
1 vote
0 answers
70 views

What model to train to restore MNIST test dataset

I came across this problem, and not sure where to start. What model would work best for this problem and why? Imagine the digits in the test set of the MNIST dataset (http://yann.lecun.com/exdb/mnist/)...
Sharhad Bashar's user avatar
1 vote
0 answers
25 views

How to go about predicting administrative fees?

We collect administrative fees from our customers based on many complex business rules albeit based on few variables. I have the history of fees colected through time (about 500 records for each ...
filippo's user avatar
  • 111
4 votes
1 answer
441 views

How can I choose the best machine learning algorithms from all kinds of algorithms?

When I want to find a model for my data set, I find that there are lots of algorithms that I can use. I know how to minimize selection choices by separating supervised and unsupervised algorithms and ...
Esi's user avatar
  • 53
0 votes
1 answer
28 views

how to choose the best machine learning algorithms from all kinds of algorithms? [duplicate]

guys, I am a beginner at data science and I’ve been learning machine learning for a while with some courses online without any help of a teacher and after I’ve got to work with some real projects on ...
Esi's user avatar
  • 53

1
2 3 4 5 6