Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

Oversampling for regression for data grouped in clusters

I am dealing with a regression problem in which I want to predict the upcoming value of a time-dependent variable by using the previous values of other variables (not including the output variable ...
1
vote
1answer
7 views

How can I tell whether my Random-Forest model is overfitting?

I was trying to generate predictions for Iris species using the UCI Machine Learning Iris dataset. I used a RandomForestClassifier with GridSearchCV and calculated the mean absolute error. However, ...
0
votes
0answers
51 views

None of the known overfitting prevention techniques works for me, according to learning curves

I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE: I have an overfitting problem according to the learning curves below. In ...
1
vote
1answer
54 views

Interpretation for test score , training score and validation score in machine learning?

Interpretation for test score , training score and validation score ? what they actually tell us? What's an acceptable difference between cross test score , validation score and test score? If ...
0
votes
0answers
20 views

Self-driving AI model starts overfitting

I'm trying to make a self-driving AI that can drive around in GTA-San Andreas by following sentdex's videos on making a self-driving AI for GTA-V but my model always starts to overfit after 4 epochs ...
0
votes
1answer
26 views

Use all available data to build Logistic Regression model [duplicate]

Using K-Fold, I chose to use Logistic Regression for a project of mine. I made it learn on my X_train (80% of data), and tested it on my X_test, with good results. My question is : now that I need ...
2
votes
2answers
29 views

Over fitting and association with regularization

Heard and read lot about regularization helps in reducing over fitting. But I'm not sure how exactly regularization works in reducing over fitting issue and whats the maths behind it? Appreciate if ...
1
vote
0answers
19 views

How to deal with training set that overfits very easily

I have a dataset consisting of 72 one-hot encoded (thus binary) features and 2.5K training examples. With this I am trying to solve a 10-class classification problem. My main problem is that no ...
0
votes
1answer
40 views

Should bias updates be porportional to overfitting?

According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases ...
2
votes
0answers
39 views

Bias Formula in Machine Learning expanded using ground truth

Why is Bias calculated for $f(x)$? Shouldn't it be calculated for $Y$ (which is $f(x)$ + Noise $\epsilon$)? We are fitting our model to $Y$, So shouldn't we be calculating bias wrt to $Y$? Also, I ...
1
vote
1answer
29 views

Training on skewed dataset

I have a problem of multi class classification and I'm using a simple 2-Layer Bi-directional LSTM with keras. The model in a simple form: ...
2
votes
1answer
28 views

Overfitting with sklearn pipeline - reasons why?

So.... I've been playing around with this for FAR TOOO LONG now and I really need some advice. Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems ...
1
vote
1answer
22 views

Using standard deviation as a metric for model selection

I'm really getting stuck with overfitting and I'm trying all I can to reduce it. I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample ...
4
votes
1answer
152 views

ROC AUC score is much less than average cross validation score

Using Lending club Dataset to find the propability of default. I am using hyperopt library to fine tune hyper parameter for an XGBclassifier and trying to maximize the ROC AUC score. I am also using ...
0
votes
0answers
8 views

How to select checkpoint for model evaluation?

I have trained a deep convolutional neural network for image similarity classification. The network returns whether the images are the same or different. I trained the network for 20 epochs and save ...
3
votes
2answers
163 views

How to split data into 3 parts in Python - training(70%), validation(15%) and test(15%) and each part have similar target rate?

I'm working on a company project which I will need to do data partition into 3 parts - Train, Validation, and Test(holdout). Does anyone know how I can split the data into 3 parts above and each ...
4
votes
2answers
59 views

over-fitting with good enough test accuracy

Let's make things simple. Imagine an underdetermined linear system with $N$ samples and $p$ features $(N<p)$. Let's say I found one of the possible (among many) solutions of such systems and ...
1
vote
0answers
16 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed RFE + stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic ...
1
vote
0answers
37 views

How to interpret training results

I am working on an image similarity network. I have around 90,000 pairs of images contain an equal number of positive and negative samples. For learning the similarity between image pairs, I used the ...
30
votes
8answers
6k views

What would I prefer - an over-fitted model or a less accurate model?

Let's say we have two models trained. And let's say we are looking for good accuracy. The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. The second has an ...
2
votes
1answer
32 views

reduction of model accuracy while using PCA for a regression problem

I am trying to build a prection problem to predict the fare of flights. My data set has several catergorical variables like class,hour,day of week, day of month, month of year etc. I am using multiple ...
1
vote
2answers
157 views

What to choose: an overfit model with higher evaluation score or a non-overfit model with lower one?

For lack of a better term, overfit here means a higher discrepancy between train and validation score and non-overfit means a lower discrepancy. This "dilemma" just showed in neural network model I'...
2
votes
3answers
72 views

Is over-fitting a matter of features engineering or a matter of parameters set?

I am using sklearn package to make models. I tried randomly to set some paramater to a sklearn.ensemble.RandomForestClassifier ...
1
vote
3answers
73 views

Will the MAE of testing data always be higher than MAE of training data?

On the Kaggle Course Page the chart below shows that MAE of testing data is always higher than MAE of training data. Why is this the case? Is it only limited to DecisionTreeRegressor model? Or the ...
1
vote
1answer
90 views

Overfitting model

I'm training two ResNet models on an image dataset. The first one has been trained with random weights, while the other has been pre-trained on ImageNet before. The second model starts overfitting ...
1
vote
1answer
58 views

Training a Siamese Neural Network for object similarity assessment

I am training a Siamese neural network with pairs of similar and dissimilar objects. The features of the objects are binary data on whether they contain some properties or not (2048 features per ...
1
vote
0answers
20 views

Validation score during training and checkpoint is different in keras

I have a tabular data with about 1500 columns where every column except the 1st column is sparse. I am trying to train a Feedforward neural network (1 hidden layer with 32 neurons) for a binary ...
1
vote
1answer
43 views

Which combination of 3 hyperparameters to combat overfitting of a convolutional neural network?

I have a small dataset with which I want to train a CNN by using Data Augmentation. Since the CNN is overfitting due to the small data set, I would like to optimize some hyperparameters. However, ...
1
vote
2answers
98 views

k-fold cross validation with RNNs

is it a good idea to use k-fold cross-validation in the recurrent neural network (RNN) to alleviate overfitting? A potential solution could be ...
2
votes
1answer
38 views

Is not having overfitting more important than overall score (F1: 80-60-40% or 43-40-40)?

I've been trying to model a dataset using various classifiers. The response is highly imbalanced (binary) and I have both numerical and categorical variables, so I applied SMOTENC and Random ...
1
vote
1answer
31 views

Problem with overfitting for a CNN

I am doing image classification with a CNN and I am having trouble building a network that does not do overfitting. I have in my training set 2000 images of 4 classes, while in my test set I have 3038 ...
2
votes
2answers
179 views

How to build an overfitted network in order to increase performances

I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to ...
4
votes
1answer
43 views

Overfitting due to features correlating with training set generation rules

As background, I am using a Deep Neural Network built using Keras to classify inputs into 5 categories. The current structure of the network is: Input layer (~450 nodes) Dense layer (750 nodes) ...
1
vote
0answers
41 views

Conv Net Model is overfitting

So I made a convolution neural network to classify between different phonemes. My input datasets are a series of 0.4-second long spectrograms, the labels are each an individual phoneme that happens at ...
1
vote
1answer
23 views

On which step should use SMOTE technique for over sampling?

I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it. My preprocessing steps are: Missing values Removing Outliers Smoothing Data Should ...
0
votes
0answers
32 views

One part of my loss function overfits. How do I fix this?

I am working on an object detection problem where the final loss that is being optimized is the sum of an L2 loss (for the error in the predicted w, h values), and three binary cross entropy losses (...
0
votes
1answer
146 views

Random Forest Overfitting, issues with mtry=1?

I am constructing what is known as an 'Expected Goals' model for football. This metric measures shot quality and a probability is assigned to a shot to achieve this, i.e. the chance a shot will be ...
3
votes
2answers
81 views

Am I overfitting my random forest model (more information in description)?

First off, sorry if this a novice question! Relatively new to all this stuff. Posted this in Stack Overflow and someone sent me here! Hope it's the right place. Anyway, I'm working with 22 datasets ...
0
votes
0answers
31 views

Neural Network is overfitting when using bigger dataset

I'm tring to train a model using CNN (supervised) to solve a binary classification problem. I have pretty big dataset containing 2 800 000 samples, each having 100+ features. Because training with ...
2
votes
2answers
57 views

If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen ...
0
votes
1answer
67 views

Decision tree and random forest over fitting

I am working on a real state data set to predict the price of buying a house in Dubai based on area, no.of bedrooms, number of baths and the town which the house is in. All variables are numerical ...
0
votes
1answer
66 views

Correctly evaluate model with oversampling and cross-validation

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue. As I see, ...
2
votes
3answers
91 views

How to fix my high validation loss and inaccuracy

New to machine learning and tried to train my bird recognization model and found very high validation loss and inaccuracy. I'm using this dataset: http://www.vision.caltech.edu/visipedia/CUB-200-2011....
0
votes
0answers
23 views

Threshold for overfitted models

It's common knowledge in DS that overfitted models perform well on training data and poorly on test data. But how do you decide if a model is really overfitting? I have nowhere (books, online courses, ...
0
votes
0answers
11 views

Does a CNN fully memorize ground truth if it has more parameters than training pixels?

ResNet consists of 25M trainable parameters. If only 30% of 600 $512 \times 512$ images is annotated, there are $600 * 512 * 512 * ~0.3 = 47,185,920$ ground truth pixels. A parameter is a floating ...
1
vote
1answer
442 views

How to avoid overfitting in Reinforcement Learning

I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its ...
1
vote
1answer
39 views

Minimum Possible Test MSE

I have a little confusion. What follows is from Introduction to Statistical Learning (2013) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. My understanding of what is going ...
0
votes
0answers
12 views
0
votes
0answers
14 views

Overfitting collaborative filtering

So I want to know whether or not my models are overfitting or the difference between train and validation errors are decent. $L$: is the number of neighbors The first column is the train error ...
7
votes
3answers
155 views

Overfitting Question

Would you consider that overfitting?