Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

86 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
388 views

Knowing when a GAN is overfitting (sequence classification study)

I have sequences of long, sparse 1_D vectors (3000 digits, made of of 0s and 1s) that I am trying to classify. I have previously implemented a simple CNN to classify them with relative success (with ...
3
votes
3answers
125 views

Overfitted model produces similar AUC on test set, so which model do I go with?

I was trying to compare the effect of running GridSearchCV on a dataset which was oversampled prior and oversampled after the training folds are selected. The oversampling approach I used was random ...
2
votes
0answers
22 views

Building machine learning models whilst penalizing them for complexity

I come from a predictive modelling background, where it's common to use differential equations to model physical or chemical or biological processes. Commonly to avoid overfitting people use AIC and ...
2
votes
0answers
21 views

When to stop the final model training?

Let's say I'm participating in a Kaggle image recognition competition. Firstly, I create a train/validation split and find the good hyperparameters for my model. Here the stopping criterion is when ...
2
votes
1answer
51 views

How to analyze neural network quality in case of overfitting?

I have a Keras neural network that has images both as input and reference data. My network demonstrates overfitting (for example, train accuracy is about 80% but test accuracy is only up to 70%) due ...
2
votes
1answer
65 views

Why Continous Variable Buckets Overfitting model

I have a continuous (high cardinal discrete) variable 'numInteractionPoints' in my dataset during training model - I binned this feature in order to avoid overffing , first top bar chart is from ...
2
votes
0answers
99 views

Multilabel Classification - Overfitting?

My task is the following: To input drug combinations and output renal failure-related symptoms from the drug combinations. Both the drug combinations and renal-failure related symptoms are represented ...
2
votes
0answers
104 views

Bias Formula in Machine Learning expanded using ground truth

Why is Bias calculated for $f(x)$? Shouldn't it be calculated for $Y$ (which is $f(x)$ + Noise $\epsilon$)? We are fitting our model to $Y$, So shouldn't we be calculating bias wrt to $Y$? Also, I ...
2
votes
0answers
426 views

Overfitting in K-means

How do you test your results for overfitting in a k-means run? Some people have said use a training set. I have about 1500 records and about 20 fields.
2
votes
0answers
244 views

How to improve a model with a high cross validation score yet with low accuracy on unseen data?

I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows: Record raw data from the subjects For each subject, compute the feature from the raw data ...
2
votes
0answers
84 views

Is my model overfitting when I add new features?

I'm working on simple 2-class classification problem. Nearly all features we have used (except one) are about the same for both classes: A random forest classifier confirms that one feature has an "...
2
votes
2answers
3k views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for classifiers ...
2
votes
0answers
75 views

Overfitting and COLT/Statistical Learning Theory

The aspect of over-fitting is typically viewed from the perspective of both- accuracy and model complexity. To mitigate over-fitting, we usually have the practical approach of having k-fold ...
2
votes
0answers
56 views

information leakage when using empirical Bayesian to generate a predictor

Consider the following problem: I want to predict the next bat of a set of baseball player. I have a training data set, where it contains the historical bat records (0-1 encoded, which is our target ...
1
vote
0answers
13 views

Rule of Thumb for number of observations required to train a model with n independent variables?

I am aware adding more features to a model leads to overfitting of a model. Is there a rule of thumb for minimum number of rows required to build a model with n features in order to build a ...
1
vote
1answer
24 views

Will repeatedly fine-tuning on new data cause overfitting?

I have a binary classification model which I have trained on a training set. On the validation set its accuracy is ~85%. I set up early stopping which ended training when validation loss increased. ...
1
vote
0answers
22 views

Random Forest overfitting with `n_estimators=1`, `max_depth=1 and `max_features =1`

I am trying to stop my RF from overfitting. I am using time series data with 1 day time lag, to predict the current price. I am using this function to shift my independent features back 1 day: ...
1
vote
0answers
62 views

CNN + LSTM model for images performs poorly on validation data set

My training and loss curves look like below and yes, similar graphs have received comments like "Classic overfitting" and I get it. My model looks like below, ...
1
vote
0answers
65 views

XGBoost regressor hyperparameter tuning with hyperopt leads to overfit

Using hyperopt to hyperparameter tuning on XGBoost regressor, I am receiving overfiting on the train set. There is any suggestion how to solve it ? I have used cross validation with ...
1
vote
0answers
24 views

Understanding model's learning curves

I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset. Below are some of the important configuration values: Batch size: 6 Optimizer: Adam Learning rate: High of 1e-4 and Low ...
1
vote
2answers
38 views

dataset split for image classification

I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a ...
1
vote
0answers
19 views

Normal distribution of errors

I'm trying to project lifetime of customers in my company, based on various parameters I've reached a 64% correlation so far, between the valid and prediction data I'm using light GBM regressor I did ...
1
vote
0answers
49 views

Data\Feature Leakage - feature too close to target?

In General: The target itself built from very correlated features, because there is no ground truth - only rule based one. I have a problem in the following method: Output: binary. built from ...
1
vote
1answer
60 views

Training Object Detection model on just 10 images

I am trying to train an object detection model using Mask-RCNN with Resnet50 as backbone. I am using the pre-trained models from PyTorch's Torchvision library. I have only 10 images that I can use to ...
1
vote
0answers
13 views

model tuning by using loss curves

I have been practicing with the following dataset: http://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength for building a prediction model based on a MLP, but I have some doubts if the ...
1
vote
0answers
45 views

What is the impact of adding a layer in neural networks?

I was playing with hyper-parameters on https://playground.tensorflow.org/ using spiral dataset (classification). So , first I trained a network with 2 hidden layers and the final test and train loss ...
1
vote
0answers
65 views

Autoencoder fails to reconstruct

I'm trying to use an autoencoder to reduce dimensionality of my features. My features are of dimension 2048. I tried to train an autoencoder to reduce the dimensionality to 50. I'm using a single ...
1
vote
0answers
21 views

Escaping from overfitting hell: introducing regularization vs increasing training data

I am trying to identify noisy intervals in geomagnetic data using logistic regression, working with scikit-learn. Here is a typical spectrum of the data that I am working with: In this example, the ...
1
vote
0answers
182 views

Data augmentation for recommendation systems

I have a user-item matrix that I use to train a denoising autoencoder to predict the top-k items to recommend to the different users. The idea is to corrupt the matrix by erasing a percentage ...
1
vote
1answer
834 views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
1
vote
0answers
77 views

Removing duplicate records before training

I am currently working on a project classifying text into classes. The specific problem is classifying job titles into various industry codes. For example "McDonalds Employee" might get classified to ...
1
vote
0answers
15 views

Oversampling for regression for data grouped in clusters

I am dealing with a regression problem in which I want to predict the upcoming value of a time-dependent variable by using the previous values of other variables (not including the output variable ...
1
vote
0answers
63 views

None of the known overfitting prevention techniques works for me, according to learning curves

I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE: I have an overfitting problem according to the learning curves below. In ...
1
vote
0answers
38 views

Self-driving AI model starts overfitting

I'm trying to make a self-driving AI that can drive around in GTA-San Andreas by following sentdex's videos on making a self-driving AI for GTA-V but my model always starts to overfit after 4 epochs ...
1
vote
0answers
26 views

How to deal with training set that overfits very easily

I have a dataset consisting of 72 one-hot encoded (thus binary) features and 2.5K training examples. With this I am trying to solve a 10-class classification problem. My main problem is that no ...
1
vote
0answers
46 views

How to interpret training results

I am working on an image similarity network. I have around 90,000 pairs of images contain an equal number of positive and negative samples. For learning the similarity between image pairs, I used the ...
1
vote
0answers
44 views

Validation score during training and checkpoint is different in keras

I have a tabular data with about 1500 columns where every column except the 1st column is sparse. I am trying to train a Feedforward neural network (1 hidden layer with 32 neurons) for a binary ...
1
vote
0answers
75 views

Conv Net Model is overfitting

So I made a convolution neural network to classify between different phonemes. My input datasets are a series of 0.4-second long spectrograms, the labels are each an individual phoneme that happens at ...
1
vote
0answers
209 views

High Variance on CNN

I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best ...
1
vote
0answers
289 views

Remedies to CNN-LSTM overfitting on relatively small image dataset

Notes Using a pretrained model, trying data augmentation (not possible knowing nature of images, lowering number of parameters in the network, all didn't help) Context I have a sequence of images. ...
1
vote
0answers
60 views

Network either overfits or underfits, but never generalizes - what to do?

I have a simple network with 1st level an LSTM, dropout, fully-connected and softmax layers; loss is cross-entropy (four classes, well balanced). Sequence length to LSTM is 172 samples, data is z-...
1
vote
0answers
64 views

Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome

I have a binomial outcome that I am trying to predict using a gbm in h2o. I have set quite a low min_rows value for each node and it appears to be overfitting. See plots below. When I group the ...
0
votes
0answers
15 views

Data augmentation within epochs vs across epochs

Usually in deep learning data augmentation is applied by creating a new augmented version of each training sample for each epoch. Therefore the amount of training samples for each epoch stays the same ...
0
votes
0answers
10 views

Model variance increases during training

I trained a regression model with lightgbm and the learning curve doesn't look good: The model variance increases during training, which shows a kind of overfitting. Now, I tried many ways to fix ...
0
votes
0answers
19 views

Possible fixes for an overfitting random forest regressor?

I'm fitting a random forest regressor on my dataset (do note its not a classifer but a regressor since the target is a continuous variable) through a grid search cross-validation in sklearn. The ...
0
votes
0answers
18 views

Can there be scenarios where an overfitted model in machine learning cannot be generalized?

Is it always possible to generalize an overfitted model? I know there are ways to handle overfitting, but can there be scenarios where overfitting cannot be handled in machine learning?
0
votes
1answer
45 views

Learning curves

I am working on a multiclass classification problem. I want to know weather my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts. 1.) Is the ordering of ...
0
votes
0answers
15 views

Systematically finding a CNN architecture?

I am trying to train a classifier from 25k images and 7k classes. Seems like my model overfits just after 3 epochs. I have tried to reduce the model complexity and increase the weight decay but still, ...
0
votes
0answers
38 views

Is my CNN model overfitting or underfitting?

I would like to be sure of whether the model is overfitting or undercutting. Being new to this, is there any specific point to identify when to stop the training process. Any help in this regard would ...
0
votes
0answers
19 views

How to improve SGD classifier performance?

I am having some issues with SGD to run with a sample of approx 5000 obs for a classification problem with imbalance. I am doing ...