Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

96 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
401 views

Knowing when a GAN is overfitting (sequence classification study)

I have sequences of long, sparse 1_D vectors (3000 digits, made of of 0s and 1s) that I am trying to classify. I have previously implemented a simple CNN to classify them with relative success (with ...
3
votes
3answers
129 views

Overfitted model produces similar AUC on test set, so which model do I go with?

I was trying to compare the effect of running GridSearchCV on a dataset which was oversampled prior and oversampled after the training folds are selected. The oversampling approach I used was random ...
2
votes
0answers
22 views

Building machine learning models whilst penalizing them for complexity

I come from a predictive modelling background, where it's common to use differential equations to model physical or chemical or biological processes. Commonly to avoid overfitting people use AIC and ...
2
votes
0answers
22 views

When to stop the final model training?

Let's say I'm participating in a Kaggle image recognition competition. Firstly, I create a train/validation split and find the good hyperparameters for my model. Here the stopping criterion is when ...
2
votes
1answer
59 views

How to analyze neural network quality in case of overfitting?

I have a Keras neural network that has images both as input and reference data. My network demonstrates overfitting (for example, train accuracy is about 80% but test accuracy is only up to 70%) due ...
2
votes
1answer
75 views

Why Continous Variable Buckets Overfitting model

I have a continuous (high cardinal discrete) variable 'numInteractionPoints' in my dataset during training model - I binned this feature in order to avoid overffing , first top bar chart is from ...
2
votes
0answers
112 views

Multilabel Classification - Overfitting?

My task is the following: To input drug combinations and output renal failure-related symptoms from the drug combinations. Both the drug combinations and renal-failure related symptoms are represented ...
2
votes
0answers
108 views

Bias Formula in Machine Learning expanded using ground truth

Why is Bias calculated for $f(x)$? Shouldn't it be calculated for $Y$ (which is $f(x)$ + Noise $\epsilon$)? We are fitting our model to $Y$, So shouldn't we be calculating bias wrt to $Y$? Also, I ...
2
votes
1answer
488 views

Overfitting in K-means

How do you test your results for overfitting in a k-means run? Some people have said use a training set. I have about 1500 records and about 20 fields.
2
votes
0answers
251 views

How to improve a model with a high cross validation score yet with low accuracy on unseen data?

I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows: Record raw data from the subjects For each subject, compute the feature from the raw data ...
2
votes
0answers
86 views

Is my model overfitting when I add new features?

I'm working on simple 2-class classification problem. Nearly all features we have used (except one) are about the same for both classes: A random forest classifier confirms that one feature has an "...
2
votes
2answers
3k views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for classifiers ...
2
votes
0answers
76 views

Overfitting and COLT/Statistical Learning Theory

The aspect of over-fitting is typically viewed from the perspective of both- accuracy and model complexity. To mitigate over-fitting, we usually have the practical approach of having k-fold ...
2
votes
0answers
56 views

information leakage when using empirical Bayesian to generate a predictor

Consider the following problem: I want to predict the next bat of a set of baseball player. I have a training data set, where it contains the historical bat records (0-1 encoded, which is our target ...
1
vote
1answer
17 views

Is there any better approach then K folds and nested K folds?

I am trying to understand what problem is K-folds solving. It does not seem to be solving data leakage at all, as we are still testing on test data and then taking an average of all test folds and ...
1
vote
0answers
11 views

Overfitting reason in 2-stage model

I'm trying to build an entity matching model. There are 2 kinds of features - binary (0/1) and text features. Initially I made a deep learning model that uses character level embeddings of some of the ...
1
vote
1answer
50 views

How do I know if this model is overfitting?

This is my example R script for a decision tree: ...
1
vote
0answers
34 views

Automate detection of overfitting models based on autoML libraries

I'm trying to use machine learning to impute missing data in series using some auto-ML libraries in python (so far : dabl, FLAML, auto-sklearn and AutoKeras). I know the way to detect overfitting in a ...
1
vote
1answer
16 views

Can Online DQN model overfit?

I am new in the area of RL and currently trying to train an online DQN model. Can an online model overfit since its always learning? and how can I tell if that happens?
1
vote
0answers
13 views

Rule of Thumb for number of observations required to train a model with n independent variables?

I am aware adding more features to a model leads to overfitting of a model. Is there a rule of thumb for minimum number of rows required to build a model with n features in order to build a ...
1
vote
1answer
34 views

Will repeatedly fine-tuning on new data cause overfitting?

I have a binary classification model which I have trained on a training set. On the validation set its accuracy is ~85%. I set up early stopping which ended training when validation loss increased. ...
1
vote
0answers
27 views

Random Forest overfitting with `n_estimators=1`, `max_depth=1 and `max_features =1`

I am trying to stop my RF from overfitting. I am using time series data with 1 day time lag, to predict the current price. I am using this function to shift my independent features back 1 day: ...
1
vote
0answers
132 views

CNN + LSTM model for images performs poorly on validation data set

My training and loss curves look like below and yes, similar graphs have received comments like "Classic overfitting" and I get it. My model looks like below, ...
1
vote
0answers
89 views

XGBoost regressor hyperparameter tuning with hyperopt leads to overfit

Using hyperopt to hyperparameter tuning on XGBoost regressor, I am receiving overfiting on the train set. There is any suggestion how to solve it ? I have used cross validation with ...
1
vote
0answers
24 views

Understanding model's learning curves

I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset. Below are some of the important configuration values: Batch size: 6 Optimizer: Adam Learning rate: High of 1e-4 and Low ...
1
vote
2answers
61 views

dataset split for image classification

I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a ...
1
vote
0answers
19 views

Normal distribution of errors

I'm trying to project lifetime of customers in my company, based on various parameters I've reached a 64% correlation so far, between the valid and prediction data I'm using light GBM regressor I did ...
1
vote
0answers
51 views

Data\Feature Leakage - feature too close to target?

In General: The target itself built from very correlated features, because there is no ground truth - only rule based one. I have a problem in the following method: Output: binary. built from ...
1
vote
2answers
71 views

Training Object Detection model on just 10 images

I am trying to train an object detection model using Mask-RCNN with Resnet50 as backbone. I am using the pre-trained models from PyTorch's Torchvision library. I have only 10 images that I can use to ...
1
vote
0answers
18 views

model tuning by using loss curves

I have been practicing with the following dataset: http://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength for building a prediction model based on a MLP, but I have some doubts if the ...
1
vote
0answers
47 views

What is the impact of adding a layer in neural networks?

I was playing with hyper-parameters on https://playground.tensorflow.org/ using spiral dataset (classification). So , first I trained a network with 2 hidden layers and the final test and train loss ...
1
vote
1answer
56 views

Regularization hyperparam tuning during training

I have an idea for a regularization-hyperparam selection method, which I haven't encountered before and can't find on Google, but I'm sure someone has already tried it and I'm wondering what are the ...
1
vote
0answers
85 views

Autoencoder fails to reconstruct

I'm trying to use an autoencoder to reduce dimensionality of my features. My features are of dimension 2048. I tried to train an autoencoder to reduce the dimensionality to 50. I'm using a single ...
1
vote
0answers
21 views

Escaping from overfitting hell: introducing regularization vs increasing training data

I am trying to identify noisy intervals in geomagnetic data using logistic regression, working with scikit-learn. Here is a typical spectrum of the data that I am working with: In this example, the ...
1
vote
0answers
215 views

Data augmentation for recommendation systems

I have a user-item matrix that I use to train a denoising autoencoder to predict the top-k items to recommend to the different users. The idea is to corrupt the matrix by erasing a percentage ...
1
vote
1answer
964 views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
1
vote
0answers
90 views

Removing duplicate records before training

I am currently working on a project classifying text into classes. The specific problem is classifying job titles into various industry codes. For example "McDonalds Employee" might get classified to ...
1
vote
0answers
15 views

Oversampling for regression for data grouped in clusters

I am dealing with a regression problem in which I want to predict the upcoming value of a time-dependent variable by using the previous values of other variables (not including the output variable ...
1
vote
0answers
66 views

None of the known overfitting prevention techniques works for me, according to learning curves

I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE: I have an overfitting problem according to the learning curves below. In ...
1
vote
0answers
39 views

Self-driving AI model starts overfitting

I'm trying to make a self-driving AI that can drive around in GTA-San Andreas by following sentdex's videos on making a self-driving AI for GTA-V but my model always starts to overfit after 4 epochs ...
1
vote
0answers
26 views

How to deal with training set that overfits very easily

I have a dataset consisting of 72 one-hot encoded (thus binary) features and 2.5K training examples. With this I am trying to solve a 10-class classification problem. My main problem is that no ...
1
vote
0answers
47 views

How to interpret training results

I am working on an image similarity network. I have around 90,000 pairs of images contain an equal number of positive and negative samples. For learning the similarity between image pairs, I used the ...
1
vote
0answers
50 views

Validation score during training and checkpoint is different in keras

I have a tabular data with about 1500 columns where every column except the 1st column is sparse. I am trying to train a Feedforward neural network (1 hidden layer with 32 neurons) for a binary ...
1
vote
0answers
78 views

Conv Net Model is overfitting

So I made a convolution neural network to classify between different phonemes. My input datasets are a series of 0.4-second long spectrograms, the labels are each an individual phoneme that happens at ...
1
vote
0answers
209 views

High Variance on CNN

I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best ...
1
vote
0answers
296 views

Remedies to CNN-LSTM overfitting on relatively small image dataset

Notes Using a pretrained model, trying data augmentation (not possible knowing nature of images, lowering number of parameters in the network, all didn't help) Context I have a sequence of images. ...
1
vote
0answers
60 views

Network either overfits or underfits, but never generalizes - what to do?

I have a simple network with 1st level an LSTM, dropout, fully-connected and softmax layers; loss is cross-entropy (four classes, well balanced). Sequence length to LSTM is 172 samples, data is z-...
1
vote
0answers
64 views

Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome

I have a binomial outcome that I am trying to predict using a gbm in h2o. I have set quite a low min_rows value for each node and it appears to be overfitting. See plots below. When I group the ...
0
votes
0answers
5 views

Correct implementation and usage of NestedCV and train and test data?

I have some confusion regarding NestedCV and the correct usage/split of the data in order to reduce/prevent overfitting/bias/leakage. Below I have implemented my understanding:- ...
0
votes
0answers
10 views

How can I reduce overfitting in CNN model for image classification, even after data augmentation?

its my first time posting here. I'm trying to build a CNN model that identifies fruits from a dataset of apples, bananas, mixed fruits, and oranges. So far, one of the things I have done to prevent ...