1
$\begingroup$

I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i decided to apply oversampling (SMOTE). Here is what i did:

  • Using the entire dataset (original 192 samples):
  • Create synthetic samples for each class using SMOTE, so i get a total of 500 samples per class (2000 in total)

I have a big suspicion about this procedure because when i apply SMOTE i get very high accuracy rates (even 100% in some cases) with the simplest models such as a 15-neuron MLP. So, i have some questions to assert the correctness of my experiments:

  1. Is it ok to oversample the entire dataset or should i apply the SMOTE only in train data (keep in mind that this will leave few samples for test)?

  2. SMOTE is originally used to deal with imbalanced datasets, creating samples for the classes with few sample. Is it ok to use it for generating samples for ALL classes in order to enlarge the entire dataset?

$\endgroup$
1
1
$\begingroup$

First of all - have you tried to do modeling without oversampling? If you dont have imbalanced classes, why do you need oversampling? You dont always need tons of data to get a good model as a baseline.

Regarding should i apply the SMOTE only in train data - always leave some test data out, dont apply any transformation on the whole dataset (even normalization or missing values imputing). If you have some preprocessing - do it on train data set, train model. Then apply the preprocessing pipeline on test data and predict on it.

simplest models such as a 15-neuron MLP. - it doesnt sound as simplest solution to me. have you tried any other approaches? also, have you checked confusion matrix to see the classification error on test data for each of the class?

$\endgroup$
5
  • $\begingroup$ Yes, i tried modeling without oversampling along with 5 different models and they perform poorly. Regarding neural networks, i checked that they stop learning with a high loss (what made me believe that it should be good if i had more samples). $\endgroup$ Apr 7 at 19:40
  • $\begingroup$ have you tested your oversampling approach on a test set? (not used during oversampling) $\endgroup$ Apr 7 at 19:44
  • $\begingroup$ Do you mean split the dataset in train/test and just oversample train data? No, but in this scenario i would have at most 96 test samples (considering a split of 50%). I will check it out and give you some feedback. $\endgroup$ Apr 7 at 19:47
  • $\begingroup$ one way or another, if you want to have a proper model you will need to test it on hold-out set, either with train-test split, or using cross-validation. otherwise, you will never understand if your model can generalize well on unseen data $\endgroup$ Apr 7 at 19:53
  • $\begingroup$ Yes, i agree. I don't want my results to be compromised due to bad practices. I will run an experiment with SMOTE only in train set for now. Thanks! $\endgroup$ Apr 7 at 19:58
1
$\begingroup$

In my experience, some notes that I can share.

  • I wouldn't trust SMOTE to the extent to create test instances. Profoundly, it may create artificial data that is too similar to each other, hence, the high accuracy. Thus, I would create SMOTE samples only in the training set.
  • Since your dataset is small I would use something like a "leave one out" cross-validation.
  • I assume to create artificial samples for all the classes is to make your model more accurate. Again you can try with only the train data but in machine learning similar (statistical sampling) approaches are integrated into the classifiers (e.g., bootstrapping, random forest etc) to make the classifiers perform better. Speaking of which since your dataset is small you could try a decision tree implementation.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.