Skip to main content
11 events
when toggle format what by license comment
Feb 19, 2023 at 16:09 comment added Flavio Brienza Indeed I think exactly like you. In my opinion he should have called the test one validation, which has very high results (not normal for a test set), then he should have imported another dataset to test the results. For me it is not wrong the approach but the variables' names. You can like him to see which is the result on the validation set. But the best approach is to use SMOTE on the entire dataset and then Grid or Randomized Search CV to find the best parameters
Feb 19, 2023 at 15:54 comment added justinlk In the last cell of the notebook, he is printing a classification report (i.e., performance metrics) using "y_test", which has been part of the data used for SMOTE. So I still believe that his approach is wrong and the classification report does not reflect an unbiased evaluation of the model. If you have another dataset available to test the model in an unbiased way, using X_test and y_test as validation set for model training would be fine.
Feb 19, 2023 at 15:44 comment added Flavio Brienza If instead of calling it X_test I'd call it X_validation, would the approach be right? Because I think that he wanted to mean the test set as the validation one. He hase only one dataset that is X.
Feb 19, 2023 at 15:42 comment added justinlk Yes, the approach in the video is wrong. There, SMOTE is performed on both the training set and the test set (in the video's notebook "X" and "y"), which you should never do. Instead, following the video's code, performing SMOTE on "X_train" and "y_train" (defined in the video notebook one cell below SMOTE) would have been the correct approach.
Feb 19, 2023 at 15:35 comment added Flavio Brienza Many thanks, is the link approach wrong or not? Because I learned how to use SMOTE from this, but some comments say that it is wrong.
Feb 19, 2023 at 15:28 comment added justinlk It is for sure not wrong to do it the way you are doing it, i.e., splitting training and validation set before SMOTE. If you find that test results improve, then stick with your approach! And again, if you feel like the validation set might introduce overfitting to your model, consider using k-fold cross-validation instead.
Feb 19, 2023 at 15:23 comment added Flavio Brienza Because I have seen that by using SMOTE on the entire dataset and then splitting it into train and validation significantly improves the result on the validation one, but the performance is reduced on the test set. I think that it is normal since if I use it on the entire dataset and then I split it, in the validation there could be some data of the training one. So I do not know if I can use it on the entire dataset and then splitting it into train and validation, or splitting before even if I am not using the other dataset as the test one. I use this approach: youtu.be/JnlM4yLFNuo
Feb 19, 2023 at 15:06 comment added justinlk Yes, I believe your approach is right. I would personally use SMOTE on training and validation set and then use cross-validation for hyperparameter tuning to reduce possible overfitting.
Feb 19, 2023 at 14:51 comment added Flavio Brienza Many thanks for your reply. So, is my approach right? I usually use the entire dataset as the training one and I perform all the preproccessing steps on it, then I split it into train and validation for the hyperparameter tuning and evaluation, later I import other data to be used as test, obviously I process them using the scaler and the imputer created on the train+validation one. My only doubt is when I use SMOTE for balancing the dataset, in that case SMOTE can be used on the trainin+validation or only on the training one?
S Feb 19, 2023 at 14:15 review First answers
Feb 19, 2023 at 18:26
S Feb 19, 2023 at 14:15 history answered justinlk CC BY-SA 4.0