I have a dataset with 20 features and 65 samples. The models performed poorly, so I used scipy.rbf for interpolation and added 300 additional samples to the dataset. The models' performance significantly improved, and almost all models achieved an R2 of 99% on the training and testing sets. After performing cross-validation, the results were still around 99%. Is there a way to determine if my new dataset is significant, and the R2 of 99% is not due to random chance or overfitting?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
As mentioned in your situation, you got a good R-Squre value. While R-squared is a common metric, it can be misleading in some cases, especially when dealing with overfitting. I would suggest to consider using other evaluation metrics such as MSE,RMSE. These metrics will penalize the large error and give the robustness of the model.You can try the performance of your model on unseen data apart from training & validation sets.
-
$\begingroup$ Unfortunately, I can't test the unseen data. But mse, rmse and mae are close in train and test set. $\endgroup$ Commented May 26 at 13:18
$\begingroup$
$\endgroup$
1
If your data is cyclical/temporal in nature & not discrete then interpolating a few more points based on radial basis functions could help.
Tho with the proposed 4x increase in data volume via radial basis functions, I suspect your model is focusing on the interpolation datapoints (which are derivable & thus easier to learn) way more than your original data so you are seeing phenomenal metrics with no signs of overfitting.
Based on Shannon’s Sampling Theorem, to prevent your dataset from getting diluted too much I would synthesize NO more than an extra half of your original data volume.
-
$\begingroup$ Thank you very much. Honestly, R2 in the test and training set are close to each other, and in fact, both are 0.99, which shows that there is no overfit, but in some models, such as xgboost or decision tree regression, there is a huge difference between the RMSE metrics. , MSE, MAE gives the test and training set, which indicates overfit $\endgroup$ Commented May 27 at 22:36