The result of ten fold cross validation for all models that i run(like linear regression DCT, rf, etc) is very close to the result of test and train, but for knn regression it is very far from the results of test and train. Finally, when I reduce the number of folds to 5 or 3, the result will be better! The dataset is small and has 70 samples and 20 features. What do you think is the cause of this problem? How can it be corrected?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
As the dataset you have has only 70 observations, it is very small! As 5-fold CV splits the dataset into more folds than 3-fold CV, each fold in 5-fold CV will have less data. So, it may be less representative of the whole dataset; hence, by random chance (because CV randomly splits the dataset), you could have gotten a worse result.
A reason why KNN may have a worse result than your other models may be that KNN is simpler than the other models. It may not be able to capture as complex of patterns as the relatively more complex models (e.g. Random Forest); so, it may not have as good as a result as the more complex models. This is not necessarily a problem, too.
If you want to get a better result in general, you could try to increase the size of the dataset (as 70 is very small).
$\begingroup$
$\endgroup$
The problem you are likely suffering from is curse of dimensionality and how knn works.
In knn regression during prediction a single point looks at it's k neighbors and calculate the average values of the k neighbors. So if we have small number of samples but the number of features are very high, then the samples would be very spread out in the feature space. So each point would have very few neighbors to calculate it's own value.
You only have 70 samples but 20 feature and further splits them into 10 splits then each split would have 63 datapoints for training and 7 for test dataset. According to how knn performs if should be gave a better result than 5 fold cross validation as you have more data during training. But this is not what happend in your case. One possible reason is that you have too few samples during testing so the results are not accurate or the knn is overfitting as it is highly depended on the value of k. small value overfits and high values underfits. On the other hand the 5 fold validation has fewer training data but not overfitting and showing more accurate test result. But with such small dataset you can trust the result easily
But at the end of the day all of this is happening because of curse of dimension. In a kaggle competition the grandmasters once said that approximately for 1 feature you should have atleast 20 samples but I can't find link now. There is also the problem of splitting data with different distribution in train and test for such a small dataset.
Suggestion:
If you want to use knn you should first do pca or umap to reduce the feature dimension then do knn and see it's performance
