1
$\begingroup$

I have a dataset consisting of 181 samples(classes are not balanced there are 41 data points with 1 label and rest 140 are with label 0) and 10 features and one target variable. The 10 features are numeric and continuous in nature. I have to perform binary classification. I have done the following work:-

I have performed 3 Fold cross validation and got following accuracy results using various models:-

LinearSVC:
0.873
DecisionTreeClassifier:
0.840
Gaussian Naive Bayes:
0.845
Logistic Regression:
0.867
Gradient Boosting Classifier
0.867
Support vector classifier rbf:
0.818
Random forest:
0.867
K-nearest-neighbors:
0.823

Please guide me how could I choose the best model for this size of dataset and make sure my model is not overfitting ? I am thinking of applying random under sampling to handle the unbalanced data.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Hey Archit, did you create a test out of the data you have. If not then please do, and update the accuracies you achieve on the training and test set. Also calculate precision and recall, because if your dataset is imbalanced you might be getting a decent accuracy but your model will really fail at the test set. Update the question with these metrics please. Thanks. $\endgroup$
    – Himanshu Rai
    Jan 12, 2017 at 6:40
  • $\begingroup$ Could you give some more context as to what was sampled and which concept you are trying to label? $\endgroup$
    – S van Balen
    Jan 12, 2017 at 13:52
  • $\begingroup$ @HimanshuRai I have updated the question, data is imbalanced. I am thinking of random under sampling but It would result in loss of some data points then there would be only 82 observations. What would you suggest? $\endgroup$
    – Archit Garg
    Jan 13, 2017 at 2:51
  • $\begingroup$ Adding an answer. $\endgroup$
    – Himanshu Rai
    Jan 13, 2017 at 4:11
  • $\begingroup$ is it possible to public your dataset? @ArchitGarg $\endgroup$
    – JJJohn
    May 11, 2019 at 3:44

2 Answers 2

Reset to default
2
$\begingroup$

This post might be of interest. Basically by selecting the model with the best crossvalidation score, you already account for overfitting.

Also, you should separate your dataset into two parts. For the first one (validation) you run the crossvalidation on to select a model, in your case LinearSVC. For the second one (testing) you run crossvalidation again, but this time only with LinearSVC to get unbiased estimates of the accuracy.

Improve this answer
$\endgroup$
1
$\begingroup$

Firstly your data's amount is very small for any kind of analysis, so if it was posssible to get more data then that would be better. Secondly as you mentioned that your data was imbalanced then the accuracy metrics you have posted lose all meaning, since 140 samples are of the same class, the algorithm is predicting that class for every sample. So for better evaluation calculate precision, recall and f-score. Thirdly, since your data is already less than needed don't undersample, instead oversample using the SMOTE (Synthetic Minority Over Sampling Technique) implementation. Using a stratified KFold, and a Random Forest will mostly be your best bet here. But remember with this less than needed data, it would be impossible to achieve a model without underfitting or overfitting.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.