0
$\begingroup$

I'm trying to find the best sorting model on a small dataset (about 600 records). I can't increase this dataset because it's real data from 600 cities and that's the scope of my study.

UPDATE: data contains information about cities (temperature, human development index, etc) and I'm aiming to use them for modelling the relationship with cases of a disease. The target variable is the class according to the disease incidence (for example: < 20 cases / mil hab is the class 0). So, I have 5 classes.

For model comparison I'm using accuracy and analyzing the confusion matrix. The algorithm is RandomForest

I would like your opinion on what I could improve in this case. Here's what I'm doing:

  1. Separate 20% of the dataset for model validation: train_test_split(X,y, test_size=0.20, random_state=42)
  2. Run GridSearch for the other 80% and get the best gsc.best_estimator_ model
  3. I apply this model in the 20% validation and check the accuracy

enter image description here

As you might notice, the accuracy is 0.40, which I think is very low. I've done other tests with other algorithms and combination of features and there are few variations (accuracy between 0.35 and 0.45).

What do you recommend I could do in this case to try to improve?

Update 2: I reviewed all the features, including some others that might be relevant to the subject of my study and excluding others that were not supported in the scientific literature. Now the accuracy is around 0.56 from GridSearch, which is far from what I would like (>0.7), but it is already a considerable improvement.

I tested it with 3 different algorithms (Random Forest, MLPClassfier and XGBoost), with RandomForest being a little better than the neural network.

As you can see in the new confusion matrix, class 2 is the one with the worst performance. It has fewer records, but it's not that big of a difference for class 1, for example. Any hints here of what I could look for in relation to this class?

enter image description here

Thank you very much.

$\endgroup$
4
  • $\begingroup$ Can you give some more background on the type of task you are trying to solve? Accuracy on it's own doesn't tell the whole story, an accuracy of 40% is not very good if you trying to classify observations into three categories, but is much better if you're using 1000 categories. $\endgroup$
    – Oxbowerce
    Commented Dec 17, 2021 at 14:45
  • $\begingroup$ @Oxbowerce thank you! I've updated the post to describe my case $\endgroup$
    – mr x
    Commented Dec 17, 2021 at 15:02
  • 1
    $\begingroup$ Looks sound from your general approach. Probably your $X$ just don't has enough predictive power to explain the target and/or the data generating process is not well understood. The full list of explanatory variables would be interesting. E.g. do you include the "country"? $\endgroup$
    – Peter
    Commented Dec 17, 2021 at 17:15
  • $\begingroup$ @Peter You're right. I've updated the post with new results. It's still not great, but I see an improvement $\endgroup$
    – mr x
    Commented Dec 28, 2021 at 16:24

1 Answer 1

1
$\begingroup$

From what I see from the discussion in the comments, I understand that there are two possible issues: class imbalance and few explanatory variables (or a lack of predictive power).

For the first issue, you may look into SMOTE (synthetic oversampling https://imbalanced-learn.org/stable/references/generated/imblearn.over_sampling.SMOTE.html). In any case you need to get more balanced classes in order to tackle the „class two issue“.

The second issue could be approached by adding new (generated) features. You could add „interactions“ e.g. $x_1 - x_2$ or $x_1 / x_2$ etc. Usually „minus“ and „divided by“ are used. By doing so you can (sometimes) gain explanatory power, especially in tree based models. Since the number of all possible interactions usually is „large“ you need to do feature selection (i.e. only keep the „best“ interactions), for instance based on feature importance. Alternatively you could use all possible interactions and use „shrinkage“ of features, e.g based on Lasso or Ridge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.