I am a researcher in Machine Learning. In my project, I have been applying ML to a small imbalanced data consisting of 8 features and 297 instances with 44 positive instances and 253 negative ones. Firstly, I split the whole dataset into a training set (80%) and a testing set (20%) using stratified sampling. Secondly, I oversampled the training set to a balanced training set using random sampling with replacement or SMOTE; and applied information gain feature selection to reduce the features of the balanced training set. Thirdly, I trained logistic regression (LR), RBF SVM, polynomial SVM, MLP (1-hidden layer networks and 2-hidden layers networks), random forest, XGBoost, the evolutionary fuzzy classifier (Weka 3.9.4), Sugeno fuzzy system (Matlab) respectively on the reduced training set. For the oversampling step, random sampling with replacement leads to better classification performance than SMOTE. The size of the balanced training set was set to 400, 600, 800, and 1000 instances respectively. After training, each classifier was tested on the test set. This training-testing process was repeated 100 times with each time using a different training set (80%) and a test set (20%).
The results of testing are not satisfactory: LR has the best average AUC of 0.58 on the test set over 100 times.
What other algorithms do I need to try to improve the performance? Much appreciated.