choosing classifiers

Question

For what I read the 5x2cv t test is

"a procedure for comparing the performance of two models (classifiers or regressors) that was proposed by Dietterich to address shortcomings in other methods such as the resampled paired t test and the k-fold cross-validated paired t test"

.

I am currently making some experiments with an unbalanced data set, which I balanced using SCUT and trained a set of different classifiers. The problem is a multiclass with three different classes to choose. I am applying a Multilayer Perceptron, a Decision Tree and a Random Forest and the results, after a 10-fold cross validation, are the following:

Multilayer Perceptron: 0.95 acc

Decision Tree: 0.93 acc

Random Forest: 0.935 acc

When I apply the 5x2cv t test I got the following results:

MLP and DT:

t-statistic 4.75

p-value 0.005

So, if I assume that for the tests I will have a value of alpha of 0.05 for rejecting the null hypothesis, which is that both algorithms perform well with the same database, then from the p-value I got I could reject the null hypothesis. This would mean that both models do not perform equally well, so it would be better to use MLP over the DT because of its higher accuracy.

When I do the same with the MLP and the RF I got the following results:

t-statistic: 2.46

p-value: 0.055

Here, I suppose that I can conclude that using MLP or RF for the current dataset is almost the same, because I failed to reject the null hypothesis. The question that I have here is if should I choose the RF even though the accuracy was lower?

The comparison with the DT and RF has the following values:

t-statistic: -2.49

p-value: 0.054

So I can reject the null hypothesis and said that there is difference between the use of the DT over the RF.

Are my conclusions correct?

Thanks

Answer 1

See, your conclusions appear to be correct, your statistical results explain your conclusions , i would say statistically speaking even with a P-value of 0.055 you still have an error rate of 15 to 25% that you may be accepting a wrong null hypothesis and maybe your alternate hypothesis is right , so combining knowledge of statistics with data science i will suggest you to use catboost(if you have some categorical features in your dataset it will give you a strong baseline prediction before you start tuning it) otherwise random forest is good any day and if you have a small dataset (e.g. less than 1500 instances in your training data) i would suggest use random forest any day over MLP because MLP based classifiers need more data to classify better with less data they tend to become bias towards your training data.But if you have a large enough dataset you can actually use MLP based classifier try some basuc feature selection techniques such as RFECV or feature_importanece_ function you can use contained in every model after you train it . And if at the end of the day you are still confused to which model to use i would suggest use all of them stack them and you will have something better.Making these decisions solely on the basis of statistics is tough because statistics puts everything into an abstract form that is a whole dataset being represented by 1 number thus it is always prone to certain errors thus i always prefer to combine my data science experiments with my statistical results and then only conclude about my results.Also while using statistical analysis a lot of things depend on your dataset i.e. is it big enough to draw meaningful conclusions out of it what might be the sampling error and many more things .Hope this helps.